my cheat sheet on nginx

Home

1 nginx vs apache

Both are very common web server software. Both have their strengths and weaknesses. I had issues with configuring flask apps to work with apache, so I successfully moved to flask on nginx

2 nginx commands

  • nginx -h for help commands
  • nginx -v shows you the version, (and confirms that nginx is installed)
  • nginx -V version plus additional info: SSL version, config options, compiled modules, etc.
  • nginx -t test your configuration
  • nginx -T test your configuration and save it to output to STDOUT i.e. try nginx -T | fsf
  • nginx -s signal your nginx to stop, quit, reopen, reload

To start nginx, use systemctl start nginx. On ubuntu, it's /etc/init.d/nginx start

You can also start and stop nginx using an nginx command itself:

  • nginx -s signal

Where signal may be one of the following:

  • stop — fast shutdown
  • quit — graceful shutdown
  • reload — reloading the configuration file
  • reopen — reopening the log files

So, for example to stop nginx: nginx -s stop to start: nginx -s reload ?? Review more of https://nginx.org/en/docs/beginners_guide.html to see what what else I don't have yet.

3 nginx configuration files & directories

-j /etc/nginx config directory

  • /etc/nginx/conf.d/ config directory where you make changes to the config
  • /etc/nginx/nginx.conf generated configs from files in /ertc/nfinx/conf.d/
  • /etc/nginx/conf.d/default.conf start with this config file

4 nginx webpages

  • /usr/share/nginx/html
  • /usr/share/nginx/html
  • /usr/share/nginx/html

4.1 workerconnections

You can directly edit /etc/nginx/conf.d to edit and set the number of worker_connections, a line near the top. Run 1 process per CPU on your vm.

You could also set it to auto.

5 TLS certificate install on nginx

You need these files in your /etc/pki/tls/certs directory:

  1. 4abe36dee147d6a4.crt The signed cert received from your CA provider
  2. ca-bundle-crt pem file symlinked to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem. This bundle contains
    • a) your public key,
    • b) your cert details,
    • c) CA signature that validates your certificate in your web client's eyes
  3. Concatenate the primary and intermediate certificates. like so:
    • cat acme.com.crt godaddy.com.crt >> bundle.crt

    For me that file became:

    cat 4abe36dee147d6a4.crt gd_bundle-g2-g1.crt >> bundlezp.crt
    

5.1 nginx ssl files conflicting info or not

Some docs claim that the public certificate, (.crt file) and the private key (.key file) are found in /etc/nginx/ssl directory. I found them to be in the /etc/pki/tls/certs and /etc/pki/tls/private directories respectively.

Upon further review, the certs/keys in those dirs are copies of one another. So all is good. I suspect maybe nginx looks in certain folders hence the duplication. But I am not sure.

5.2 Encrypting your keys

A good blog discussing how to increase your security of your private keys is found in nginx.com blog. Read this and follow the recommendations.

5.3 Some excerpts from my apache ssl.conf file

THis file is /etc/httpd/conf/ssl.conf.

# SSLCertificateFile /etc/pki/tls/certs/localhost.crt                                                                                     
SSLCertificateFile /etc/pki/tls/certs/4abe36dee147d6a4.crt                                                                                

#   Server Private Key:                                                                                                                   
SSLCertificateKeyFile /etc/pki/tls/private/zintis.net.key                                                                                 

#   Server Certificate Chain:                                                                                                             
#   Point SSLCertificateChainFile at a file containing the                                                                                
#   concatenation of PEM encoded CA certificates which form the                                                                           
#   certificate chain for the server certificate.
#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt                                                                              

#   Certificate Authority (CA):                                                                                                           
#   Set the CA certificate verification path where to find CA                                                                             
#   certificates for client authentication or alternatively one                                                                           
#   huge file containing all of them (file must be PEM encoded)                                                                           
#SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt                                                                                    
SSLCACertificateFile /etc/pki/tls/certs/gd_bundle-g2-g1.crt                                                                               

6 403 Forbidden

If your browser tries to access a static web page, gets index.html fine, but gets a 403 Forbidden error on any other file, you might have problems in:

7 Various attacks on zintis.net

In viewing /var/log/nginx/error.log I come across various hack attempts on my domain. These are a few of them:

awk '/43.129.24/ {print $2, $7}' error.log

This was from Singapore

11:24:05 "/usr/share/nginx/html/dns-query"
11:24:05 "/usr/share/nginx/html/dns-query"
11:24:05 "/usr/share/nginx/html/resolve"
11:24:05 "/usr/share/nginx/html/resolve"
11:24:05 "/usr/share/nginx/html/doh"
11:24:05 "/usr/share/nginx/html/doh"
11:24:05 "/usr/share/nginx/html/doh/family-filter"
11:24:05 "/usr/share/nginx/html/doh/family-filter"
11:24:05 "/usr/share/nginx/html/doh/secure-filter"
11:24:05 "/usr/share/nginx/html/doh/secure-filter"
11:24:05 "/usr/share/nginx/html/query"
11:24:05 "/usr/share/nginx/html/query"
11:24:05 "/usr/share/nginx/html/ads"
11:24:05 "/usr/share/nginx/html/ads"
11:24:05 "/usr/share/nginx/html/uncensored"
11:24:05 "/usr/share/nginx/html/uncensored"
11:24:05 "/usr/share/nginx/html/dns-query"
11:24:05 "/usr/share/nginx/html/dns-query"
11:24:05 "/usr/share/nginx/html/resolve"
11:24:05 "/usr/share/nginx/html/resolve"
11:24:05 "/usr/share/nginx/html/doh"
11:24:05 "/usr/share/nginx/html/doh"
11:24:05 "/usr/share/nginx/html/doh/family-filter"
11:24:05 "/usr/share/nginx/html/doh/family-filter"
11:24:05 "/usr/share/nginx/html/doh/secure-filter"
11:24:05 "/usr/share/nginx/html/doh/secure-filter"
11:24:05 "/usr/share/nginx/html/query"
11:24:05 "/usr/share/nginx/html/query"
11:24:05 "/usr/share/nginx/html/ads"
11:24:05 "/usr/share/nginx/html/ads"
11:24:05 "/usr/share/nginx/html/uncensored"
11:24:05 "/usr/share/nginx/html/uncensored"
11:24:05 "/usr/share/nginx/html/dns-query"
11:24:05 "/usr/share/nginx/html/dns-query"
11:24:05 "/usr/share/nginx/html/resolve"
11:24:06 "/usr/share/nginx/html/resolve"
11:24:06 "/usr/share/nginx/html/doh"
11:24:06 "/usr/share/nginx/html/doh"
11:24:06 "/usr/share/nginx/html/doh/family-filter"
11:24:06 "/usr/share/nginx/html/doh/family-filter"
11:24:06 "/usr/share/nginx/html/doh/secure-filter"
11:24:06 "/usr/share/nginx/html/doh/secure-filter"
11:24:06 "/usr/share/nginx/html/query"
11:24:06 "/usr/share/nginx/html/query"
11:24:06 "/usr/share/nginx/html/ads"
11:24:06 "/usr/share/nginx/html/ads"
11:24:06 "/usr/share/nginx/html/uncensored"
11:24:06 "/usr/share/nginx/html/uncensored"

All within 2 seconds. with an error: failed (2: No such file or directory),

Similarly:

awk '/37.0.15.238/ {print $2, $7}' error.log

This was from the Netherlands

05:46:46 "/usr/share/nginx/html/wp-includes/wlwmanifest.xml"
05:46:46 "/usr/share/nginx/html/xmlrpc.php"
05:46:47 "/usr/share/nginx/html/blog/wp-includes/wlwmanifest.xml"
05:46:47 "/usr/share/nginx/html/web/wp-includes/wlwmanifest.xml"
05:46:48 "/usr/share/nginx/html/wordpress/wp-includes/wlwmanifest.xml"
05:46:48 "/usr/share/nginx/html/website/wp-includes/wlwmanifest.xml"
05:46:48 "/usr/share/nginx/html/wp/wp-includes/wlwmanifest.xml"
05:46:48 "/usr/share/nginx/html/news/wp-includes/wlwmanifest.xml"
05:46:49 "/usr/share/nginx/html/2018/wp-includes/wlwmanifest.xml"
05:46:49 "/usr/share/nginx/html/2019/wp-includes/wlwmanifest.xml"
05:46:49 "/usr/share/nginx/html/shop/wp-includes/wlwmanifest.xml"
05:46:49 "/usr/share/nginx/html/wp1/wp-includes/wlwmanifest.xml"
05:46:49 "/usr/share/nginx/html/test/wp-includes/wlwmanifest.xml"
05:46:50 "/usr/share/nginx/html/media/wp-includes/wlwmanifest.xml"
05:46:50 "/usr/share/nginx/html/wp2/wp-includes/wlwmanifest.xml"
05:46:50 "/usr/share/nginx/html/site/wp-includes/wlwmanifest.xml"
05:46:50 "/usr/share/nginx/html/cms/wp-includes/wlwmanifest.xml"
05:46:51 "/usr/share/nginx/html/sito/wp-includes/wlwmanifest.xml"

Again, within a short time, and error: failed (2: No such file or directory),

Here is an attempt to POST to a logservice:

  This was all on one line:
2022/04/05 08:54:29 [error] 172184#0: *265 open() "/usr/share/nginx/html/mifs/.
\;/services/LogService" failed (2: No such file or directory),
client: 45.155.204.146, server: www.zintis.net, request:
"POST /mifs/.;/services/LogService HTTP/1.1", host: "139.177.192.45:443",
referrer: "https://139.177.192.45:443"

In debugging my permission errors, I used this awk command often:

  • awk '/Permission denied), / {print $2, $7, $12, $13}' error.log

7.1 Where are the clients that try?

It would be easy to have a crontab job that ran every day to do this:

psuedo code:

1. awk for client: ip address
2. sort numerically
3. get the uniq addresses, and their count.
4. sort by the access count, in decreasing order
5. top 20 clients, by count, run a whois lookup
6. print the country of origin.

8 Permissions for apache and nginx

These are some useful and common commands when working with web server permission errors:

## apache

sudo add-user $(whoami) www-data && \
sudo chown -R www-data:www-data /var/www && \
sudo chmod -R g+rw /var/www; find /var/www -type d -print0 | sudo xargs -0 chmod g+s

## nginx

sudo add-user $(whoami) nginx  
sudo chown -R nginx:nginx /usr/share/nginx/html
sudo chmod -R g+rw /usr/share/nginx/html
find /usr/share/nginx/html -type d -print0 | sudo xargs -0 chmod g+s

Another approach, and possibly easier, is simply make the user a member of the nginx group, so sudo usermod -a -G nginx zintis

9 SELinux issues preventing execution

I thought I was having SELInux problems when I continued to get the following permissions error when starting recipes:

zintis systemd[1]: Started Gunicorn serving recipes.
zintis systemd[22813]: recipe.service: Failed to execute command: Permission denied
zintis systemd[22813]: recipe.service: Failed at step EXEC spawning
               /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: Permission denied
zintis systemd[1]: recipe.service: Main process exited, code=exited, status=203/EXEC
Zintis systemd[1]: recipe.service: Failed with result 'exit-code'.

So I executed these SELinux commands, but to no avail. So I might have to undo these commands in the near future:

1080  sudo semange fcontext --list
1081  sudo semanage fcontext --list
1082  sudo semanage fcontext --list | grep nginx
1083  sudo semanage fcontext -a -t bin_t '/home/zintis/bin/app-flask'
1084  sudo chcon -Rv -u system_u -t bin_t '/home/zintis/bin/app-flask'
1085  sudo restorecon -R -v /home/zintis/bin/app-flask
1086  ssr recipe

And this was my sudo systemctl status recipe output:

(venv-flask) zintis@zintis.net /etc/systemd/system[1087]:
$ sss recipe
● recipe.service - Gunicorn serving recipes
   Loaded: loaded (/etc/systemd/system/recipe.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2023-06-06 15:16:14 EDT; 3s ago
  Process: 22813 ExecStart=/home/zintis/bin/app-flask/venv-flask/bin/gunicorn --workers 3 --bind unix:app-flask.sock -m 007 wsgi:app (code=exited, status=>
 Main PID: 22813 (code=exited, status=203/EXEC)

Jun 06 15:16:14 zintis.net systemd[1]: Started Gunicorn serving recipes.
Jun 06 15:16:14 zintis.net systemd[1]: recipe.service: Main process exited, code=exited, status=203/EXEC
Jun 06 15:16:14 zintis.net systemd[1]: recipe.service: Failed with result 'exit-code'.
(venv-flask) zintis@zintis.net /etc/systemd/system[1088]:

9.1 Changed recipe.service to run /bin/bash

I changed the ExecStart line to be:

# from
ExecStart=/home/zintis/bin/app-flask/venv-flask/bin/gunicorn --workers 3 --bind unix:app-flask.sock -m 007 wsgi:app

# to
ExecStart=/bin/bash /home/zintis/bin/app-flask/venv-flask/bin/gunicorn --workers 3 --bind unix:app-flask.sock -m 007 wsgi:app

Now I got a different output from systemctl status recipe:

$ sss !$
sss recipe
● recipe.service - Gunicorn serving recipes
   Loaded: loaded (/etc/systemd/system/recipe.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2023-06-06 15:29:07 EDT; 2s ago
  Process: 22881 ExecStart=/bin/bash /home/zintis/bin/app-flask/venv-flask/bin/gunicorn --workers 3 --bind unix:app-flask.sock -m 007 wsgi:app (code=exite>
 Main PID: 22881 (code=exited, status=2)

Jun 06 15:29:07 zintis.net systemd[1]: Started Gunicorn serving recipes.
Jun 06 15:29:07 zintis.net bash[22883]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 3: import: command not found
Jun 06 15:29:07 zintis.net bash[22884]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 4: import: command not found
Jun 06 15:29:07 zintis.net bash[22885]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 5: from: command not found
Jun 06 15:29:07 zintis.net bash[22881]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 7: syntax error near unexpected token `('
Jun 06 15:29:07 zintis.net bash[22881]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 7: `    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '>
Jun 06 15:29:07 zintis.net systemd[1]: recipe.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 06 15:29:07 zintis.net systemd[1]: recipe.service: Failed with result 'exit-code'.
lines 1-14/14 (END)

With this seen in /var/log/messages

Jun  6 15:29:07 zintis systemd[1]: Started Gunicorn serving recipes.
Jun  6 15:29:07 zintis bash[22883]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 3: import: command not found
Jun  6 15:29:07 zintis bash[22884]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 4: import: command not found
Jun  6 15:29:07 zintis bash[22885]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 5: from: command not found
Jun  6 15:29:07 zintis bash[22881]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 7: syntax error near unexpected token `('
Jun  6 15:29:07 zintis bash[22881]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 7: `    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])'
Jun  6 15:29:07 zintis systemd[1]: recipe.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun  6 15:29:07 zintis systemd[1]: recipe.service: Failed with result 'exit-code'.

9.2 Home