my cheat sheet on nginx
1 nginx vs apache
Both are very common web server software. Both have their strengths and
weaknesses. I had issues with configuring flask
apps to work with apache,
so I successfully moved to flask
on nginx
2 nginx commands
nginx -h
for help commandsnginx -v
shows you the version, (and confirms that nginx is installed)nginx -V
version plus additional info: SSL version, config options, compiled modules, etc.nginx -t
test your configurationnginx -T
test your configuration and save it to output to STDOUT i.e. trynginx -T | fsf
nginx -s
signal your nginx to stop, quit, reopen, reload
To start nginx, use systemctl start nginx
. On ubuntu, it's
/etc/init.d/nginx start
You can also start and stop nginx using an nginx command itself:
nginx -s signal
Where signal may be one of the following:
stop — fast shutdown
quit — graceful shutdown
reload — reloading the configuration file
reopen — reopening the log files
So, for example to stop nginx: nginx -s stop
to start: nginx -s reload
??
Review more of https://nginx.org/en/docs/beginners_guide.html to see what
what else I don't have yet.
3 nginx configuration files & directories
-j /etc/nginx
config directory
/etc/nginx/conf.d/
config directory where you make changes to the config/etc/nginx/nginx.conf
generated configs from files in/ertc/nfinx/conf.d/
/etc/nginx/conf.d/default.conf
start with this config file
4 nginx webpages
/usr/share/nginx/html
/usr/share/nginx/html
/usr/share/nginx/html
4.1 workerconnections
You can directly edit /etc/nginx/conf.d
to edit and set the number of
worker_connections
, a line near the top. Run 1 process per CPU on your vm.
You could also set it to auto
.
5 TLS certificate install on nginx
You need these files in your /etc/pki/tls/certs
directory:
4abe36dee147d6a4.crt
The signed cert received from your CA providerca-bundle-crt
pem file symlinked to/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
. This bundle contains- a) your public key,
- b) your cert details,
- c) CA signature that validates your certificate in your web client's eyes
Concatenate
theprimary
andintermediate certificates
. like so:cat acme.com.crt godaddy.com.crt >> bundle.crt
For me that file became:
cat 4abe36dee147d6a4.crt gd_bundle-g2-g1.crt >> bundlezp.crt
5.1 nginx ssl files conflicting info or not
Some docs claim that the public certificate, (.crt file) and the private key
(.key file) are found in /etc/nginx/ssl
directory. I found them to be in
the /etc/pki/tls/certs
and /etc/pki/tls/private
directories respectively.
Upon further review, the certs/keys in those dirs are copies of one another. So all is good. I suspect maybe nginx looks in certain folders hence the duplication. But I am not sure.
5.2 Encrypting your keys
A good blog discussing how to increase your security of your private keys is found in nginx.com blog. Read this and follow the recommendations.
5.3 Some excerpts from my apache ssl.conf file
THis file is /etc/httpd/conf/ssl.conf
.
# SSLCertificateFile /etc/pki/tls/certs/localhost.crt SSLCertificateFile /etc/pki/tls/certs/4abe36dee147d6a4.crt # Server Private Key: SSLCertificateKeyFile /etc/pki/tls/private/zintis.net.key # Server Certificate Chain: # Point SSLCertificateChainFile at a file containing the # concatenation of PEM encoded CA certificates which form the # certificate chain for the server certificate. #SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt # Certificate Authority (CA): # Set the CA certificate verification path where to find CA # certificates for client authentication or alternatively one # huge file containing all of them (file must be PEM encoded) #SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt SSLCACertificateFile /etc/pki/tls/certs/gd_bundle-g2-g1.crt
6 403 Forbidden
If your browser tries to access a static web page, gets index.html fine, but
gets a 403 Forbidden error
on any other file, you might have problems in:
7 Various attacks on zintis.net
In viewing /var/log/nginx/error.log I come across various hack attempts on my domain. These are a few of them:
awk '/43.129.24/ {print $2, $7}' error.log
This was from Singapore
11:24:05 "/usr/share/nginx/html/dns-query" 11:24:05 "/usr/share/nginx/html/dns-query" 11:24:05 "/usr/share/nginx/html/resolve" 11:24:05 "/usr/share/nginx/html/resolve" 11:24:05 "/usr/share/nginx/html/doh" 11:24:05 "/usr/share/nginx/html/doh" 11:24:05 "/usr/share/nginx/html/doh/family-filter" 11:24:05 "/usr/share/nginx/html/doh/family-filter" 11:24:05 "/usr/share/nginx/html/doh/secure-filter" 11:24:05 "/usr/share/nginx/html/doh/secure-filter" 11:24:05 "/usr/share/nginx/html/query" 11:24:05 "/usr/share/nginx/html/query" 11:24:05 "/usr/share/nginx/html/ads" 11:24:05 "/usr/share/nginx/html/ads" 11:24:05 "/usr/share/nginx/html/uncensored" 11:24:05 "/usr/share/nginx/html/uncensored" 11:24:05 "/usr/share/nginx/html/dns-query" 11:24:05 "/usr/share/nginx/html/dns-query" 11:24:05 "/usr/share/nginx/html/resolve" 11:24:05 "/usr/share/nginx/html/resolve" 11:24:05 "/usr/share/nginx/html/doh" 11:24:05 "/usr/share/nginx/html/doh" 11:24:05 "/usr/share/nginx/html/doh/family-filter" 11:24:05 "/usr/share/nginx/html/doh/family-filter" 11:24:05 "/usr/share/nginx/html/doh/secure-filter" 11:24:05 "/usr/share/nginx/html/doh/secure-filter" 11:24:05 "/usr/share/nginx/html/query" 11:24:05 "/usr/share/nginx/html/query" 11:24:05 "/usr/share/nginx/html/ads" 11:24:05 "/usr/share/nginx/html/ads" 11:24:05 "/usr/share/nginx/html/uncensored" 11:24:05 "/usr/share/nginx/html/uncensored" 11:24:05 "/usr/share/nginx/html/dns-query" 11:24:05 "/usr/share/nginx/html/dns-query" 11:24:05 "/usr/share/nginx/html/resolve" 11:24:06 "/usr/share/nginx/html/resolve" 11:24:06 "/usr/share/nginx/html/doh" 11:24:06 "/usr/share/nginx/html/doh" 11:24:06 "/usr/share/nginx/html/doh/family-filter" 11:24:06 "/usr/share/nginx/html/doh/family-filter" 11:24:06 "/usr/share/nginx/html/doh/secure-filter" 11:24:06 "/usr/share/nginx/html/doh/secure-filter" 11:24:06 "/usr/share/nginx/html/query" 11:24:06 "/usr/share/nginx/html/query" 11:24:06 "/usr/share/nginx/html/ads" 11:24:06 "/usr/share/nginx/html/ads" 11:24:06 "/usr/share/nginx/html/uncensored" 11:24:06 "/usr/share/nginx/html/uncensored"
All within 2 seconds. with an error: failed (2: No such file or directory),
Similarly:
awk '/37.0.15.238/ {print $2, $7}' error.log
This was from the Netherlands
05:46:46 "/usr/share/nginx/html/wp-includes/wlwmanifest.xml" 05:46:46 "/usr/share/nginx/html/xmlrpc.php" 05:46:47 "/usr/share/nginx/html/blog/wp-includes/wlwmanifest.xml" 05:46:47 "/usr/share/nginx/html/web/wp-includes/wlwmanifest.xml" 05:46:48 "/usr/share/nginx/html/wordpress/wp-includes/wlwmanifest.xml" 05:46:48 "/usr/share/nginx/html/website/wp-includes/wlwmanifest.xml" 05:46:48 "/usr/share/nginx/html/wp/wp-includes/wlwmanifest.xml" 05:46:48 "/usr/share/nginx/html/news/wp-includes/wlwmanifest.xml" 05:46:49 "/usr/share/nginx/html/2018/wp-includes/wlwmanifest.xml" 05:46:49 "/usr/share/nginx/html/2019/wp-includes/wlwmanifest.xml" 05:46:49 "/usr/share/nginx/html/shop/wp-includes/wlwmanifest.xml" 05:46:49 "/usr/share/nginx/html/wp1/wp-includes/wlwmanifest.xml" 05:46:49 "/usr/share/nginx/html/test/wp-includes/wlwmanifest.xml" 05:46:50 "/usr/share/nginx/html/media/wp-includes/wlwmanifest.xml" 05:46:50 "/usr/share/nginx/html/wp2/wp-includes/wlwmanifest.xml" 05:46:50 "/usr/share/nginx/html/site/wp-includes/wlwmanifest.xml" 05:46:50 "/usr/share/nginx/html/cms/wp-includes/wlwmanifest.xml" 05:46:51 "/usr/share/nginx/html/sito/wp-includes/wlwmanifest.xml"
Again, within a short time, and error: failed (2: No such file or directory),
Here is an attempt to POST to a logservice:
This was all on one line: 2022/04/05 08:54:29 [error] 172184#0: *265 open() "/usr/share/nginx/html/mifs/. \;/services/LogService" failed (2: No such file or directory), client: 45.155.204.146, server: www.zintis.net, request: "POST /mifs/.;/services/LogService HTTP/1.1", host: "139.177.192.45:443", referrer: "https://139.177.192.45:443"
In debugging my permission errors, I used this awk command often:
awk '/Permission denied), / {print $2, $7, $12, $13}' error.log
7.1 Where are the clients that try?
It would be easy to have a crontab job that ran every day to do this:
psuedo code:
1. awk for client: ip address 2. sort numerically 3. get the uniq addresses, and their count. 4. sort by the access count, in decreasing order 5. top 20 clients, by count, run a whois lookup 6. print the country of origin.
8 Permissions for apache and nginx
These are some useful and common commands when working with web server permission errors:
## apache sudo add-user $(whoami) www-data && \ sudo chown -R www-data:www-data /var/www && \ sudo chmod -R g+rw /var/www; find /var/www -type d -print0 | sudo xargs -0 chmod g+s ## nginx sudo add-user $(whoami) nginx sudo chown -R nginx:nginx /usr/share/nginx/html sudo chmod -R g+rw /usr/share/nginx/html find /usr/share/nginx/html -type d -print0 | sudo xargs -0 chmod g+s
Another approach, and possibly easier, is simply make the user a member of the
nginx group, so sudo usermod -a -G nginx zintis
9 SELinux issues preventing execution
I thought I was having SELInux problems when I continued to get the following permissions error when starting recipes:
zintis systemd[1]: Started Gunicorn serving recipes. zintis systemd[22813]: recipe.service: Failed to execute command: Permission denied zintis systemd[22813]: recipe.service: Failed at step EXEC spawning /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: Permission denied zintis systemd[1]: recipe.service: Main process exited, code=exited, status=203/EXEC Zintis systemd[1]: recipe.service: Failed with result 'exit-code'.
So I executed these SELinux
commands, but to no avail. So I might have to undo
these commands in the near future:
1080 sudo semange fcontext --list 1081 sudo semanage fcontext --list 1082 sudo semanage fcontext --list | grep nginx 1083 sudo semanage fcontext -a -t bin_t '/home/zintis/bin/app-flask' 1084 sudo chcon -Rv -u system_u -t bin_t '/home/zintis/bin/app-flask' 1085 sudo restorecon -R -v /home/zintis/bin/app-flask 1086 ssr recipe
And this was my sudo systemctl status recipe
output:
(venv-flask) zintis@zintis.net /etc/systemd/system[1087]: $ sss recipe ● recipe.service - Gunicorn serving recipes Loaded: loaded (/etc/systemd/system/recipe.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2023-06-06 15:16:14 EDT; 3s ago Process: 22813 ExecStart=/home/zintis/bin/app-flask/venv-flask/bin/gunicorn --workers 3 --bind unix:app-flask.sock -m 007 wsgi:app (code=exited, status=> Main PID: 22813 (code=exited, status=203/EXEC) Jun 06 15:16:14 zintis.net systemd[1]: Started Gunicorn serving recipes. Jun 06 15:16:14 zintis.net systemd[1]: recipe.service: Main process exited, code=exited, status=203/EXEC Jun 06 15:16:14 zintis.net systemd[1]: recipe.service: Failed with result 'exit-code'. (venv-flask) zintis@zintis.net /etc/systemd/system[1088]:
9.1 Changed recipe.service to run /bin/bash
I changed the ExecStart line to be:
# from ExecStart=/home/zintis/bin/app-flask/venv-flask/bin/gunicorn --workers 3 --bind unix:app-flask.sock -m 007 wsgi:app # to ExecStart=/bin/bash /home/zintis/bin/app-flask/venv-flask/bin/gunicorn --workers 3 --bind unix:app-flask.sock -m 007 wsgi:app
Now I got a different output from systemctl status recipe
:
$ sss !$ sss recipe ● recipe.service - Gunicorn serving recipes Loaded: loaded (/etc/systemd/system/recipe.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2023-06-06 15:29:07 EDT; 2s ago Process: 22881 ExecStart=/bin/bash /home/zintis/bin/app-flask/venv-flask/bin/gunicorn --workers 3 --bind unix:app-flask.sock -m 007 wsgi:app (code=exite> Main PID: 22881 (code=exited, status=2) Jun 06 15:29:07 zintis.net systemd[1]: Started Gunicorn serving recipes. Jun 06 15:29:07 zintis.net bash[22883]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 3: import: command not found Jun 06 15:29:07 zintis.net bash[22884]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 4: import: command not found Jun 06 15:29:07 zintis.net bash[22885]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 5: from: command not found Jun 06 15:29:07 zintis.net bash[22881]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 7: syntax error near unexpected token `(' Jun 06 15:29:07 zintis.net bash[22881]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 7: ` sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '> Jun 06 15:29:07 zintis.net systemd[1]: recipe.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jun 06 15:29:07 zintis.net systemd[1]: recipe.service: Failed with result 'exit-code'. lines 1-14/14 (END)
With this seen in /var/log/messages
Jun 6 15:29:07 zintis systemd[1]: Started Gunicorn serving recipes. Jun 6 15:29:07 zintis bash[22883]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 3: import: command not found Jun 6 15:29:07 zintis bash[22884]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 4: import: command not found Jun 6 15:29:07 zintis bash[22885]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 5: from: command not found Jun 6 15:29:07 zintis bash[22881]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 7: syntax error near unexpected token `(' Jun 6 15:29:07 zintis bash[22881]: /home/zintis/bin/app-flask/venv-flask/bin/gunicorn: line 7: ` sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])' Jun 6 15:29:07 zintis systemd[1]: recipe.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jun 6 15:29:07 zintis systemd[1]: recipe.service: Failed with result 'exit-code'.