My Learning Python Notes

Home

1 Passing parameters in imported modules:

A good example follows:

scrmodule.py 

def fun(a, b):
    # possibly do something here
    return a + b

# same thing but this time restricting types and return types
def fun(a:int, b:int) -> int:
    # possibly do something here
    return a + b

def main():
    #process command line argumens
    a = 1 #will be read from command line
    b = 2 #will be read from command line
    # call fun()
    res = fun(a, b)
    print "a", a
    print "b", b
    print "result is", res

if __name__ == "__main__":
    main()

Reusing it from another place: 

from scrmodule import fun

print "1 + 2 = ", fun(1, 2)

2 Generic Python Libraries

2.1 Core Python Libraries to use

from pprint import pprint    #  Pretty Print
import sys                   #  Python interpreter utilities
import os                    #  Operating system interfaces
import datetime              #  Date and time utilities
from datetime import date
import pandas as pd          # good
from pandas import *         # bad.  Do not do this because you could be clobbering
                               # system commands with some of the * imported methods.

2.2 import sys

A very common library to access system facilities. 

import sys     # gives us details about the running state of python
len(sys.argv)  # gives us how many arguments on the command line
str(sys.argv)  # gives all the arguments as a list, including sys.argv[0]
sys.argv[0]    # this is the name of the program I am running
sys.argv[1]    # this is the first command line argument passed to the script
sys.argv[2]    # this is the second command line argument passed to the script
sys.exit("Error Occurred")    # exit python with a specific error message
                              # ee also link here
str(sys.argv)  # if you print this, you can see all the arguments

if __name__ == '__main__':
    _, *script_args = sys.argv

    print(f"len(sys.argv) is: {len(sys.argv)} ")
    print(f"str(sys.argv) is {str(sys.argv)} \n")
    print(f"undersocre is: {_} \n")
    print(f"len(script_args) is: {len(script_args)} ")
    print(f"script_args is: {script_args} \n")
    print(f"type(script_args) is: {type(script_args)}\n")

After running this last part with scrypt.py a b c d e f I got: 

len(sys.argv) is: 7 
str(sys.argv) is ['./test-4-json.py', 'a', 'b', 'c', 'd', 'e', 'f']

undersocre is: ./test-4-json.py

len(script_args) is: 6 
script_args is: ['a', 'b', 'c', 'd', 'e', 'f']

type(script_args) is: <class 'list'>

2.2.1 An example:

print(f'Number of arguments: {len(sys.argv)} arguments.')
print (f'Argument Lisf: {str(sys.argv)}')

Now run the above script as follows − 

$ python test.py arg1 arg2 arg3

This produces the following result − 

Number of arguments: 4 arguments.
Argument List: ['test.py', 'arg1', 'arg2', 'arg3']

A common use is reading in the arguments only if a script was executed and not imported is the following: 

if __name__ == '__main__':
    _, *script_args = sys.argv      # _ tells python to ignore this,  *script_args tells load
    main(*script_args)              # all the remaining arguments until I run out of arguments
                                    # into a list.

2.3 import os

Access and manipulate directories, files, environment varialbes.

  • os.getcwd()
  • os.executable * good way to see which python you are running
  • os.environ
  • os.environ["USER"]
  • os.environ["VARFROMPYTHON"] = "OK, this was set within python"
  • os.scandir('/Users/zintis/bin/python/bin')
  • entries = os.scandir('/Users/zintis/bin/python/bin')
  • os.walk
  • os.system('cp source.txt destination.txt') # copies files, but any cmd ok

I have also seen os.environ.get('USER') but this is deprecated. They BOTH still WORK but the newer is os.environ["USER"] In python 3.9 docs it looks like os.environ.get does not exist. This is not confirmed, as if the dictionary does NOT contain the key, get() will return NONE but NOT raise an exception, potentially making your life easier. Without using get(), your code should use a try: except: block for those corner cases where the dictionary element does NOT exist.

2.3.1 subtleties on os.environ

  • os.getenv() does not raise an exception, but returns None
  • os.environ.get() similarly returns None
  • os.environ[] raises an exception if the environmental variable does not exist so put it in a try: block. 
# Getting non-existent keys, i.e. FOO, BAR, and BAZ do not exist:
FOO = os.getenv('FOO') # None
BAR = os.environ.get('BAR') # None
BAZ = os.environ['BAZ'] # KeyError: key does not exist.

You can just as easily set environment variables like so:

  • os.environ['USER'] = 'zintis'

Environment variables are useful when you want to keep your access credentials or other variables out of your code, and git repository.

General examples on using the os module 



>>> import os
>>> os.getcwd()
'/Users/zintis/bin'
>>> os.chdir('../Documents')
>>> os.getcwd()
'/Users/zintis/Documents'
>>> os.environ["USER"]
'zintis'
>>> os.environ["VAR_FROM_PYTHON"]

Traceback (most recent call last):
File "<stdin>", line 1, in <module
>
File "/Users/zintis/.pyenv/versions/3.6.5/lib/python3.6/os.py", line 669, in __getitem__
raise KeyError(key) from None
KeyError: 'VAR_FROM_PYTHON'    # because VAR_FROM_PYTHON was not yet set.

>>> os.environ["VAR_FROM_PYTHON"] = "Ok, set this from Python"
>>> os.environ["VAR_FROM_PYTHON"]
'Ok, set this from Python'     # now it is set and there is no error
>>>

2.4 glob.glob

glob is a standard python library module that can be used to effectively work your python script through multiple files in a directory.

For example: 

import os
import glob

files = [file for file in glob.glob("*.json") if "tcp" in file or "udp" in files]
files = [file for file in glob.glob("/Users/zintis/covid-output/*.json")
files = [file for file in glob.glob("/Users/zintis/covid-output/*.json")
for file_name in files:
    with open(file_name, 'r') as ingest
    # do something with this file, like maybe substitute a regexp ?

# compare that to os.listdir like this:
res = [f for f in os.listdir(path) if re.search(r'(tcp|udp).*\.json$', f)]
for f in res:
    print f

2.5 os.walk as alternative to glob.glob

You can use os.walk to scan all files in a directory 

for root, dirs, files in os.walk('/Users/home/zintis'):
   for file in files:
       filename, extension = os.path.splitext(file)
       if extension == '.txt'
           chmod a-x file

       if 'hello' in filename:
          mv filename ~/all-hello-files

Another simpler example: 

 In [6]: for root, dirs, files in os.walk('/Users/zintis/appjobs/'): 
...:     for file in files: 
...:         filename, extension = os.path.splitext(file) 
...:         print(f"Extension is {extension} and name is {filename} ") 
...:                                                                                                                                         
Extension is .json and name is mar30 
Extension is .json and name is apr19.1pm 
Extension is .json and name is apr15 
Extension is .json and name is apr19 
Extension is .json and name is apr18 
Extension is .json and name is apr17-905 
Extension is .json and name is apr17 
Extension is .json and name is apr17-930 
Extension is .json and name is apr16

2.6 os.plath.splitext(file) filename

The os.path.splitext(file) gets you the full path to the file, but if you only want the relative path you must first see what dir(os.path) shows you. From that, you will see that help(os.path.basename) gives you exactly what os.path.basename does, which is "return the final component of a pathname".

  • os.path.basename(file) # try it

2.7 os.listdir('mydirectory') LEGACY PYTON 2

See os.scandir() for python3 below:

Another good use of the built-in os module 



import os
entries = os.listdir('/Users/zintis/bin/python/bin')

Result is a list, so [file1, file2, file3, … lastfile] Which means to print out a human readable form: 

for entry in entries:
    print(entry)

2.8 os.scandir() PYTHON 3 and deprecates os.listdir

See os.listdir('mydirectory') LEGACY PYTON 2 

import os
with os.scandir('/Users/zintis/eg') as entries:
    for entry in entries:
        print(entry.name)

See a similar (but not so elegant) approach here: 

import os
entries=os.scandir("/Users/zintis/eg")
for i in entries:
   print(i.name)

2.9 Which is the best way to show files in a directory?

Using pathlib.Path() or os.scandir() instead of os.listdir() is the preferred way of getting a directory listing, especially when you’re working with code that needs the file type and file attribute information. pathlib.Path() offers much of the file and path handling functionality found in os and shutil, and it’s methods are more efficient than some found in these modules.

Function Description
os.listdir() Returns a list of all files and
(legacy python 2.x) folders in a directory
os.scandir() Returns an iterator of all the objects
  in a directory including file
  attribute information
pathlib.Path.iterdir() Returns an iterator of all the objects
  in a directory including file
  attribute information
glob.glob glob.glob("/Users/zintis/config.d/*.json")
   

2.9.1 shutil

From https://docs.python.org/3/library/shutil.html you perform file operations from your python program.

To copy files, use shutil.copyfile(src, dst, *, follow_symlinks=True)

2.9.2 datetime

See the script calendar-eg.py

You can also convert iso formatted dates to a python datetime object with: 

datetime import date
date.fromisoformat('2019-12-04')
datetime.date(2019, 12, 4)

And the user added modules, downloaded from github, and installed using pip install xxxmodulexxx end up here:

~/.pyenv//versions/3.6.5/lib/python3.6/site-packages Users/zintis.pyenv/versions/3.6.5/bin/python

Three simple examples of date manipulation: 














2.10 for the local libraries, including cisco's dnac





I use pip freeze to list them.  As of May 2nd, 2019 my pip freeze is:
Last login: Wed May  1 21:27:04 on ttys000
/Users/zintis[2] % pip freeze















pip freeze
pip freeze (new mbp)





pip 19.0.1???  check
pip 19.2.2






appdirs==1.4.3
 





asn1crypto==0.24.0
 





attrs==19.1.0
 






autopep8==1.4.4
autopep8==1.4.4





bcrypt==3.1.6
 





black==19.3b0
 






certifi==2018.11.29
 





cffi==1.12.2
 





chardet==3.0.4
 






ciscosparkapi==0.9.2
 





Click==7.0
 





cryptography==2.6.1
 






 
decorator 4.4.0





dnac==1.2.10.1
 





elpy==1.999
elpy==1.999





entrypoints==0.3
entrypoints 0.3






flake8==3.7.7
flake8==3.7.8





future==0.17.1
 





 
ipython==7.7.0





 
ipython-genutils==0.2.0





idna==2.5
 






jedi==0.13.3
jedi==0.15.1





lxml==4.3.3
 





mccabe==0.6.1
mccabe==0.6.1






meraki==0.34
 





ncclient==0.5.3
 





netmiko==1.4.2
 






paramiko==2.4.2
 





parso==0.4.0
parso==0.5.1





prettytable==0.7.2
 






pyang==1.7.4
 





pyasn1==0.4.5
 





pycodestyle==2.5.0
 






pycparser==2.19
 





pyflakes==2.1.1
 





PyNaCl==1.3.0
 






PyYAML==5.1
 





requests==2.18.2
 





requests-toolbelt==0.9.1
 






rope==0.14.0
 





scp==0.13.2
 





six==1.12.0
 






toml==0.10.0
 





urllib3==1.21.1
 





virtualenv==16.4.3
virtualenv==16.7.6






xmltodict==0.11.0
 





yapf==0.27.0
 





 
setuptools 41.1.0





 
wheel 0.33.4













2.11 Additional info and example





Pip install documention on pip.pypa.io





This doc.python-guide.org link has some good explanation of additional info.
Worth the read.  








2.11.1 review setup.py as a means of specifying which modules are needed





and which should be kept up to date automatically.  It turns out that there
are multiple package managers for python, no just pip.  I think I will stick
to pip but so that I am not mislead or confused and unknowingly start using
two managers and get all messed up here is a list of package managers:




    

  • distutils : part of the standard library

    • 

  • distribute : discountinued

    • 

  • distribute2 : discountinued

    • 

  • setuptools :

    • 

  • pip : Python Install Package

    • 

  • eggs : introduced 2004 and was never official PEP

    • 

  • wheels : newest official package manager available on pythonwheels
introduced in 2012

    • 






Some installs actually try to use the newer setuptools and still fall back
to the older distribute tools







try:
    from setuptools import setup
except:
    from distutils.core import setup







Recommendation is to use pip.  Although python setup.py install and pip
    install <Package-name> does the same thing, pip gives you these advantages:




    

  • pip will automatically download all dependencies for a package for you

    • 

  • pip keeps track of various metadata that lets you easily uninstall 
and update packages with a single command pip unistall <package>

    • 

  • pip lets you easily upgrade pip install --upgrade <package>

    • 

  • pip automatically downloads the files for you (setup insists you have
them pre-downloaded)

    • 

  • pip will let you install wheels

    • 













2.12 Fixing permission errors in pips.  –user option





–user to install with user permission and user libraries and NOT
the system library that need root permissions.





Here is an example of a pip install with errors, and subsequent Johnions:




/Users/zintis/bin[11] % pip install --upgrade virtualenv
Collecting virtualenv
Downloading https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e\
707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-\
any.whl (2.0MB)
| ████████████████████████████████ | 2.0MB 474kB/s |
Installing collected packages: virtualenv
Found existing installation: virtualenv 16.4.3
Uninstalling virtualenv-16.4.3:

ERROR: Could not install packages due to an EnvironmentError: [Errno 13]\
Permission denied: 'zip-safe'
Consider using the `--user` option or check the permissions.

/Users/zintis/bin[12] % pip install --upgrade --user virtualenv
Collecting virtualenv
Using cached https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3\
e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl
Installing collected packages: virtualenv
Successfully installed virtualenv-16.5.0














3 Virtual Environment -m venv





The recommended modern tool (as of Python 3.4) is to use the built-in venv
module
 .  Therefore: python -m venv venv-programming-skills will create a
virtual environment in the directory "venv-programming-skills".






For more info on deprecated virtual environments, see pyenv.org






Note: as of summer 2020, I am using the built-in venv module instead of
  virtualenv.  I used to use virtualenv for python2.7 or 3.5 but I was able to
  create a venv version using python3.7 while my brew was already at 3.9. 
  Continue reading to see how I did that:








3.1 Keeping multiple python versions with venv





If you have more than one Python version and you want to specify which one
to create the venv with, do it on the command line, like this:



    

  • python3.6 -m venv ENV_DIR where ENVDIR is the pre-existing dir you want
for your virtual environment

    • 










3.2 Upgrading python used by a particular venv





You can upgrade a venv to a new version of python, with an important
prerequiste : You must first have the new version of python somewhere on
your system before you can upgrade a venv to that version.





So, you can 



    

  • use brew to upgrade your system python, where python will be in a versions
subdirectory of /usr/local/Frameworks/Python.framework/Versions.  You would
use something like: brew update && brew upgrade python  Question: will this
actually keep the old version around?? Not

    • 

  • download python and run the setup.py install script in some subirectory.

    • 

  • download a .dmg disk image and install into the system library which will
put python in /Library/Frameworks/Python.framework/Versions

    • 





Notice that both methods seems to support multiple Versions of python.





Now, assuming you have upgraded the python referenced by python3, then you
Can upgrade the venv using this:




    

  • python3 -m venv --upgrade ENV_DIR

    • 





You can of course also choose from differenet version 3 pythons that you havve
installed in the Versions framework by using this instead:




    

  • python3.9 -m venv --upgrade ENV_DIR

    • 





To get a specific version, you may have to ensure that that version appears
in your PATH before the others, so for example in March 2021 I had change my
environment using:






export PATH="/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/bin:$PATH"







Before I could run python3.9 -m venv --upgrade venv-3.9 otherwise I was
just getting 3.9.12 as my 3.9 version and I was not upgrading anything.





Johnion that did NOT work.  Rather I simply changed the symlink in 
/Users/zintis/bin/python/venv-3.9/bin/python3 to point to
/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/bin/python3
where it was earlier pointing to 
/usr/local/Cellar/python@3.9/3.9.1_2/Frameworks/Python.framework/Versions/3.9/bin/python3





That did the trick as here is the before and after:



    

  • Before:

    (venv-3.9) /Users/zintis/bin/python/venv-3.9/bin[588]:
$ python
Python 3.9.1 (default, Dec 24 2020, 16:23:16) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

    

    • 


  • After:

    (venv-3.9) /Users/zintis/bin/python/venv-3.9/bin[592]:
$ deactivate

$ ln -s  /usr/local/Cellar/python\@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/bin/python3

$ . venv-3.9/bin/activate

$ python
Python 3.9.1 (default, Jan  8 2021, 17:17:43) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 



    

    • 






Johnion  I also had to fix python3.8 in venv-meraki as it was pointing to 



    

  • /usr/local/bin/python3.8 and that in turn was pointing to

    • 

  • /usr/local/bin/python3.8 -> ../Cellar/python@3.8/3.8.62/bin/python3.8

    • 

  • /usr/local/bin/python3.8 -> ../Cellar/python@3.8/3.8.62/bin/python3.8

    • 





I changed that to:



    

  • /usr/local/bin/python3.8 -> ../Cellar/python@3.8/3.8.81/bin/python3.8

    • 

  • /usr/local/bin/python3.8 -> ../Cellar/python@3.8/3.8.81/bin/python3.8

    • 





Using the command: ln -s ../Cellar/python\@3.8/3.8.8_1/bin/python3.8 python3.8







See Mystery solved section below











3.3 Fix for multi-version python venvs





As of January 30, 2021 I came across this issue:



    

  • python3 -m virtualenv aci-learning-labs --python=python3.7

    • 




This works if you have 3.7 installed.  I had 3.9 already, so the above command
did NOT work.  Here's what I did to fix this, i.e. install an older 3.7 python
for the purposes of creating a virtualenv that ran 3.7.





Maybe I should have tried this?:



virtualenv --python=C:/python27/bin/python2.7 /path/to/new/virtualenv/
/path/to/new/virtualenv/Scripts/activate.bat




I have never tried this….   but wanted to record this as my problem might
have simply been not having python3.7 in my $PATH ?   









3.3.1 First identifiy the problem.





As of python 3.5 or 3.6, venv does NOT support creating a virtual environment
with a different python version.  So, if you have 3.9, then venv environments
you create will be 3.9





So I was trying python3 -m virtualenv aci-learning-labs --python=python3.7
and getting this error:





RuntimeError: failed to find interpreter for Builtin discover of pythonspec='python3.7'










3.3.2 Second, download and install python3.7





I went to https://www.python.org/downloads/release/python-379/ , downloaded
the file: https://www.python.org/ftp/python/3.7.9/python-3.7.9-macosx10.9.pkg










3.3.3 Third, find where python3.7.9 was installed.





It turns out it was in /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7










3.3.4 Four, upgrade 3.7.9 pip version




    

  • cd /Library/Frameworks/Python.framework/Versions/3.7/bin 

    • 

  • ./python3.7 -m pip install --upgrade pip

    • 










3.3.5 Five, create the 3.7 virtualenv




    

  • ./python3.7 -m venv ~~/bin/python/venv-aci

    • 





That did it!  Notice that I did not run virtualenv to create the virtual
environment, but rather, the usual -m venv but using the 3.7 binary.





One final note.  The /Library/Frameworks/Python.framework/Versions directory
only had the one version, 3.7.  That begs the question; where is my 3.9.1
version?










3.3.6 Mystery solved





Well, not much of a mystery, but good to be aware of my $PATH I had to
install 3.7 as a .dmg download from the above link.  which then installed it
in my /Libarary/Frameworks/Python...  When I use brew, it does the right
thing by installing in /usr/local/Frameworks/Python.framework/Versions
















4 Setting up Virtual Environments on CentOS8





pip3.8 install virtualenv  On my CentOS8 host I also used the built-in 
python module
, venv as such: python3 -m venv venv-ansible








4.1 Cisco DevNet approach





Based on instructions back in 2018, they suggested using virtualenv like so:



    

  • virtualenv Cisco-devnet-dnac-zp --python=python3

    • 

  • source Cisco-devnet-dnac-zp/bin/activate

    • 










4.2 Once Virtual Environment is up





Once you have installed and activated your virtual enviroment, everything else is
the same.  So, you would have to pip install the modules you want, you would have to check
what is installed with pip freeze,  etc..
for example:



    

  • pip install pyang

    • 

  • pip install -r requirements.txt

    • 





By default, pip should only have 2 or 3 packages in the virgin virtual env.
Mine had:



    

  • pip 19.1

    • 

  • setuptools 41.0.1

    • 

  • wheel 0.33.1

    • 





So go ahead and install the packages you need for this environment:



    

  • pip install numpy

    • 

  • pip install pytz

    • 

  • pip install pprint

    • 

  • pip install requests

    • 





To save the list of packages in this environment as a list of requirements.txt
you would first 



    

  • pip list,  then run 

    • 

  • pip freeze --local > requirements.txt

    • 


  • ls 

    • 

  • less requirements.txt  # to confirm that the list is there.

    • 


  • deactivate  # (to close this virtual environment.)

    • 


  • pip list # will now show your global environment as before.

    • 










4.3 Recreating the Virtual Environment (requirements.txt)





With this requirements.txt file you now still have, you can recreate the
virtual environment by running an new virtual environment, and 
pip installing with this file:
pip install -r requirements.txt










4.4 Pip knows to not duplicate everything





In fact, I noticed that after creating a virtual envirment, that I called
venv-prg-basics 
pip had created the sub directories as expected, but only created symbolic links
to the existing environment. 




...
lrwxr-xr-x  1 zintis  staff  58  2 May 15:07 token.py -> /Users/zintis/.pyenv/versions/3.6.5/lib/python3.6/token.py
lrwxr-xr-x  1 zintis  staff  61  2 May 15:07 tokenize.py -> /Users/zintis/.pyenv/versions/3.6.5/lib/python3.6/tokenize.py
lrwxr-xr-x  1 zintis  staff  58  2 May 15:07 types.py -> /Users/zintis/.pyenv/versions/3.6.5/lib/python3.6/types.py
lrwxr-xr-x  1 zintis  staff  61  2 May 15:07 warnings.py -> /Users/zintis/.pyenv/versions/3.6.5/lib/python3.6/warnings.py
lrwxr-xr-x  1 zintis  staff  60  2 May 15:07 weakref.py -> /Users/zintis/.pyenv/versions/3.6.5/lib/python3.6/weakref.py
/Users/zintis/bin/venv-prog-basics/lib/python3.6[19] % 





A very good video that explains virtual environement is here in youtube.
Another one is realpython.com  










4.5 Removing a virtual environment with rm





To clean up permanently, after deactivating the virtual environment, remove the whole 
folder structure.  For example:



    

  • ls

    • 

  • rm -rf venv-project-blue

    • 

  • ls 

    • 














5 IDE (Interactive Development Environments)





I prefer emacs, with a couple of python packages installed from melpa such as:



    

  • elpy

    • 

  • pyvenv

    • 





Most good modern IDEs come with (or can be extended with) syntax highlighting, 
code completion, static code analysis, debugging, variable exploring.  They 
also should let you save and reload files, run code from the environment, and
do automatic code formatting according to PEP8 conventions.








5.1 List of most commom IDEs:




    

  • IDLE (comes standard with python and accessed with python3 -m idlelib.idle

    • 

  • Spyder
Light-weigt open-source IDE included in Anaconda, and meant for data sciences
So, includes NumPy, SciPy, Matplotlib, Pandas, IPython
Very good help toolbar.

    • 

  • PyCharm
deFacto standard
Good if you have multiple scripts interacting with each other
Supports Anaconda (and all the data science packages in Anaconda)
Supports web dev frameworks such as Django, Flask, Node.js, web2py HTML/CSS 

    • 

  • eric

    • 

  • Vim

    • 

  • Atom

    • 

  • Jupyter Notebook 
(Julia + Python + R) which are open-source languages for data science
As such this is a good choice for data analysis tools, like matplotlib and scipy
Jupyter's predecessor was iPython…  interesting.
Anaconda is strongly recommended if you want to use Jupyter
Understands LaTex

    • 

  • Anaconda
Includes Python, Jupyter Notebook, and common scientific and data 
  science packages

    • 

  • PyDev

    • 

  • Thonny (Python IDE for beginners)

    • 

  • VSCode (Visual Studio Code)
For Microsoft / Windows environments (also Mac now)

    • 













6 Colouring Print Output and other print options






6.1 Print statement options, sep=" ", and end="\n"





The print statement has two optional parameters, sep and end.




    

  • sep
Default is a space, i.e. sep=" " which is the separator character(s)

    • 


  • end
Default is newline, i.e end="\n" which specifies how each line should end.
So double-spaced output simply set end = "\n\n"

    • 











6.2 Python output colouring in print:





The escape codes are entered right into the print statement.







print("\033[1;32;40m Bright Green  \n")







The above ANSI escape code will set the text colour to bright green. The format is;




    

  • \033[  Escape code, this is always the same

    • 


  • 1 = Style, 1 for normal.

    • 


  • 32 = Text colour, 32 for bright green.

    • 


  • 40m = Background colour, 40 is for black.

    • 






This table shows some of the available formats;




TEXT COLOR CODE	 TEXT STYLE  CODE      BACKGROUND COLOR	CODE
Black           30    No effect   0         Black      40
Red             31    Bold        1         Red        41
Green           32    Underline   2         Green      42
Yellow          33    Negative1   3         Yellow     43
Blue            34    Negative2   5         Blue       44
Purple          35                          Purple     45
Cyan            36                          Cyan       46
White	   37			       White      47 
-----
colours are RGB  so #ff0000 is red, #00ff00 is green, and #0000ff is blue
#ff00ff is purple





To reset the colour to what it is normally use \033[1;32;40m










6.3 Auto Colouring Python Code in idlelib





Changing colour themes to a custom theme by editing config-highlight.def in
Users/zintis.pyenv/versions/3.6.5/lib/python3.6/idlelib/
idlelib should exist in non-pyenv setups too.  Just have to figure out where.





[IDLE emacsDark]
comment-foreground = #DD0000
console-foreground = #EE4D4D
error-foreground = #FFFFFF
hilite-background = #7E7E7E
string-foreground = #CC968D
stderr-background = #171717
stderr-foreground = #FFB3B3
console-background = #171717  
hit-background = #EFEFEF
string-background = #171717
normal-background = #171717
hilite-foreground = #FFFFFF
keyword-foreground = #FF8000
error-background = #C86464
keyword-background = #171717
builtin-background = #171717
break-background = #808000
builtin-foreground = #B800FF
definition-foreground = #5E5EFF
stdout-foreground = #00D0DC
definition-background = #171717
normal-foreground = #0FD400
cursor-foreground = #FFD400
stdout-background = #171717
hit-foreground = #005090
comment-background = #171717	
break-foreground = #FFFF00






See the "RBG-emacs-theme-for-python.pages" document











6.4 Python Script for Decimal to Hex Conversion





I wrote a simple conversion to hex script to get to the colours I wanted.  I used
pages and the 0-255 RGB values for coloured fonts, and then converted them to hex
for use in the above theme.  Note that this is NOT the same codes used in 
Python print colouring.







#!/usr/bin/env python
'''Convert a decimal number to a hex equivalent.

Requires python 3.x


'''

print("\033[1;36;40m This script converts decimal numbers to  hex numbers.  0 to stop:\033[1;33;40m \n")

while True:
    try: 
        decimalnum = int(input("\033[1A  Your decimal number: "))

    except (IndexError, ValueError) as e:
        print("whooa, touchy keyboard... try again.\n\n")
        continue

    print(" Hexidecimal equivalent: ", hex(decimalnum), "\n\n")

    if decimalnum == 0:
        print('\033[1;32;40m This was simply "hex(interger-input)"\n Thanks.\n\n')
        break

    # could also do:
    # int("0xff",  16)
    # int("FFFF",  16)
    # ast.literal_eval('0xdeadbeef')
    # 3735928559 is the result
    #






There is of course the f-string output conversions that can be used if you are
only looking at outputting values in different bases.  See: f string number conversions












7 Importing






7.1 Overview





import does exactly that, but it also executes the code exactly once.
if you want to execute the code again, you reload (import had to come before reload)
You need to get "reload" from the imp standard library module









so:







from imp import reload
reload(script1)







The from command simply copies a name out of a module
 .





import is a statement
reload is a function (hence the need for parenthesis)










7.2 Reload imported modules





As of 3.5 reload is part of imp so you can:






import imp
# and then :
imp.reload(M)

# or,...
from imp import reload
# and then use:
reload(M)






    

  • names loaded with a from are NOT directly updated by a reload.

    • 

  • names accessed with an import statement ARE.

    • 





If your names don't seem to change after a reload, try using import and then
module.attribute name references instead.










7.3 Import vs from module import func






7.3.1 import





You can import the entire module , or be more resource conscious and import
just the functionality (i.e. methods) you need.  It also makes it easier to
read your code, as every sub-method that you call will need to be specified
in the dot notation when referenced.





For example






import os
x = os.path.abspath()







compared to






from os.path import abspath 
x = abspath()







or even






from os.path import abspath as ap
x = ap()















7.4 Detailed example on Import





Here is a simple example that illustrates using a simple module that assigns
three variables, a, b, and c.







threewords.py
'''set three strings and print them. '''

a = 'dead'
b = 'parrot'
c = 'sketch'
print("."*30,"\n", a, b, c, " ===> printed from threewords.py")







If I was to run this module by calling python like this:



    

  • python threenames.py

    • 





Output is:




$ python threenames.py
.............................. 
dead parrot sketch
$





Now if I write a python program that imports the threenames.py module:






#!/usr/bin/env python
''' call three words to illustrate how modules overwrite variables

So you must be careful
'''

a = 'first'
b = 'second'

import threewords

print(a, b)
print(threewords.a, threewords.b, threewords.c, "\n", "_"*30, "\n")
print(dir(threewords))








You will get this output:




.............................. 
dead parrot sketch ===> printed from threewords.py
first second
dead parrot sketch 
 ______________________________ 

['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'a', 'b', 'c']





The first 2 lines are from the import statement that executes all lines of the
called module when imported.   Lines 3,4,5, and 6 are from the 3 print
statements in the calling program.





Finally this next example illustrates the from import import module line
and how variables can be overwritten when done this way.  Just be careful.







    #!/usr/bin/env python
    ''' call three words to illustrate how modules overwrite variables

       This is one of two programs that both call the threewords.py module.
       This program used the from module import method(or var) construct

    '''

    a = 'first'
    b = 'second'

    from threewords import a, b, c

    print(a, b, c, "\n", "_"*30, "\n")

    print("\n  Expect to get an error that 'threewords' is not defined.")
    print("only a, b, and c are defined now. \n")
    print(dir(threewords))
qui







Gives this output:



.............................. 
dead parrot sketch
dead parrot sketch 
 ______________________________ 


  Expect to get an NameError that 'threewords' is not defined.
only a, b, and c are defined now. 

Traceback (most recent call last):
  File "/Users/zintis/bin/python/bin/./three-words-call-from.py", line 17, in <module>
    print(threewords.a, threewords.b, threewords.c, "\n", "_"*30, "\n")
NameError: name 'threewords' is not defined





Notice that the variables a, b, from this program were overwritten.  So using
"from" clobbers  the variables a,b,c, in the current module.  Wile using
"import" leaves a,b, as they were in the current module.











7.5 Miscellaneous example on splitting input into 2 variables:





good example on reading a file, line by line, and splitting each word delimited by "," into separate values
For example if input file is:



995957,16833579
995959,16777241
995960,16829368
995961,50431654





and you want each line to be:






x=995957
y=16833579







Then:






a = "123,456"
b = a.split(",")   # b will be a list of strings
print(b)
#  ['123', '456']

# converting the list of strings to a list of integers
c = [int(e) for e in b]
print(c)
#  [123, 456]

x, y = c
print(x)
#  123
print(y)
#  456
















8 Everything is an object!!





In python, everthing is an object.  Objects have attributes (variables)
and objects have methods (functions).  There are accessed through python's
dot.syntax  





>>> a = 57
… a.bitlength()
6
>>> dir(a)   # gives you all the methods available
>>> "I AM a Walrus".lower()
'i am a walrus'











9 Object Types





Excerpt From: Mark Lutz. “Learning Python.” iBooks. 





“ Every object in Python is classified as either immutable (unchangeable) or
not. In terms of the core types, numbers, [2], and tuples are immutable;
lists, dictionaries, and sets are not—they can be changed in place freely,
as can most new objects you’ll code with classes. This distinction turns out
to be crucial in Python work”















Immutable
Example literals/creation





Object Types
 






Numbers
1234, 3.1415, 3 + 4 j, 0, Decimal(), Fraction()





 
these are types int, float, complex, boolean






Strings
'spam', "Bob's", b'a\x01c', u'sp\xc4m'





 
these are type str






Tuples
(1, 'spam', 4, 'U'), tuple('spam'), namedtuple





 
these can contain any types











9.1 Immutable objects cannot be changed by definition







S = 'spam'
S[0] = 'z'  







This code produces this TypeError:



TypeError: 'str' object does not support item assignment






But we can create a new object:







S = 'spam'
S = 'z' + S[1:]        # But we can run expressions to make new objects
print(S)               # will print 'zpam\342\200\235







Note that for readability, python ignores _ in numbers.  So rather than write
trillion = 1000000000000 we can write trillion = 1_000_000_000_000
















Muttable
Example literals/creation





Object Types
 






Lists
[1, [2, 'three'], 4.5], list(range(10))





Dictionaries
{'food': 'spam', 'taste': 'yum'}, dict(hours=10)





Sets
set('abc'), {'a', 'b', 'c', 3.14, 100}





Files
open('eggs.txt'), open(r'C:\ham.bin', 'wb')



















Other Object Types
Example literals/creation






Other core types
Booleans, types, None





Program unit types
Functions, modules, classes





Implementation-related types
Compiled code, stack tracebacks









From youtube
  lists-tuples-sets-dictionaries.png














10 Sequences ([2], lists, and tuples)





Sequences include [2], lists, and tuples.
With sequences you can do indexing, slicing, adding/concatentating,
multiplying, checking membership, iterating, length, minimum,
maximum, summing, sorting, and counting  Each of which 
is explained below:








10.1 Indexing - use square brackets [ ]





One can access any item by its index, starting from zero.





For example if your list was:






l = ['a', 1, 18.2]







You can then access and update the elements within the list via the elements
index, and like all good programming languages - indexes start at 0. You can
also append items to a list with the .append() method.







print([2])     # prints  18.2

# assign a new value to an element
l[2] = 20.4
print(l)     # prints  ['a', 1, 20.4]

# append to a list
l.append("new")
print(l)     # prints  ['a', 1, 20.4, 'new']







There are several other "batteries included" methods natively available with
lists - learn more in the Python docs, or help(l) (usin the above example
where l was defined as a list.








cast = ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam']
print(cast[2]) # prints Chapman

mystring = cast[0]
print(mystring[2]) # prints e which is the third letter in the string 'Cleese'

math_constants = (3.14159, 2.71828, 4.66920)
print(math_constants[2])  # prints 4.66920












10.2 Indexing sequence1.index(item)





Returns the first occurance of item in the sequence (there may be more but 
they are ignored)







10.2.1 Strings







x = 'Mississippi'
print(x.index('s'))  # gives you 2
print(x.index('p'))  # gives you 8












10.2.2 Lists







print(cast.index('Cleese')) # gives you 0
print(cast.index('Gilliam')) # gives you 5















10.3 Slicing [start: end+1: step]





You can slice out sub[2], sublists, subtuples using [start: end+1: step]
The defaults are [first: last: 1] i.e. from first to last and 1 at a time
The defaults kick in if you leave the entry  blank so: [::] would be the
the whole string.







mystring = "Cleese is a founding member of Monty Python's Flying Circus"
print(mystring[1:3])   # = le  items 1 to 2 in Cleese
print(mystring[0:5:2]) # = Ces items 0 3 and 5
print(mystring[55:])   # = ircus   items 3 to the end
print(mystring[0:5])   # = Clees items from beginning up to 4
print(mystring[-1])    # = e      the last item
print(mystring[-6:])   # = Circus   the last six items
print(mystring[:-2])   # = Cleese is a founding member of Monty Pythong's Flying Circu"
                # all but the last two items
print(mystring[::])    # all defaults so "Cleese is a founding member of Monty Python's Flying Circus"
print(mystring[::-1])  # from first to last, but backwards one at a time, so reverses
print(mystring[::-1])  # "sucriC gniylF s'nohtyP ytnoM fo rebmem gnidnuof a si eseelC"

# this last one can be used to check if a string is a pallindrom. i.e.
name = "hannah"
print(name == name[::-1])   # will print "True"
name = "Hannah"
print(name == name[::-1])   # will print "False"
print(name.lower() == name.lower()[::-1])  # will print "True"













10.4 string.replace()  (very similar to slicing)





replace is a built-in python string function.






x = string.replace(old, new, count)






returns a copy of the string after substitutions.





So if 







script = 'This parrot is dead'
script.replace('parrot', 'ferret') 






would be 'This ferret is dead'












10.5 substrings





There is a very good analysis of various methods of extracting substrings from
long strings at geeksforgeeks.com/python-substrings





Summarizing the analysis they did on these five methods and their computation
complexity I have come up with this table.  Given n is the length of the
original string and m is the length of the substring you are searching for in
the string, and k is the number of instances of the substring in the original
string

















Method
Time
Memory





 
Complexity
Used






startswith
O(n*m)
O(k)






re.finditer
O(n)
O(n+m)






find()
O(n*m)
O(k)






string slicing
 
 





in while loop
O(n*m)
O(k)






re.finditer()
 
 





with reduce()
O(n)
O(m)





 
 
 










From that analysis, it seems that the last table entry is the simplest and most
efficient (full discussion at the web site).  It uses the regular expression
module re as well as the finditer module available in functools  Remember to
import them.




    

  • re.finditier() is used to find all occurences of the substring

    • 

  • reduce is used to get the start indices of all the occurences found in the
re.finditier() step.

    • 







import re
from functools import reduce

# initializing string
test_str = "GeeksforGeeks is best for Geeks"

# initializing substring
test_sub = "Geeks"

# using re.finditer() to find all occurrences of substring in string
occurrences = re.finditer(test_sub, test_str)

# using reduce() to get start indices of all occurrences
res = reduce(lambda x, y: x + [y.start()], occurrences, [])

# printing result
print("The start indices of the substrings are : " + str(res))
#This code is contributed by Jyothi pinjala


















10.6 Adding/Concatenating (of the same type)






10.6.1 Strings







x = 'John' + 'Cleese'   






gives you 'JohnCleese'










10.6.2 Lists







x = cast + ['Cleveland', 'Booth']






gives you x = ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam', 'Cleveland', 'Booth]













10.7 Multiplying






10.7.1 Strings





print('Grail ' 3)  # prints Grail Grail Grail









10.7.2 Lists





x = [1, 8] * 4
print(x)  # prints [1, 8, 1, 8, 1, 8, 1, 8]









10.7.3 numbers





See the section: numeric operators












10.8 Checking Membership "in" or "not in"






10.8.1 Strings





x = "Monty"
print('u' in x)  # prints False
print('M' in x)  # prints True










10.8.2 Lists (checking membership in or not in)







cast = ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam']
print('Atlas' not in cast)  # prints True
print('Cleese' in cast) # prints True
print('Palin' not in cast) # prints False

alsocast = ['Cleveland', 'Booth']
fullcast = cast + alsocast
# fullcast is now ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam', 'Cleveland', 'Booth']
cast += alsocast
#      cast is now ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam', 'Cleveland', 'Booth']















10.9 use of all





from help(all)








mustcontain = ['x', 'y', 'z']
ingred1 = ['a', 'b', 'x', 'q', 'z', 'y']
ingred2 = ["a","b","x","y"]

all(element in ingred1 for element in mustcontain)
# returns True


all(element in ingred2 for element in mustcontain)
# returns False

ingred2.append('z')


all(element in ingred2 for element in mustcontain)
# returns True

"x" in ingred1
# returns True













10.10 Iterating






10.10.1 Items





Iterate through itmes in a sequence using a for loop.







for actor in cast:
        print(actor)







would result in:



Cleese
Palin
Chapman
Jones
Idle
Gilliam





And, 






x = [2, 4, 6]
for num in x:
    print(num**2)







would result in:



4
16
36










10.10.2 Items and Index





To get the the index as well, you use enummerate







x = [2, 4, 6]
for index, num in enumerate(x):
    print(index, num**2)






would result in:



0 4
1 16
2 36












10.11 Length len(sequence1)







cast = ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam'] 









10.11.1 Strings







len(cast[2])   # the length of the string "Chapman" which will return 7 












10.11.2 Lists







len(cast)  # the number of cast members which will return 6
















10.12 Minimum min(sequence1)





Minimums can be numeric minimums or lexicographical minimim as long as all
items in the sequence are of the same type.  intergers can be compared to floats.
So min([2, 4, 6]) returns 2
min(cast) returns "Chapman" which is the first in alphabetical order











10.13 Maximum max(sequence1)





Obvious counter to min above.










10.14 Summing sum(sequence1)





Must be of a numberic type, so float and interger






x = [2, 4, 6, 8, 10]
sum(x) returns 30
sum(x[-2:]) # returns 18,  the sum of the last two items in the list













10.15 Sorting sorted(sequence1[1:3]])





It returns a new list in sorted order, leaving the original list unchanged.







10.15.1 Strings







print(cast[2]) # gives you 'Chapman'
print(sorted(cast[2]) #  gives you ['C', 'a', 'a', 'h', 'm', 'n', 'p']











10.15.2 Lists







print(sorted(cast)) prints
['Chapman', 'Cleese', 'Gilliam', 'Idle', 'Jones', 'Palin']
and cast is still ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam']













10.16 Sorting sort(list)





Unlike "sorted" that leaves the original list unsorted, sort will actually
change the original list into a sorted version.











10.17 Counting sequence1.count(item)






10.17.1 Strings







x = 'Mississippi'
print(x.count('i'))  # gives you 4 











10.17.2 Lists







print(cast.count('Jones')) # gives you 1















10.18 Constructors





These can create a new list






x = list()
x = ['Cleese', 'Palin', 3.14, -5]
x = list(tuple1)












10.19 List Comprehensions (as constructors)







x = [m for m in range(11)]
# resuts in x = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

x = [m**2 for m in range(11) if m**2 % 3 == 0]
# results in [0, 9, 36, 81] as %3 is modulo 

# interestingly
x = [m**2 for m in range(11) if m2 % 3 == 0]
# results in [0, 9, 36, 81] as well.







An excellent example of how list comprehensions can reduce code and make it 
easier to read is the following example:







def find_employees(self, role: Role) -> list[Employee]:
    ''' Find all employees with a particular role '''
    employees = []
    for employee in self.employees:   
        if employee.role == role:
            employees.append(employee)
    return employees







Can be written as such, using list comprehensions:






def find_employees(self, role: Role) -> list[Employee]:
    ''' Find all employees with a particular role.'''
    employees = [employee for employee in self.employees if employee.role is role]
    return employees      







You can actually directly return employees like so:






def find_employees(self, role: Role) -> list[Employee]:
    ''' Find all employees with a particular role.'''
    return [employee for employee in self.employees if employee.role is role]










10.19.1 Quirk of True, True, True == (True, True, True)





Python comprehensions can let you write nice things like:






# assignments (one equal sign)
x, y, z = "red", "blue", "green"
# is the same as:
x = "red"
y = "blue"
z = "green"








But in interactive python:



# comparisons (two equal signs)
>>> True, True, True 
(True, True, True) 
>>> True, True, True == (True, True, True)
(True, True, False)





That seems confusing, but if you change that to:



# comparisons (two equal signs)
>>> q = True, True, True 
>>> q
(True, True, True) 
>>> q == (True, True, True)
True
>>> True, True, True == (True, True True)
(True, True, False)





That makes more sense.  Really what is happening is explained by:



True, True, (True == (True, True, True))
# which is obviously => True, True, False.





Because the first & second true, evaluate to True, the third true is not the
boolean True, but a tuple so you are comparing a boolean to a tuple, which
is obviously false.












10.20 Adding to lists (append(), insert(), extend())




    

  1. append() add the element to the end of the list (as one element)

    2. 

  2. insert()

    3. 

  3. extend() adds each element to the end of the list as individual elements


    
Given a list of 30 elements.
If the object you are adding to the end of your list is itself a  list
of five elements, append() will make a list of 31 elements (the last entry
is a list of 5 elements), where  extend() will make a list of 35 elements.

    

    4. 






  1. Examples
    

    


    mylist = ['single element']
yourlist = ['two', 'three', 'four']
mylist.append(yourlist)

    

    

    
results in mylist being ['single element', ['two', 'three', 'four']] 
i.e. a list of two elements, the second of which is a list of 3 elements.

    


    
if it was

    

    


    mylist.extend(yourlist)

    

    


    
then the result would be 
['single element', 'two', 'three', 'four']:   a list of four elements.

    


    
Be careful of adding a string to the list.  The correct way is:

    

    


    mylist = ['single element']
lookingfor = 'Fluffy'
mylist += lookingfor
print(mylist)
# results in ['single element', 'F', 'l', 'u', 'f', 'f', 'y']

    

    


    
The incorrect way is

    

    


    mylist = ['single element']
lookingfor = 'Fluffy'
mylist.append(lookingfor)
print(mylist)
# results in ['single element', 'Fluffy']


    

    

    










10.21 Removing deleting from a list





The remove() method removes the first matching element (which is passed as an
argument) from the list.







# create a list
cast = ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam']
# remove Chapman from the list
cast.remove('Chapman')








But can I remove the third element, just based on it being the third element?
Yes you can.  You remove it by the list's index.







cast = ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam']
# remove the second (starting from zero) element of the list
cast.pop(2)
















10.22 Finding duplicates in a list





Sometimes a good example make words superfluous.






a = [[1], [2], [3], [1], [5], [3]]

no_dupes = [x for n, x in enumerate(a) if x not in a[:n]]
print no_dupes # [[1], [2], [3], [5]]

dupes = [x for n, x in enumerate(a) if x in a[:n]]
print dupes # [[1], [3]]













10.23 not getting an IndexError on lists





Sometimes you don't know how long a list will be, for instance when you split
all the words of a line into seperate words.  





You can easily get an IndexError if you try referring to the second word when
in fact the line has only 1 word or zero words.  listofwords = line.split()





A common fix is to use range(len(listofwords)).






x=[1,2,3,0,0,1]
for i in range(0,len(x)):
   if x[i]==0:
       x.pop(i)







Or simply for i in range(len(mylist))










10.24 numeric operators




    

  • + addition

    • 

  • - subtraction

    • 

  • * multiplication

    • 

  • / division

    • 

  • // floor division

    • 

  • % modulus 

    • 

  • ** exponentiation

    • 










10.25 string operators




    

  • + concatenation

    • 

  • * multiplication

    • 










10.26 comparison operators




    

  • < less than

    • 

  • > greater than

    • 

  • <= less than or equal to

    • 

  • >= greater than or equal to

    • 

  • == equal to

    • 

  • != not equal to

    • 

  • in contains element

    • 





and combine that with and and or and not for full flexibility







x = 27
if x > 10 and x <= 27:
    print (x, " is greater than 10 and less than or equal to 27 ")













10.27 Looping through lists in a for loop







eachline = "-rwxrwxrwx@  1 zintis  staff  33751120 29 Jul  2016 2018-7-29 05:52 DF doe-following**.AVI"
eachword = eachline.split(" ")
print(eachword)
for word in eachword:
    print(f"{word:>45}")
print(eachword[5])
print(eachword[8])








Results in output:



['-rwxrwxrwx@', '', '1', 'zintis', '', 'staff', '', '33751120', '29', 'Jul', '', '2016', '2018-7-29', '05:52', 'DF', 'doe-following**.AVI']
                                  -rwxrwxrwx@
                                             
                                            1
                                       zintis
                                             
                                        staff
                                             
                                     33751120
                                           29
                                          Jul
                                             
                                         2016
                                    2018-7-29
                                        05:52
                                           DF
                          doe-following**.AVI
staff
29














11 Regular expressions:





You can use the re module, so import re to then search for matches
in [2] using regular expressions.  Remember that whenever you write regular
expressions in your code, you should ALWAYS COMMENT what the regex is
intending to do.  This makes you code much more readable, as regex can be
somewhat cryptic.





for example:  code that reads input from dice rolls may need a pattern that
only accepts digits from 1 to 6, and no letters.  Here is a possiblity:






dicepattern = re.compile('[^a-zA-Z][1-6]{4,5}[^a-zA-Z]')

print(dicepattern.match("123456fghijklmnop"))
# results in: <re.Match object; span=(0, 6), match='123456'>

print(dicepattern.match("ghijklmnop"))
# results in:  None








So when there is no match, the .match function returns false (None)





Here it is in an if statement:







if dicepattern.match("123456"):
    print(' pattern is ok')
else:
    print('no match')









The most common re methods are :



    

  • match = re.search(pattern, string)

    • 

  • match = re.search(pattern = "[0-9]+", string = lines)

    • 

  • match = re.findall(pattern, string)

    • 










11.1 regex patterns in python





Characters just match themselves, unless they are special characters in which
case, they must be \ escaped first.





Special characters 














characters <c>
meaning <l>






.
any single character, but not newline \n






\w
any single letter, [a-zA-Z0-9_]





 
that would be in a "word"






\b
boundary between word and non-word






\s
any single whitespace character





 
space, newline, return, tab, form [\n\r\t\f]






§
any single non-whitespace character






\t\n\r
tab, newline, return






\d
any single decimal digit [0-9]






^, $
the start or end of a line






\
escape the special meaning of the following





 
character





 
 






 
RANGES






[abc]
match a single a, or b, or c






[a-z]
match a single character between a to z






[a-zA-Z]
same, but include any capital letters






[0-9]
match any single digit






[02468]
match any single even digit






[0-9]
match any single character that is NOT a digit






 
OCCURRENCES






*
0 or more occurrences of the pattern to the left





+
1 or more occurrences of the pattern to the left





?
0 or 1 occurrence of the pattern to the left





{n}
match exactly n occurrences





{n,}
match at least n occurrences





{m,n}
match between m and n times





 
 






 
EXAMPLES






\w+
matches 1 or more word characters





\w*
matches 0 or more word characters





\w?
matches 0 or 1 word characters





\w{5}
matches any 5 letter word





\w{5,}
matches any word with at least 5 letters





\w{5,8}
matches words between 5 and 8 characters long





 
 








You can also use repetition in your matching.










11.2 Match 0 or more times, e.g. \w* means match 0 or more word characters




    

  • + Match 1 or more times, e.g. \w+ means match 1 or more word characters

    • 

  • ? Match 0 or 1 times, e.g. \w? means match 0 or 1 word characters

    • 

  • {n} Match exactly n times, e.g. \w{3} means match exactly 3 word characters

    • 

  • {n,} Match at least n times, e.g. \w{5,} means match at least 5 word characters

    • 

  • {m,n} Match between m and n times, e.g. \w{5,7} means match 5-7 word characters

    • 




We can use this to find all lines that contain words with 10-12 characters, by typing;





The search proceeds from the start of the string, to the end, and will stop
at the first match found.  All of the pattern must match, but not all of the
string.





Python regex + and * are greedy.  They will try to match as much of the
string as possible.





Python regex matches the left-most part of the string first.








11.2.1 re.search(pattern, string)





First an example, using just a docstring as your lines.











import re

lines = '''
netstat -tulpn
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -                   
tcp        0      0 192.168.111.1:53        0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::5355                 :::*                    LISTEN      -                   
tcp6       0      0 :::111                  :::*                    LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
tcp6       0      0 ::1:631                 :::*                    LISTEN      -                   
udp        0      0 0.0.0.0:56649           0.0.0.0:*                           -                   
udp        0      0 192.168.111.1:53        0.0.0.0:*                           -                   
udp        0      0 127.0.0.53:53           0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:67              0.0.0.0:*                           -                   
udp        0      0 192.168.128.76:68       0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:111             0.0.0.0:*                           -                   
udp        0      0 127.0.0.1:323           0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:5355            0.0.0.0:*                           -                   
udp6       0      0 :::111                  :::*                                -                   
udp6       0      0 ::1:323                 :::*                                -                   
udp6       0      0 :::50164                :::*                                -                   
udp6       0      0 :::5353                 :::*                                -                   
udp6       0      0 :::5355                 :::*                                -                   
zintis@c8host ~ [1040]$
'''
first_match = re.search(pattern = "\w*tcp\w*",
                  string = lines)

# alternatively
first_match = re.search(r"dream", line)  
first_match = re.search(r"dream", line, re.ignorecase)  
# the r means that the string "dream" is a raw string which should not be escaped.













11.2.2 match object on re.search(pattern, string)





So, this will ONLY return the first match it found in the lines.  But if
you were to print first_match you would NOT see the line, but rather a 
match object  i.e. <re.Match object; span=(309, 312), match='tcp'>





This just tells you the characters matched were in position  309 to 312.
and it matched the string "tcp"





If no matching pattern was found, re.search(pattern, string) returns None










11.2.3 re.group()





So, if the index isn't useful, but you'd rather see what was matched, use
the group() method:






print(first_match.group())






Using this print statement, the following table shows what each re
expression would print.















pattern in firstmatch = \
output of firstmatch.group()





re.search(pattern, lines)
 






tcp
tcp






.*tcp
tcp






.*STE
tcp    0    0 0.0.0.0:5355    0.0.0.0:*   LISTE






\w*STE\w*
LISTEN






".*ST\w*"
tcp   0     0 0.0.0.0:5355    0.0.0.0:*   LISTEN',






 
 








Note: you can provide defaults, so instead of just



    

  • first_match = re.search(".STE", lines)

    • 




You could have



    

  • first_match = re.search(pattern = ".STE", string = lines)

    • 












11.2.4 recalling indexes .start() and .end()





Rarely you would want the indexes, but if you did, they are:






print(first_match.start())
print(first_match.end())






Would result in 309 and 312 being printed each on a line.










11.2.5 re.findall





Matching all occurances of the pattern you can use findall that will
let you write all of the matched strings into a list like this:







all_matched_strings_in_a_list = re.findall(pattern = "tcp.*", string = lines)






Notice that this would result in the "tcp" followed by any character (.) any
number of times (*).  So, all characters to the end of line.






To find just whole words with "tcp" inside the word, but limit it to words,
the regex should be: "\w*tcp\w*"  Which is :



    

  • \w* zero or more word characters

    • 

  • tcp the letters "tcp"

    • 

  • \w* zero or more word characters  (stops at the first whitespace character)

    • 









import re
p = re.compile('[A-Za-z0-9]+\-[A-Za-z0-9]+\-[A-Za-z0-9]+\-[A-Za-z0-9]+') 
# p is the regular expression pattern
x = '83b62e00-5395-4b7a-a4b4-252f63bc6bd6'
print(p.match(x))  
<re.Match object; span=(0, 23), match='83b62e00-5395-4b7a-a4b4'>







If the pattern does not match, you would get 'none' returned.





More common is to query a match in an if statement:






p = re.compile('[A-Za-z0-9]+\-[A-Za-z0-9]+\-[A-Za-z0-9]+\-[A-Za-z0-9]+') 
x = '83b62e00-5395-4b7a-a4b4-252f63bc6bd6'
m = p.match(x)

if m:
    print ('Match found : ', m.group())
else:
    print ('no match')







These are all the ways to extract alphanumeric characters from a string:
Remember that \w is shorthand for "word characters" i.e. [a-zA-Z0-9_]








$ python -m timeit -s \
     "import string" \
     "''.join(ch for ch in string.printable if ch.isalnum())" 
10000 loops, best of 3: 57.6 usec per loop

$ python -m timeit -s \
    "import string" \
    "filter(str.isalnum, string.printable)"                 
10000 loops, best of 3: 37.9 usec per loop

$ python -m timeit -s \
    "import re, string" \
    "re.sub('[\W_]', '', string.printable)"
10000 loops, best of 3: 27.5 usec per loop

$ python -m timeit -s \
    "import re, string" \
    "re.sub('[\W_]+', '', string.printable)"                
100000 loops, best of 3: 15 usec per loop

$ python -m timeit -s \
    "import re, string; pattern = re.compile('[\W_]+')" \
    "pattern.sub('', string.printable)" 
100000 loops, best of 3: 11.2 usec per loop







So, if I choose the fastest approach it is as follows:






 import re, string
 dirtystring = "abcd^#$^!#^&%$!!#^&efg"
 pattern = re.compile('[\W_]+')
 pattern = re.compile('[\W_]+')
 cleanstring = pattern.sub('', dirtystring)
 pattern.sub('', "../goodfile.txt")

def sanitize_string(dirtyname):
    pattern = re.compile('\.\.\/')
    return pattern.sub('', dirtyname)














11.3 substitutions, individual words, ignore case







line = "Lovely plumage!  The Norwegian Blue prefers keeping on its back.  Lovely plumage!"

line = re.sub(r"Norwegian", "Scottish", line)
#  "Lovely plumage! The Scottish Blue prefers keeping on its back.  Lovely plumage!"

line = re.sub(r"plumage", "song", line)
#  "Lovely song! The Scottish Blue prefers keeping on its back.  Lovely song!"

line = re.sub(r"song", "plumage", line, 1)  # replace only 1 occurrence
#  "Lovely plumage! The Scottish Blue prefers keeping on its back.  Lovely song!"

line = re.sub(r"song|back", "plumage", line)
#  "Lovely plumage! The Scottish Blue prefers keeping on its plumage.  Lovely plumage!"

line = re.sub(r"Scottish", "Norwegian", line)
#  "Lovely plumage! The Norwegian Blue prefers keeping on its back.  Lovely plumage!"














11.4 re as grep to edit lines in a file






11.4.1 First how to extract matches from a file:





To extract lines from a file that match a regexp string:
Example 1:






import re, string
pattern = re.compile('*.png')

for file in dirlisting:
    with open{file} as f:
         for line in f:
             result = pattern.search(line)







Example 2:






import re
pattern = re.compile("some-regex-expression")
with open("file.txt") as f:
    for line in f:
        result = pattern.search(line)







Use example 2, because Python automatically compiles and caches the regex,
so a separate compile step is *   #+BEGINEXAMPLE
   #+BEGINEXAMPLE
   not required*. 






import re
with open('file.txt') as f:
    for line in f:
        match = re.search('some-regex-expression', line)








Example 3






import re
#Define the search term:
pattern = r"f\(\s*([^,]+)\s*,\s*([^,]+)\s*\)" #pattern must be enclosed in quotes

#Create an empty list:
data = []

#then

for line in open(r'file.txt'):
    if line !='':  #<-- To make sure the whole file is read
        word = re.findall(pattFinder1, line)
        data.append(str(word))   












11.4.2 Then, how to edit a file in place.





All the above examples extracted matching strings, or lines.  If you
want to edit a file and change a matching string to another string, you
will have to write to the file, while it is still open.  





Here is an example where you do not want to use a extract lines from a file







# constants (easier to change into parameters for a function
stringiwanttochange = "org-graphics2"
whatstringwillbecome = "org-graphics/"
file_type = ".org"
targetdirectory = "/Users/zintis/eg/plain-files"
regex = re.compile(stringiwanttochange)

# create a list of strings of filenames in the specified directory
directory = os.listdir(targetdirectory)
os.chdir(targetdirectory)

for file in directory:
    # first restrict changes to certain file types,
    if (file_type in file):
        open_file = open(file, 'r')
        read_file = open_file.read()
        open_file.close()
        read_file = regex.sub(whatstringwillbecome, read_file)
        write_file = open(file, 'w')
        write_file.write(read_file)
        write_file.close()







For each file that is of the filetype, these 5 steps



    

  1. open the file in read only mode.
open_file is a data object of type "_io.TextIOWrapper"

    2. 

  2. create a string from the open file, called read_file

    3. 

  3. close the file, as we are done with it.

    4. 

  4. change the string using regex.sub("newstring", readfile) which is a string

    5. 

  5. open the same file for writing

    6. 

  6. dump the entire string to the file.

    7. 

  7. close the file.

    8. 











11.4.3 Reading a file into a pyton list





Each line will become an element of a python list of strings.  There will be
as many elements in the list as there are lines in the file.







actors = [line.strip() for line in open("cast.txt", "r")]







If the file cast.txt looks like this:



Cleese
Palin
Chapman 
Jones
Idle
Gilliam





will look like:






print(actors)
#  ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam']







Don't forget to close the file.














12 Inputs





As described in Strings, inputs are always strings.  They are entered
after the enter key is pressed.








12.1 Inputs without hammer





You can install pyinput from pypi.org that allows one to read keyboard
inputs (and mouse inputs) on single keystrokes, without needing the enter key
pressed.





Here are two variations:







from pynput import keyboard

print("     press the escape key to quit    ")          
with keyboard.Events() as events:
    for event in events:
        if event.key == keyboard.Key.esc:
            break
        else:
            print(f' event.key is {event.key}')







and







from pynput import keyboard

loopcontrol = 'X'
# loop until a timeout
while loopcontrol:
    with keyboard.Events() as events:
        # wait for at most 15 seconds
        event = events.get(15)
        if event is None:
            print('Ok, finishing up loop after no input for 15 seconds.')
            loopcontrol = None
        else:
            print(f' Received event {event}')
            print(f' event.key is {event.key} ')
            loopcontrol = event.key















13 Strings






13.1 Inputs are always strings.





Note that every input() will always be a string.  If you need something else, you have to 
convert it.   For example:






x = int(input("\n Enter an integer >= 1 \n"))







If you want to limit the input to a particular selection of strings, for
example yes or no options, I have done inputs in a while loop until one of
the options is chosen.  For example:







response = ''
while not (response == 'y' or response == 'n'):
    response = input(" enter either a 'y' or 'n'")
    if response == 'y':
        print("Yes")
    if response == 'n':
        print("No")













13.2 String quotation marks:





Can use any of these four ways to quote a string:



    

  • single tics  'string'

    • 

  • single dirks "string"

    • 

  • triple tics  '''string'''      # allows your string to cross lines

    • 

  • triple dirks """string"""      # allows your string to cross lines

    • 










13.3 split and join methods on strings





These are methods on type(str).  You "some string".join([a list of strings])





Example







'.'.join(['John', 'Eric', 'Graham', 'Michael', 'Terry']
# results in John.Eric.Graham.Michael.Terry










first = "Michael"
last = "Palin"
full = first + " " + last
name = full.split()
print(type(name))
# name = ['michael, 'palin']
full_name = ", ".join([first, last])   # join() takes on a list as an argument.
# fullname = 'michael, palin'


eachword = eachline.split(" ")
lastword-fields = eachword[8].split("-")

++++++++++++++++++
str.join(sequence)
where sequence is a sequence of elements to be joined, and str is the string to join them with??

jj=">";
seq = ("a", "b", "c");  # this is the sequence of [2]
print jj.join( seq )







//// >>> ||| <<<  \\\\//// >>> ||| <<<  \\\\//// >>> ||| <<<  \\\\//// >>>








13.3.1 Convert a string to a list





Often as a user input, when you want each input as its own element of a list:






first_ten_primes = "2 3 5 7 11 13 17 19 23 29"
primes_as_strings = first_ten_primes.split()
# will be ['2', '3', '5', '7', '11', '13', '17', '19', '23', '29'] as strings

primes = list(map(int, first_ten_primes.split()))
# will be [2, 3, 5, 7, 11, 13, 17, 19, 23, 29] as integers













13.3.2 Convert a list to a string





Reverse the above example:







primes =  [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]

primes_as_single_string = ' '.join(map(str, primes))

digits = [1, 2, 3, 4, 5, 6]
single_number = ''.join(map(str, digits))  # will be a single string

















13.4 print.format with dictionary keys





If you have a dictionary, you can use the .format(**dictname) construct that
allows you to enter any key in curly braces and get value substitution from
that key's value from the dictionary.







# my little dictionary "actor" has two keys, 'name' and 'age'
actor = {'name': 'Eric', 'age': 81}

print(" Hello, {name}.  You are {age}.".format(**actor))
print(" {age} is {name}'s age.".format(**actor))
print(" As you can see order does not matter, just the key. \n\n")








This gives you the following output:



Hello, Eric.  You are 81.
81 is Eric's age.
As you can see order does not matter, just the key. 













14 f strings aka f-strings





Python version 3.6 introduced f strings.  They are string literals that have an
f at the beginning and curly braces containing expressions that will be replaced
with their values. The expressions are evaluated at runtime and then formatted
using the format protocol.  Note: if you actually want to print curly braces,
simply double them up.  See in the next example:








14.1 first some examples







name = "Eric Idle"
loose_change = 4.0499
total = 10_000_000_000
print("_ "*35)
print(f" Hello, {name = }.  You have {loose_change = } in your pocket.  needs python 3.8 or newer")
print(f" putting the equals sign in the curly braces is called debugging mode ")
print(f" Hello, {name}.  You have {loose_change} in your pocket.")
print(f" Hello, {name}.  You have ${loose_change:.2f} in your pocket.")
print(f" but you have ${total:,.2f} in your chequing account.\n")
print(f" {{single braces actually being printed}} ")







Gives this output: 



_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 Hello, name = 'Eric Idle'.  You have loose_change = 4.0499 in your pocket.  needs python 3.8 or newer
 putting the equals sign in the curly braces is called debugging mode 
 Hello, Eric Idle.  You have 4.0499 in your pocket.
 Hello, Eric Idle.  You have $4.05 in your pocket.
 but you have $10,000,000,000.00 in your chequing account.

 {single braces actually being printed} 








name = "Eric Idle"
profession = "actor"
affiliation = "Monty Python"

message = (f"Hi {name}.  You are a {profession}.  You were in {affiliation}.")
print(message)

message = (f"Hi {name}.  "  f"You are a {profession}.  "  f"You were in {affiliation}.")
print(message)

message = (f"Hi {name}.  "
           f"You are a {profession}.  "
           f"You were in {affiliation}.   --> notice the lack of commas. \n")
print(message)







Gives this output:



Hi Eric Idle.  You are a actor.  You were in Monty Python.
Hi Eric Idle.  You are a actor.  You were in Monty Python.
Hi Eric Idle.  You are a actor.  You were in Monty Python.   --> notice the lack of commas. 










14.2 f strings with strings





Examples of f strings when printing strings or variables that contain strings:







name = "Eric Idle"
print(f"  {{Hello}} used doulbe braces to show actual braces in the output")
print(f"  Hello {name}")

def to_lowercase(input):
    '''this func used in f string formats to convert to lower case.
    obviously one would just use the .lower() function directly, but
    this is used here to illustrate the customizable f string cases
    '''
    return input.lower()

print(f' >>> {to_lowercase(name)} is funny.')








Gives this output: 




{Hello} used doulbe braces to show actual braces in the output
Hello Eric Idle
>>> eric idle is funny.








14.2.1 f string conversions





You can use !a or !r after the string in f strings to covert the output to a
repr (!r) or ascii (!a) These are called 'conversions' They are useful when
debugging prints that have special characters.  Also, notice the lack of commas
between f"strings".





They were kept around since they are similar to the str.format way of doing
things, but with f strings check out: repr(string) which when you think of it
is far more human readable, hence more pythonic, in my opinion. 








name = "Eric Idle"
profession = "actor"
affiliation = "Monty Python"

message = (f"Hi {name}.  "  f"You are a {profession}.  "  f"You were in {affiliation}.\n")
print(f"Message is:  {message}")
print(f"with !r is: {message!r}")
print(f"with !a is: {message!a}")







GIves this output:



Message is:  Hi Eric Idle.  You are a actor.  You were in Monty Python.

with !r is: 'Hi Eric Idle.  You are a actor.  You were in Monty Python.\n'
with !a is: 'Hi Eric Idle.  You are a actor.  You were in Monty Python.\n'




    

  • !a converts any non-ascii strings to an ascii equivalent.

    • 

  • !r converts the string to its repr

    • 










14.2.2 f string & repr() & ascii() replace !r and !a





As noted in the above section, f strings lets you use repr(string) to print
out the string representation, and not the string itself.  Easiest to just see
it in an example again:







message = (f"Hi {name}.\n"  f"You are a {profession}.\n"  f"You were in {affiliation}.\n")
print(f"{message}")
print("_ "*35)
print(f"{repr(message)}")
print(f"{ascii(message)}")







Which gives the output of:



Hi Eric Idle.
You are a actor.
You were in Monty Python.

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
'Hi Eric Idle.\nYou are a actor.\nYou were in Monty Python.\n'
'Hi Eric Idle.\nYou are a actor.\nYou were in Monty Python.\n'













14.3 f strings with integers






14.3.1 = sign for debugging





When you have an expression or a variable in curly braces, it is evaluated
before printing.  If you place an = inside the curly brackes, python will print
the actual variable, or expression first and then its evaluation








myvar: str = 'Flying Circus'
feigenbaum = 4.669201609
print(f"Feigenbaum constant is {feigenbaum}")
print(f"{feigenbaum = }")
print(f"{feigenbaum = :.4f}")
print(f"{feigenbaum + 10000 = :,.4f}\n")  
print(f"{myvar}")
print(f"{myvar=}")
print(f"{myvar = }")








Gives this output: 



Feigenbaum constant is 4.669201609
feigenbaum = 4.669201609
feigenbaum = 4.6692
feigenbaum + 10000 = 10,004.6692

Flying Circus
myvar='Flying Circus'
myvar = Flying Circus











14.3.2 _ and , for larger interger values





within the curly braces, you can append a colon : followed by a , or _
for integers python will add a comma or an underscore every 3 digits






us_dept = 34_515_550_184_614

print(f"{us_dept}")
print(f"{us_dept:,}")
print(f"{us_dept:_}")
print(f"{us_dept:e}")
















14.4 f strings with float values






14.4.1 floating :.2f





within the curly braces, you can append a colon : followed by a decimal . and
the number of digits after the decimal you want printed, followed by f (float)
Python will do the expected rounding for you.







minwage = 14
print(f'\n ${minwage:.2f} minimum wage per hour')
print(f' ${minwage*40:.2f} minimum wage per week')
print(f' ${minwage*40*52:,.2f} minimum wage per year')
print(' ${:,.2f} minimum wage per year'.format(minwage*40*52))

print('\n The cost of a square pie is ${:,.2f}\n'.format(math.pi**2))












14.4.2 f strings & specific precision floating points





When it comes to float numbers, you can use format specifiers:




    

  • f'{value:.{precision}}'

    • 





where value is some floating point value.  Optionally you can add a comma to
separate thousands for larger numbers, as in :,.2f seen below.







import math
minwage = 14.355
print(f'\n ${minwage:.2f} hourly minimum wage')
print(f' ${minwage*40:.2f} per week') 
print(f' ${minwage*40*52:.2f} per year')
print(f' ${minwage*40*52:,.2f} per year')
print(' ${:,.2f} per year'.format(minwage*40*52))

print(f'\n A circle of radius 3 has an area of {math.pi*3**2:.5f}')
print(' A circle of radius 3 has an aera of {:.1f}\n'.format(math.pi*3**2))







Gives this output: 



$14.36 hourly minimum wage
$574.20 per week
$29858.40 per year
$29,858.40 per year
$29,858.40 per year

A circle of radius 3 has an area of 28.27433
A circle of radius 3 has an aera of 28.3











14.4.3 f strings & specific preciesion include width





If you also specify a width along with the precision as in:



    

  • f'{value:{width}.{precision}}'

    • 





where value is some floating point value, width is the total width of the value
and precision is the number of significant digits to display. width is a way of
column aligning output.






  1. exceeding width
    

    
When python runs out of the available width, in this case 6 the number is still
printed, taking up additional spaces to the right as needed.

    

    


    import math
print(" 1st try keeping width at 6 and changing precision from 3 to 6")
print(f'{math.e:{6}.{3}}')
print(f'{math.e:{6}.{4}}')
print(f'{math.e:{6}.{5}}')
print(f'{math.e:{6}.{6}}')

    

    


    
Gives this output: 

    

     1st try keeping width at 6 and changing precision from 3 to 6
  2.72
 2.718
2.7183
2.71828

    

    

    2. 



  2. varying width
    

    
Notice that although when width is less than the precision, in this case 8, the
output will use up all the space needed to get the precision printed.

    


    
But when width is greater than precision, python will print the floating point
value right justified in the width requested.

    

    


    import math
print("\n 2nd try keeping precision at 8 and changing width from 1 to 16")
print("....5....0"*7)
print(f'{math.e:{1}.{8}}')
print(f'{math.e:{2}.{8}}')
print(f'{math.e:{3}.{8}}')
print(f'{math.e:{4}.{8}}')
print(f'{math.e:{5}.{8}}')
print(f'{math.e:{6}.{8}}')
print(f'{math.e:{7}.{8}}')
print(f'{math.e:{8}.{8}}')
print(f'{math.e:{9}.{8}}')
print(f'{math.e:{10}.{8}}')
print(f'{math.e:{11}.{8}}')
print(f'{math.e:{12}.{8}}')
print(f'{math.e:{13}.{8}}')
print(f'{math.e:{14}.{8}}')
print(f'{math.e:{15}.{8}}')
print(f'{math.e:{16}.{8}}')
print(f'{math.e:{17}.{8}}')
print("....5....0"*7)

    

    

    
Gives this output: 

    

     2nd try keeping precision at 8 and changing width from 1 to 16
....5....0....5....0....5....0....5....0....5....0....5....0....5....0
2.7182818
2.7182818
2.7182818
2.7182818
2.7182818
2.7182818
2.7182818
2.7182818
2.7182818
 2.7182818
  2.7182818
   2.7182818
    2.7182818
     2.7182818
      2.7182818
       2.7182818
        2.7182818
....5....0....5....0....5....0....5....0....5....0....5....0....5....0


    

    

    3. 




  3. varying precision
    

    
When the width is sufficient, but precision is less, python will right justify
the output.

    


    


    import math
print("....5....0"*7)
print(f'{math.e:{12}.{11}}')
print(f'{math.e:{12}.{10}}')
print(f'{math.e:{12}.{9}}')
print(f'{math.e:{12}.{8}}')
print(f'{math.e:{12}.{7}}')
print(f'{math.e:{12}.{6}}')
print(f'{math.e:{12}.{5}}')
print(f'{math.e:{12}.{4}}')
print(f'{math.e:{12}.{3}}')
print(f'{math.e:{12}.{2}}')
print("....5....0"*7)

    

    


    
Gives this output: 

    

    ....5....0....5....0....5....0....5....0....5....0....5....0....5....0
2.7182818285
 2.718281828
  2.71828183
   2.7182818
    2.718282
     2.71828
      2.7183
       2.718
        2.72
         2.7
....5....0....5....0....5....0....5....0....5....0....5....0....5....0         

    

    

    4. 



  4. significant digits vs precision
    

    
The precision is just the number of digits past the decimal place, which is
NOT the same thing as significant digits from a science perspective.  As you
can see in the next example the last two outputs have two more significant
digits as the first two, even though both have .2f and .3f 

    

    


    import math
print("....5....0"*7)
print(f'{math.pi:.2f}')
print(f'{math.pi:.3f}')
print(f'{math.pi*100:.2f}')
print(f'{math.pi*100:.3f}')
print("....5....0"*7)

    

    


    
Gives this output: 

    

    ....5....0....5....0....5....0....5....0....5....0....5....0....5....0
3.14
3.142
314.16
314.159
....5....0....5....0....5....0....5....0....5....0....5....0....5....0


    



    
Reminder that by default floating point numbers are left aligned, taking up
all the space they need to the right.
#+BEGINSRC python
feigenbaum = 4.66920160910299067
print(f"\nFeigenbaum const.:   {feigenbaum}")
print(f"       scientific:   {feigenbaum:e}")
print(f"six decimal places:  {feigenbaum:.6f}")
print(f"two decimal places:  {feigenbaum:.2f}")
print(f"six signif. digits:  {feigenbaum:.6}")
print(f"two signif. digits:  {feigenbaum:.2}\n")

    


    
print(f"six signif. digits:  {feigenbaum:.6}   notice that the trailing 0 is not displayed in 4.66920 ")
print(f"six signif. digits:  {4.123456789:.6}  but there realy are six signif. digits\n")

    


    
print(f"  {myprime}")
#+ENDSR

    


    
Gives this output: 

    

    Feigenbaum const.:   4.66920160910299
       scientific:   4.669202e+00
six decimal places:  4.669202
two decimal places:  4.67
six signif. digits:  4.6692
two signif. digits:  4.7

six signif. digits:  4.6692   notice that the trailing 0 is not displayed in 4.66920 
six signif. digits:  4.12346  but there realy are six signif. digits

  1031
  

    

    












14.5 f strings alignment





The above shows some automatic alignment of float numbers when width  is
specified, but there are more flexible alignments of both floats as well as
strings.







14.5.1 alignment right





right alignment is the default for intergers and floats






myvar: str = 'Monty Python'
myfloat: float = 3.1415926535
print("....5....0"*7) 
print(f'{myvar:>20}')
print(f'{myvar:20}')
print(f'{myfloat:>20}')
print(f'{myfloat:20}')












14.5.2 alignment left





Left alignment is the default alignment for strings






myvar: str = 'Monty Python'
myfloat: float = 3.1415926535
print("....5....0"*7) 
print(f'{myvar:<20}')
print(f'{myvar:20}')
print(f'{myfloat:<20}')
print(f'{myfloat:20}')








The last two examples give this output:



....5....0....5....0....5....0....5....0....5....0....5....0....5....0
        Monty Python
Monty Python        
        3.1415926535
        3.1415926535
....5....0....5....0....5....0....5....0....5....0....5....0....5....0
Monty Python        
Monty Python        
3.1415926535        
        3.1415926535










14.5.3 alignment centre





Works for both strings and floats






myvar: str = 'Monty Python'
myfloat: float = 3.1415926535
print("....5....0"*7) 
print(f'{myvar:^20}')
print(f'{myvar:20}')
print(f'{myfloat:^20}')
print(f'{myfloat:20}')







....5....0....5....0....5....0....5....0....5....0....5....0....5....0
    Monty Python    
Monty Python        
    3.1415926535    
        3.1415926535









14.5.4 alignment with fill spaces





The default fill space is the blank space.  All the examples so far have used it
We can override that with any character by preceeding the alignment char with
the character we want.  In this use case, we MUST have the alignment character
as without it, python tries to interpret the _ or - or . or *





For example:






myvar: str = 'Monty Python'
myfloat: float = 3.1415926535
print("....5....0"*7) 
print(f'{myvar:_^20}')
print(f'{myvar:-^20}')
print(f'{myfloat:|^20}')
print(f'{myfloat:*^20}')







Results in this output:



____Monty Python____
----Monty Python----
....3.1415926535....
****3.1415926535****













14.6 f string number conversions





f strings also let you easily display an integer in hexadecimal, octal, binary,
versions.







my_prime = 1031

print(f"default:     {my_prime}")
print(f"hexadecimal: {my_prime:x}")
print(f"octal        {my_prime:o}")
print(f"binary       {my_prime:b}")
print(f"scientific:  {my_prime:e}")
print(f"zero padded: {my_prime:07}")
print(f"as a percentage:     {.9517521:%}")
print(f"as a percentage:     {.9517521:.1%}")


national_defecit = 95_570_000_000
print(f"\nCanada's deficit  {national_defecit:,}")
print(f"Canada's deficit ${national_defecit:,.2f}")
print(f"Canada's deficit ${national_defecit:_.2f}")







Gives this output: 



default:     1031
hexadecimal: 407
octal        2007
binary       10000000111
scientific:  1.031000e+03
zero padded: 0001031
as a percentage:     95.175210%
as a percentage:     95.2%


Canada's deficit 95,570,000,000
Canada's deficit $95,570,000,000.00
Canada's deficit $95_570_000_000.00












14.7 Converting number to arbitary bases





You can use the f-string conversions described above to convert between the
most common bases only.  i.e. binary, octal, decimal, hexadecimal.  But to
convert an integer to an arbitrary base you can use the mod % function.






For example to convert a decimal integer 47 to base 3 do the following:



1. 47 mod 3 = 2   **
   (47-2)/3 = 15
2. 15 mod 3 = 0   **
   (15-0)/3 = 5
3. 5 mod 3 = 2    **
   (5-2)/3 = 1
4. 1 mod 3 = 1    **

47 in base 3 is 1202.







def numberToBase(n:int, base:int) -> str:
    '''
    Given a decimal number, n, convert it to base b

    Call using numberToBase(n, b)

    Returns the decimal number in base b 

    '''
    if n == 0:
        return [0]
    digits = []
    while n:
        digits.append(int(n % base))
        n //= base

    basexNumber = ''.join(map(str, digits[::-1]))
    return basexNumber



 print(f' 47 in base 3 is {numberToBase(47, 3)}')







Reversing that numberFromBase takes a given number and its base, and computes
that number in base 10 (decimal)







def baseToDecimal(numberToConvert:list, base:int) -> int:
    '''
    pass in an integer number as a list of digits, and its base.  Returns that
    number in base 10.  For example: baseToDecimal([1, 1, 1, 1, 1, 1], 6) will
    return 9331 which is 111111 base six converted to decimal

    The number entered is checked for having digits outside of the base entered.
    Then, the number is converted into its equivalent decimal number.

    Returns an interger, base 10.
    '''

    # check that the number does not have digits outside the base

    # split the number into a list of individual digits. if number passed is an
    # integer with this line:
    # listofDigits = [int(i) for i in str(numberToConvert)]

    # if number passed is a list of digits already, just read that list.
    listofDigits = numberToConvert

    decimalsum = 0
    for i,digitinbase in enumerate(listofDigits[::-1]):
        # print(f' in the {i}th position the digit is {digitinbase}')
        # if number passed was a list of digits already, do not need to use int()
        decimalsum += (digitinbase*base**i)
        # print(f' .  .  .  .  .  .  .  .  .  The running decimal total is {decimalsum}')
    return decimalsum

x = numberToBase(47, 3)
y = baseToDecimal([6, 1, 5, 4, 2, 5], 6)













14.8 f strings with datetime objects





There are several commonly used date object output formats that python has built
in.  For the following examples, assume that I have this code already present:






from datetime import datetime
now: datetime = datetime.now()










14.8.1 %y%m%d







print(f"{now:%d.%m.%y}")






15.03.24










14.8.2 %H:%M:%S







print(f"{now:%d.%m.%y  %H:%M:%S}")
print(f'{now:%d.%m:%y  -->  %H:%M}')
print(f'{now:%d.%m:%y  -->  %I%p}')







results in:



16.03.24  14:24:04
16.03:24  -->  14:24
16.03:24  -->  02PM










14.8.3 Date format as variable (nested f-strings)





With datestrings as a variable, you will need to nest f-strings.






import math
can_debt : float = 1_219_222_184_614.45

currency: str = ',.2f'
science: str = '>20,.6f'
print(f'{math.pi:{science}}')
print(f'${can_debt:{currency}}')




















14.9 f strings using raw strings for back-slashes





In Windows, your path uses many back-slashes \ which of course are escape
characters in the normal word. To get around this, python offers the r
special f-string type, r = raw







newapp_dir: str = 'opendns'
mypath: str = r'\Windows\System\Registry'

newapp_path: str = fr'\Windows\System\Registry\{newapp_dir}'















14.10 f strings outside of print statements









15 Tuples





x = (1, 2, 3)






Tuples function almost exactly like lists with one major difference:
Immutability. A list can be change (mutated) after it is created (add items,
update items, remove items, etc.). A tuple is immutable, which means it cannot
be changed after creation.





You create a tuple by using parentheses () and separating the elements of the
tuples with commas. Accessing elements via indexing works just like it does
with lists, though you cannot update an element of a tuple.






Tuples are useful (and efficient) if the contents of the list are not to be
changed.  i.e. immutable







cast = ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam']
# cast is a list.

x = tuple(cast)
# x is a tuple made from the list "cast".  Is the same as:
x = ('Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam')












16 Sets







mynullset = set()
some_primes = {2, 3, 5, 7, 11, 13, 17, 19, 23}







Sets can contain anything, but every element is unique.  So if I add "a" to
the alphabet set more than once, like this:



    

  • alfabit = {"a", "b", "c", "a"}

    • 




the alphabet still only has one "a" like this



    

  • alfabit

    • 




{"a", "b", "c"}
But python does not give you an error, it just says 
"ok I'll add "a" to the set if it isn't already there."





help(set) will show you the available methods for sets, including set



    

  • differenceupdate,

    • 

  • update,

    • 

  • intersection

    • 





also: dir(set)  , or dir(alfabit)  etc….








16.1 set methods





using the dir(set) and help(set.methodofinterest) you will see many built-in
set operations.  Some of them I explain here:








16.1.1 intersection







letters = {a, b, c, d, e}
guess = {e, f, g}
invalid_guess = letters.intersection(guess)
# invalid_guess will be now the set {e} which is NOT the null set
# so equates to TRUE.  Remember that the null set evaluates to FALSE
if invalid_guess:
   # easy way to act when the intersection of two sets is not empty.
   print(f"Set is NOT empty.  In fact contains the elements {invalid_guess}")












16.1.2 update





Update a set with the union of itself and an other set.






# we are ADDING ELEMENTS to the set of all rejected letters
letters.update(guess)












16.1.3 differenceupdate





Remove all elements of another set from this set






# we are REMOVING ELEMENTS from the set of letters
letters.difference_update(guess)












16.1.4 add





Add the elements of one set to another set.






letters.add(guess)   # adds the letters in guess to the letters set
















16.2 Set comprehensions





Like list comprehensions and dictionary comprehensions, sets can be created
using set comprehensions:







x = {3*x for x in range(10) if x > 5}







Results in {21, 24, 18, 27}  # notice that the order is not maintained in sets.
However, each element in a set has to be unique.  Trying to add a duplicate
does nothing.





Or you can do a list comprehension and convert it to a set, like I have done
in my wordle program:







remaining_choices = set([word for word in english_words_lower_set if len(word) == 5])
remaining_letters = set(string.ascii_lowercase)  # the set of all lowercase aphabet
rejected_letters = set()
must_have = set()















17 Iterables and iterators





A list is iterable but it is NOT an iterator.







17.1 Iterable





It is something that can be looped over.



    

  • a list

    • 

  • a tupple

    • 

  • a range ?

    • 

  • a dictionary

    • 

  • a string

    • 

  • a file

    • 

  • a generator (a generator is similar to, but older than, list comprehensions.)

    • 





All iterable objects have a method __iter__() "Dunder iter" method.  All the
methods available to an object can be shown by using the dir(object) command,
typically in an interactive python window/session.







marks = [85, 81, 93, 90, 78]

print(dir(marks))

['__add__',
'__class__',
'__contains__',
'__delattr__',
'__delitem__',
'__dir__',
'__doc__',
'__eq__',
'__format__',
'__ge__',
'__getattribute__',
'__getitem__',
'__gt__',
'__hash__',
'__iadd__',
'__imul__',
'__init__',
'__init_subclass__',
'__iter__',           <-----     this one means this list is iterable
'__le__',
'__len__',
'__lt__',
'__mul__',
'__ne__',
'__new__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__reversed__',
'__rmul__',
'__setattr__',
'__setitem__',
'__sizeof__',
'__str__',
'__subclasshook__',
'append',
'clear',
'copy',
'count',
'extend',
'index',
'insert',
'pop',
'remove',
'reverse',
'sort']













17.2 Iterators





Iterators are object with a state, so that it remembers where it is during
iteration.  Running the dunder iter method on our list, it will return an
iterator.






The state is what keeps track of where in the iterable object it is.  Typically
the next method has to keep track of where it is in the list, and then be able
to move to the next item in the list.





This is good to understand how python for loops run through these iterator
methods automagically for you, and take care of the state, and the error case
when there is a problem








17.2.1 _next__





this method would be availble for all iterator objects.  You can see that
the list above does NOT have the __next__ method.   












17.3 Running an iterator directly (for learning purposes only)





You would never do this for a real program, but to show how iterators work
you can run it directly on your list, for example: marks.__iter__()







     marks = [85, 81, 93, 90, 78]
it_marks = marks.__iter__()   * can also just say it_marks = iter(marks)
print(i_marks)
<list_iterator object at 0x105f04750>

print(dir(it_marks))
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__',
'__ge__', '__getattribute__', '__gt__', '__hash__', '__init__',
'__init_subclass__', '__iter__', '__le__', '__length_hint__', '__lt__',
'__ne__', '__new__', '__next__', '__reduce__', '__reduce_ex__', '__repr__',
'__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__']








See slang.org, for more, but double underscore slang is "dunder", hence
these are "dunder methods".





So, now we can see that the for loop shown below can be written manually
like follows:
(not that you would ever do that)







marks = [85, 81, 93, 90, 78]

for mark in marks:
    print(mark)

it_marks = iter(marks)

while True:
    try:
        item = next(it_marks)
        print(item)
    except StopIteration:
        break:2






These do the same thing.  Cool, huh?










17.4 Why bother learning this?





We can add these built-in methods to our own classes and make them
iterable as well.





For example, creating a zintRange class that acts like the range function:







#!/usr/bin/env python
'''Learn about iterator dunder methods, such as __iter__, and __next__.'''

class ZintRange:

     def __init__(self, start, end):
         self.value = start
         self.end = end

     def __iter__(self):
         '''iter methods must return an object that has the dunder next method'''
         return self

     # will need to add the dunder next method to this class
     def __next__(self):
         if self.value >= self.end:
             raise StopIteration
         current = self.value
         self.value += 1  # increment to the next value
         return current

 nums = ZintRange(1,10)

 for num in nums:
     print(num)

 nums = ZintRange(1,10)
 print(next(nums))
 print(next(nums))
 print(next(nums))
 print(next(nums))














17.5 Looping techniques on each iterable





Reminder that the following are all iterable:




    

  • a list

    • 

  • a tupple

    • 

  • a range ?

    • 

  • a dictionary

    • 

  • a string

    • 

  • a file

    • 

  • a generator (a generator is similar to, but older than, list comprehensions.)

    • 








17.5.1 Iterating a list







zintlist = ["saw", "hammer", "drill"]
for i in zintlist:
    print(i)












17.5.2 Iterating a list using list comprehensions





See: List Comprehensions (see also Dictionary Comprehensions)










17.5.3 Iterating a tupple







zintup = (1, 1, 2, 3, 5, 8, 13, 21)
for i in zintup:
    print(i)












17.5.4 Iterating a dictionary





dictionary.iteritems() vs dictionary.items()





  1. dic.items()
    

    
Used in python3 and returns a copy of the dictionary's list in the form of
(key, value) tuple pairs. So, to print you can simply print dic.items():

    

    


    print(dic.items())

    

    

    

    2. 


  2. dic.iteritems()
    

    
Used in python2, and returns an iterator of the dictionary's list as (key,
value) tuple pairs.  So, to print you should iterate over the
dic.iteritems():

    

    


    for i in dic.iteritems():
    print(i)

    

    



    
You might think that to get the values out of a dictionary you
have to loop over the keys and manually extract the value like so:

    


    


    zint_dic = {'year': 1980, 'sport': 'football', 'trophy': "Grey Cup"}
for key in zint_dic:
    a_value = zint_dic[key]
    print(f"My {key} has value {a_value}")
    print("My {} has value {}".format(key, a_value))

    

    

    
This works, but is bad practice.  Much neater to use the items method.

    



    


    zint_dic = {'year': 1980, 'sport': 'football', 'trophy': "Grey Cup"}
for key,val in zint_dic.iteritems():
    print(f"My {key} has value {val}")
    print("My {} has value {}".format(key, val))

    

    


    
Python dictionaries are mapping objects.  They inherit some special
methods, which Python uses internally to peform some operations.  These
are dunder methods, like contains, delattr, iter, class

    


    


    for key in zint_dic:
print(f"{key} --> {zint_dic[key]}")

print(zint_dic.items()) 

    

    

    
Notice here that key, by default takes on the value of each key in the dict,
one at a time.  The following two are identical.  Use the first one.

    

      

    • for key in zint_dict 

      • 

    • for key in zint_dict.keys():

      • 

    


    
Since dictionary items are views, and they can be iterated, you can iterate
over these views as so:

    


    


    for item in zint_dic.items():
    print(items)

    

    

    








17.5.5 Remember that keys are [2]





Keys are [2].  Always.   So, adding or accessing dictionary items are 
done by specifying key [2], like so:






ctd["keyzp"]=newvalue or
if ctd["country"] == Canada.







Compare that to initially setting up a dict like so:







ctd=dict(country="Canada", population=33000000, language="English/French")






here the keys, country, population, language did NOT use "" but they ARE str







print("I live in " + ctd["country"])
ctd["newkey"] = "string to store in dict ctd"






Look to see if key is in dictionary before trying to retrieve it:







if 'location' in ctd:
    print("I found myself in "  + ctd["location"])






Alternatively use try:







try:
    print("I found myself in "  + ctd["location"])
except KeyError:
    print("the location key does not exist")








From realpython.com:
Python is smart enough to know that a_dict is a dictionary and that it
implements .__iter__(). In this example, Python called .__iter__()
automatically, and this allowed you to iterate over the keys of a_dict.





This is the simplest way to iterate through a dictionary in Python. Just
put it directly into a for loop, and you’re done!





If you use this approach along with a small trick, then you can process
the keys and values of any dictionary. The trick consists of using the
indexing operator [] with the dictionary and its keys to get access to the
values:







  1. Iterating through .keys()
    

    
#+BEGINSRC python
zintdic = {'year': 1980, 'sport': football, 'trophy': "Grey Cup"}
for key in zintdic.keys():
    print(key)

    

    








17.5.6 Unpacking dictionary elements into tuples







for key, value in zint_dic.items():
    print(f" Key is {key} --> Value is {value}")











17.5.7 Unpacking tuples







a, b = (1,2)  # is unpacking a tuple into two variables
a, _ = (1,2)  # is unpacking a tuple into one variable
              # and ignoring the second variable.
              # "_" means "ignore "

a, b, c = (1, 2, 3, 4, 5) # will get an error "too many values to unpack"
a, b, c, _, _ = (1, 2, 3, 4, 5)
                           # this fixes the error. 












  1. Can ignore the extra values:
    

    


    a, b, *_ = (1, 2, 3, 4, 5)
# results in  a=1, b=2

    

    

    

    2. 


  2. Can use splat to catch all the remaning values
    

    


    a, b, *c = (1, 2, 3, 4, 5)
# results in  a=1, b=2, c=[3, 4, 5]

    

    

    

    3. 


  3. Can use the last first two plus 'last' value
    

    
#+BEGINSRC python
a, b, *c, d = (1, 2, 3, 4, 5, 6, 7)

    


    
2    #+ENDSRC

    

    

    4. 

  4. swapping varialbes
    

    


    a, b = b, a  # easy peasy
a, b = b, a+b  # evaluate RS with "old values" and only then assign to LS

    

    

    







17.5.8 Unpacking lists





Must have exactly the same number of variables as items in the sequence.






cast = ['Cleese', 'Palin', 'Chapman', 'Jones', 'Idle', 'Gilliam']
a, b, c, _, _, _ = cast







Results in three variables, each being assinged as follows:
a = 'Cleese', b = 'Palin', c = 'Chapman'










17.5.9 Iterating set







   zintset = (1,2,3,5,7,11,13,17,19,23,29,31)
for i in zintset:
    print(i,end=", ")















17.6 Breaking out of loops





You can break or you can continue.





  1. break
    

    
break and continue work the same way with for loops as with while loops.
break terminates the loop completely and proceeds to the first statement
following the loop:

    

    

    2. 


  2. continue
    

    
Continue terminates the current iteration and proceeds to the next
iteration:

    


    


    for i in list:
    if i < value:
        break    # loop will break and knock you right out
    if i > value
        continue # loop will continue

    

    

    












19 Methods (compare with functions)





Methods and functions are blocks of code that perform tasks when executed.
In python PEP8, function names are in lower case.  In python naming
functions as these restrictions:



    

  • cannot start with a number

    • 

  • cannot be a python keyword (for obvious reasons)

    • 

  • can contain any alphanumeric letter ([a-zA-Z0-9_-])

    • 










20 Functions (compare with methods)





Function is block of code that is also called by its name. (independent)
The function can have different parameters or may not have any at all. If any
data (parameters) are passed, they are passed explicitly.





A function may or may not return any data.
Function does not deal with Class and its instance concept.








20.1 Basic Function Structure in Python :







def function_name ( arg1, arg2, ...) :
 # ...... 
 # function body 
 # ......    












20.2 User Defined Function







def Subtract (a, b): 
    return (a-b)  











20.3 Built-in Function







s = sum([5, 15, 2]) 
print( s ) # prints 22 

mx = max(15, 6) 
print( mx ) # prints 15 













20.4 Functions called with keyword-only parameters





When calling the function, the order of the parameters does not matter,
as long as you specify the keyword for each parameter.  The keywords come
from the function definition, and must match those function arg names.






def welcome(name, age):
    print(f"Hello {name}.  I see you are {age} years old.")

welcome(name="Bob", age=12)   # works fine
welcome(age=18, name="Sue")   # also works just fine














20.5 Functions called with positional-only parameters





When calling the function, all of the parameters must be passed in the
order that the function has them defined. For example:







def welcome(name, age):
    print(f"Hello {name}.  I see you are {age} years old.")

welcome("Bob", 12)   # would print a sensical welcome message
welcome(18, "Sue")   # would print nonesense.













20.6 Arguments vs parameters





Technically, arguments are what you pass into functions, while parameters
are defined and used within the function and are the names/placeholders
for the arguments that are passed to it.  So:



    

  • pass arguments to a function that are held in parameters within the funct.

    • 





More detailed explanation in this Stack Overflow link.












21 Function type restrictions and defaults





Use the def keyword to define a function.  You must follow the same rules
when naming functions as you do when naming variables:




    

  • Cannot start with a number [0-9]

    • 

  • Cannot conflict with a language keyword

    • 

  • Can contain: [A-Za-z0-9_-]

    • 








21.1 basic





def followed by a name in lower case, followed by () followed by a colon :
All function definition lines end in a :







def functionname():












21.2 with parameters





a comma separated list of parameters.  These are variable names that will be
only locally significant within the function, and undefined outside the
function.






def functionname(x, y, z):












21.3 with parameters and default values





Each parameter can optionally have a default value supplied to it.






def functionname(x=["pi", 3.14, 42], y, z=True):












21.4 with parameters restricted to types





After each parameter, you can optionally restrict that parameter to a type.
Python calls these type hints.  The syntax is a colon, space, and the type.
Then comes the comma if more parameters are to be defined for the function.







def functionname(x: list, y: int, z: boolean):
def functionname(x: list[int], y: int, z: boolean):







list[int] in the above example restricts x to a list of integers.










21.5 with restricted return types





Return type hints let you can explicitly specify the returned type.  An "arrow"
and type with the closing function colon of course.







def functionname(x:list, y:int, z: boolean) -> str:







Here is a simple example showing both argument and return type hints:






def is_a_python(someactor:str) -> bool:
    ''' returns true if the actor is a member of Monty Python's Flying Circus'''
    MontyPython = {'John Cleese', 'Graham Chapman', 'Terry Gillian', 'Michael Palin',
                   'Eric Idle', 'Terry Jones'}
    if someactor in MontyPython:
        return True
    else:
        return False

working_actors = ['Jennifer Lawrence', 'Andrew Scott', 'Michael Palin',
                  'Sam Rockwell','Benedict Cumberbatch']

for actor in working_actors:
    if is_a_python(actor):
        print(f'\n {actor} is a legendary Monthy Python member \n')

    else:
        print(f'{actor} is a great actor, but grew up watching Monty Python')








Type hints for returns can be list of strings -> list[str] or -> list[int]
that specify they types inside the list, or more general as in -> list[].





Same goes for tuples and sets, so -> tuple[str] or -> set[int]





Note that since version 3.5 list typing is in stdlib, so nothing extra need be
done.  Before that you had to import List from
the typing package, i.e. from typing import List I think, and of course:
pip install typing before you can import it.










21.6 Shortcut return of boolean





You can check conditions directly in the return statement and if conditions
are met, return a True boolean, and if not, return a False boolean value.





For example, you can easily check if two values match specific strings,  and
return the boolean result like this:







   # using ipython as it is easy to show this:
In [1]: def check_auth(username, password) -> bool:
   ...:     # will return a boolean according to the two conditions
   ...:     # both have be match for the return to be true
   ...:     return username == "Monty" and password == "Python"
   ...: 

In [2]: x = check_auth("joe", "black")

In [3]: x
Out[3]: False

In [4]: check_auth("Monty", "Python")
Out[4]: True












21.6.1 Comparing sha256 hashes vs passwords





Related to the above, you can compare the hash of a password and not
the password itself using the built-in hashlib.





in bash, create the hash to save in your program






echo "this is my pass" | sha256sum
857571d81ecdbb959b686bb167bb2b49ff0aecf0c379d1da84df6b8eac254090








Then in python:






import hashlib

from hashlib import sha256

someguess = input('Enter your passwd guess: ')
print(sha256(input_.encode('utf-8')).hexdigest())

# run a sha256 on a password and save this hash
my_secret_hash = 'ec4c1a65b0f994e6d25e29dd74a21b03fa2a3dd28f8fbf1034009588868ca682'
someguess = "this is my pass"  # change to someguess = input('Enter your passwd guess')
someguesse = someguess.encode()

hashed_string = hashlib.sha256(someguess.encode('utf-8')).hexdigest()
print(hashed_string)
print(my_secret_hash)

if hashed_string == my_secret_hash:
    print("you must have entered the correct guess")
elif:
    print("wrong guess!")









From the hashlib documentation  you can see an easier way to do this:







import hashlib
m = hashlib.sha256()
m.update(b"Nobody inspects")
m.update(b" the spammish repetition")
m.digest()
b'\x03\x1e\xdd}Ae\x15\x93\xc5\xfe\\\x00o\xa5u+7\xfd\xdf\xf7\xbcN\x84:\xa6\xaf\x0c\x95\x0fK\x94\x06'
m.hexdigest()
'031edd7d41651593c5fe5c006fa5752b37fddff7bc4e843aa6af0c950f4b9406'

# more condensed
hashlib.sha256(b"Nobody inspects the spammish repetition").hexdigest()
'031edd7d41651593c5fe5c006fa5752b37fddff7bc4e843aa6af0c950f4b9406'


















21.7 functions and the kitchen sink





Putting it all together variable=value:type on each parameter.  And then
-> return-type.   For example:






def functionname(x=["pi", 3.14, 42]:list, y=42:int, z=True:boolean) -> str:








These are all valid function statements:






def join(z_array, delimiter):
def join(z_array, delimiter) -> str:
def join(z_array: list[int], delimiter: str) -> str:
def join(z_array: list[int], delimiter=" "::str) -> str:
def test_join_use_case() -> None:
def google_append(SUMMARYSTATS):
def get_file():
def main():
def __init__(self, first, last, role, lines, networth=0: int):
def fullname(self):
def actor_from_string(cls, actstring):
def list_files(self, page_size=20):
def convert_to_RFC_datetime(year=1900, month=1, day=1, hour=0, minute=0):
def host_list(apic, ticket, ip=None, mac=None, name=None):
def apic_login(apic, username, password):
def network_device_list(apic, ticket, id=None:str):   # not sure about this one













21.8 When to use methods vs functions





From developer.cisco.com:



"Code that performs a discrete task, even if it happens only once, may be a
candidate for encapsulation. Classic examples include utility functions that
evaluate inputs and return a logical result (for example, compare two strings
for length), perform I/O operations (for example, read a file from disk), or
translate data into other forms (for example, parse and process data). Here,
we're encapsulating for clarity and testability, as well as for possible
future re-use or extension.










21.9 Task code that is used more than once should probably be encapsulated.





If you find yourself copying several lines of code around, it probably needs
to be a method or function. You can accommodate slight variations in usage
using logic to assess passed parameters (see below).






We have already seen methods such as iter ("dunder iter method")
A method in python is somewhat similar to a function, except it is associated
with object/classes. Methods in python are very similar to functions except
for two major differences. 



    

  1. The method is implicitly used for an object for which it is called.

    2. 

  2. The method is accessible to data that is contained within the class.

    3. 




from tutorialspoint.com










21.10 Arguments and parameters





variables that are passed to a function or method. The types in the function or
method must match the types passed to them.  Python > 3.5 supports typing which
fixes the types and provides type hints when you get it wrong.






  1. The Difference Between Arguments And Parameters?
    

    
Parameters are defined by the names that appear in a function definition,
whereas arguments are the values actually passed to a function when calling
it. 

    

    


    1. Parameters
      

      
Parameters define what types of arguments a function can accept. For
example, given the function definition:

      


      


      def func(foo, bar=None, **kwargs):
    pass

      

      


      
foo, bar and kwargs are parameters of func.

      

      

      2. 


    2. Arguments
      

      
The values actually passed to a function.  So for example when calling 
func for example:

      


      


      func(42, bar=314, extra=somevar)

      

      


      
The values 42, 314, and somevar are arguments.

      

      

    








21.11 Default values in function parameters





As mentioned above, in the Function syntax def section above, parameters can
be given a default value. See also restricting values and restricting return
values.







def join(z_array: list[int], delimiter=" ": str) -> str:
def list_files(self, page_size=20):
def convert_to_RFC_datetime(year=1900, month=1, day=1, hour=0, minute=0):
def host_list(apic, ticket, ip=None, mac=None, name=None):
def zinfunc(saywhat: str="Hello Jack!"):
   print(f"Hello Monty.  Say {str(saywhat)}")
def network_device_list(apic, ticket, id: NoneType=None):   # not sure about this one







Seems straight forward, however there is a "gotcha".  This is known as the 
default mutable objects.  If the parameter is an immutable object, there is
nothing to worry about.





However, if the object is mutable, like lists or sets, be aware this rule:




    

  • Argument defaults are *defined when the function is defined*, not when the
function is run

    • 





In other words, the default value is set ONLY THE FIRST TIME a function is
called.  Subsequent calls to the function will use the SAME value previously
   used.





So a list parameter with a default value of [], the empty list, will only
be the empty list the very first time the function is called.  After that
you might think that calling the function again will again have an empty
list, but it will NOT   The list will have in it whatever was put in it
the last time the function was called!





Ok, so to fix this, do not set a default list to = [], but rather = None





Here is the wrong approach:







def addcastmembers(actor, cast=[]):
    cast.append(actor)
    return cast

MPcast = addcastmembers("Cleese")    #["Cleese"]
MPcast = addcastmembers("Palin")    #["Cleese", "Palin"]  still ok.

SLcast = addcastmembers("Cumberbatch")  #oops...we have SLcast = ["Cleese", "Palin", "Cumberbatch"]







And here is the John approach:






 def addcastmembers(actor, cast=None):
     ''' append an actor to a cast.  '''
     # first check if the cast is None, and if so, reset the list to the empty list
     if cast is None:
          cast = []
     cast.append(actor)
     return cast

MPcast = addcastmembers("Cleese")    #["Cleese"]
MPcast = addcastmembers("Palin")    #["Palin"]  wait, we lost John Cleese... 
MPcast = addcastmembers("Cleese", MPcast)  # ok we now have ["Palin", "Cleese"]
SLcast = addcastmembers("Cumberbatch", SLcast)  # ok SLcast is now ["Cumberbatch"]
# the cast is set to the empty list on default only.  If you specify the list
# then the function will append the cast member.












21.12 Python is a dynamically typed language





Python does NOT restrict functions/method arguments to static or fixed types.
Python is a dynamic typing language.  Other languages such as C++, Java return
types and argument types must be specified. They are statically typed languages.





That being said, since version 3.10, python does support type hints to both
arguments to functions, and function returns.





I recommend using type hints, as you will often catch bugs in your code faster
if python error tells you that you cannot pass a string to a function that
expects to see a boolean, etc.








21.12.1 Type Hints for functions.








21.12.2 Type Hints on returns









21.13 return statements





The returned value is captured by the calling code. i.e. A function can return a
a number, say 3.14.  The calling code is where you assign that to a variable,
so x = myfunction(),










21.14 Basic Python method







class class_name 
    def method_name () : 
        # ...... 
        # method body 
        # ......    












21.15 User-Defined Method







class ABC : 
    def method_abc (self): 
        print("I am in method_abc of ABC class. ") 

class_ref = ABC() # object of ABC class 
class_ref.method_abc()






output: I am in methodabc of ABC class.










21.16 In-built Method







import math 

ceil_val = math.ceil(15.25) 
print( "Ceiling value of 15.25 is : ", ceil_val)  















22 Dictionaries





Python dictionaries are similar to other language's "associative arrays"
a.k.a "hashes".   They are both key:value pairings.





Also called "mappings" because they map a 'key' to an arbitrary 'value'.





A dictionary is a data structure that stores simple key-value pairs. They make
it very easy and fast to store and retrieve a value by supplying a key to be
used as an index for that value.








22.1 Values:





The values in a Python dictionary can be anything, and like lists the
types don't have to be consistent.










22.2 Keys





A key has to be unique within the dictionary.





A dictionary's keys have an important restriction - whatever you want to use
as a key has to be immutable and hash-able, meaning they cannot change in
their lifetim.  This means that you could use a tuple as a key (immutable),
but not a list (mutable). Also note that all of the basic data types (int,
float, bool, str, bytes) are immutable and can be used as dictionary keys.










22.3 Syntax mydic={"a":1, "b":2, "c":3}





You create a dictionary by using curly braces {}, separating a key from its
value with a colon : and separating the key-value pairs with commas. You
access and update elements using indexing; however instead of using numerical
sequential index numbers, you use use the key as the index. You can add new
elements simply by assigning a value to a new key.










22.4 Unordered





Dictionaries store "unordered" key-value pairs. In Python v3.6+ it might
look like they maintain the order in which items have been added to the
collection (due to an implementation detail), but maintaining order is not
guaranteed and indeed is not done in versions below v3.6. If you need a
dictionary collection to maintain the order of the items added to it, use the
OrderedDict collection - we'll talk about this in the next Step of this lab.




>>> d = {"apples": 5, "pears": 2, "oranges": 9} 
>>> d["pears"] 
2 
>>> d["pears"] = 6
>>> d["bananas"] = 12
>>> d 
{'apples': 5, 'pears': 6, 'bananas': 12, 'oranges': 9}





There are a number of "batteries included" methods available for
dictionaries - learn more in the Python docs.










22.5 key: value





Also, the order is randomized, i.e. different every time you iterate it.






The following are examples of when to use dictionaries.










22.6 iterators over dictionaries





iterators of a dictionary iterate over just the keys, and not the values.
So






mydict = {"apples": "red", "pears": "green", "oranges": "orange"}
for x in mydict:
  print(x)
# will only print out all the keys.






You want






for x y in mydict.items():
  print(x, y)
# much better.  now printing both keys and values
















22.7 Simpler use of if, elif, elif…





Supposing you have this code:







if name == "John": 
    print "John Cleese plays the dark knight."
elif name == "Eric":
    print "Eric Idle plays Sir Robin."
elif name == "Graham":
    print "Graham Chapman plays Brian."






That could be replaced with the more scalable (by far) dictionary:







name_role_dict = {
   "John": "John Cleese plays the dark knight.",
   "Eric": "Eric Idle plays Sir Robin.",
   "Michael": "Michael Palin plays Sir Galahad",
   "Terry": "Terry Jones plays mother of Brian",
   "Gilliam": "Terry Gilliam plays a deaf mad mute",
   "Eric": "Eric Idle plays Sir Robin.",
   "Graham": "Graham Chapman plays Brian."
}
print(name_job_dict["Eric"])

actors = ("John", "Eric", "Michael", "Graham", "Terry", "Gilliam")
for name in actors:
   print(name_role_dict[name])












22.8 When you need a default value for a key:







def count_duplicates(numbers):
   result = {}
   for number in numbers:
      if number not in result:  # No need for if here
         result[key] = 0
      result[number] += 1
   return result







By using setdefault we can get cleaner code:







def count_duplicates(numbers):
   result = {}
   for number in numbers:
      result.setdefault(number, 0)  # this is clearer
      result[number] += 1
   return result








From: toptal.com
We can also use the dictionary to manipulate lists:
   ,#+BEGINSRC python
   >>> characters = {'a': 1, 'b': 2}
   >>> characters.items() / return a copy of a dictionary’s list of (key, value) 
       pairs (https://docs.python.org/2/library/stdtypes.html#dict.items)
       [('a', 1), ('b', 2)]
   >>> characters = [['a', 1], ['b', 2], ['c', 3]
   >>> dict(characters) / return a new dictionary initialized from a list
       (https://docs.python.org/2/library/stdtypes.html#dict)





{'a': 1, 'b': 2, 'c': 3}
 #+ENDSRC







If necessary, it is easy to change a dictionary form by switching the keys and
values – changing {key: value} to a new dictionary {value: key} – also known as
 inverting the dictionary:






>>> characters = {'a': 1, 'b': 2, 'c': 3}
>>> invert_characters = {v: k for k, v in characters.iteritems()}
>>> invert_characters
{1: 'a', 2: 'b', 3: 'c'}








The final tip is related to exceptions. Developers should watch out for
exceptions.  One of the most annoying is the KeyError exception. To handle this,
developers must first check whether or not a key exists in the dictionary.







>>> character_occurrences = {'a': [], \342\200\230b\342\200\231: []}
>>> character_occurrences[\342\200\230c\342\200\231]
KeyError: 'c'
>>> if \342\200\230c\342\200\231 not in character_occurrences:
        character_occurrences[\342\200\230c\342\200\231] = []
>>> character_occurrences[\342\200\230c\342\200\231]
[]
>>> try:
        print character_occurrences[\342\200\230d\342\200\231]
    except: 
        print \342\200\234There is no character `d` in the string\342\200\235








However, to achieve clean and easily testable code, avoiding exceptions catch
and conditions is a must. So, this is cleaner and easier to understand if the
defaultdict, in collections, is used.







>>> from collections import defaultdict
>>> character_occurrences = defaultdict(list)
>>> character_occurrences['a']
[]
>>> character_occurrences[\342\200\230b\342\200\231].append(10)
>>> character_occurrences[\342\200\230b\342\200\231].append(11)
>>> character_occurrences[\342\200\230b\342\200\231]
[10, 11]













22.9 Standard methods available to dictionaries





Remember to use dir(<dictionary object>) to refresh your memory.  Here I am
listing some common/popular methods.  I assume mydict = {x: 1, y: 2, z: True}





mydict.keys()
mydict.values()
mydict.items()
mydict.clear() #removes all items from the dictionary
mydict._sizeof_()  # not the number of elements but size in bytes





if 'x' in (mydict.keys()):










22.10 dictionary view objects:





The objects returned by dick.keys(), dict.values(), dict.items() are
view objects.  They are dynamic, so when a dictionary changes the view
reflects those changes.




    

  • len(viewobject)

    • 

  • (mydict.keys()

    • 

  • (mydict.values()

    • 

  • (mydict.items()

    • 

  • (mydict.clear() #removes all items from the dictionary

    • 

  • (mydict._sizeof_()  # not the number of elements but size in bytes

    • 













23 Collections Module





Standard python library includes a module
  called "collections' Here is a list
of "collections" inside the collections module









    

  • namedtuple()

    • 

  • deque

    • 

  • ChainMap

    • 

  • Counter

    • 

  • OrderedDict

    • 

  • defaultdict

    • 

  • UserDict

    • 

  • UserList

    • 

  • UserString

    • 





These add common, useful functionality to python.  







from collections import OrderedDict as OD
od = OD()
od["straberry"] = 'red'
od["banana"] = 'yellowd'
od["blueberries"] = 'blue'
od["mangos"] = 'green'












24 *args and **kwargs





Magic variables, that are by convention only args and kwargs  You could just
as well use var and *vars  or anything else you want/need.  It is the
* and ** that are important.   The consises view of this is the following:




*args collects all the positional arguments in a tuple
**kwargs collects all the keyword arguments in a dictionary









24.1 *args





Used mostly in function definitions, they allow you to pass an unspecified
number of arguments to a function.  *args sends a non-keyworded variable
length argument list to the function.   *args accepts all the arguments as
positional arguments.







def print_all_my_args(my_regular_arg, *argv):
        print("first normal arg:", my_regular_arg)
        for arg in argv:
            print(f"another arg through *argv:  {arg}")

print_all_my_args('cast', 'John', 'Michael', 'Terry', 'Eric', 'Graham', 'Terry')








Another simple example of using *args follows.  you’re passing any number of
different positional arguments. addthisup() takes all the parameters that
are provided in the input and packs them all into a single iterable object
named args.








# sum_integers_args.py
def add_this_up(*args):
    result = 0
    # Iterating over the Python args tuple
    for x in args:
        result += x
    return result

print(add_this_up(1, 2, 3))













24.2 **kwargs





Used to pass keyworded variable length arguments to a function.  You should
use **kwargs when you need to process named arguments in a function.  The
key to remember this is that *args accepts all the arguments as positional
arguments, i.e. they are accepted in the order that they were passed to the
function.  Contrast that with **kwargs where the position does not matter
because each argument passed is actually named, and these named arguments
are collected into a dictionary, keyed on the names.








def print_my_kwargs(**kwargs):
 for key, value in kwargs.items():
     print("{0} = {1}".format(key, value))

 >>> print_my_kwargs(name="Palin")
 name = "Palin"














24.3 PEP8 conventions.





Follow these!  Just do it.




    

  • function names variables use snakecase

    • 

  • use camelCase

    • 

  • global variables use all caps


    
Global variables should only be used only when there really is a need to have

    

    • 




a global variable that is modifieable EVERYWHERE.





Otherwise, don't use global variables.















25 File Importing with Context Manager (with open)






25.1 Importing files the old way (not recommended)





Excerpt From: Mark Lutz. “Learning Python.” iBooks. 





WHen working with files:







#!/usr/bin/env python
 import string
 s = open("/usr2/py/mount.txt","r+")
 for line in s.readlines():
     print line
     string.replace(line, 'mickey','minnie')
     print line

 s.close()












25.2 Better to use Context Manager (with open*)







with open('poetry.txt',   'r')as f:
     file_contents = f.read()







Reading a file into a list, with one list entry per line you can do:







with open(file_of_words, "r")as vardi:
    wordlist = vardi.readlines()






While that works, every list entry in wordlist will have a \n attached to
the string.  If you do NOT want this \n on each string, do this:







with open(file_of_words, "r") as vardi:
    wordlist = [line.rstrip() for line in vardi]






Result is the same, but without the \n at the end for each string.












25.3 open multiple files at once





You can open multiple files with on with statement as follows:







with open('yes.txt', 'r') as yes_file, open('no.txt', 'r') as no_file:









Since python version 3.10 you can write the open statement across multiple
lines as well:







   with (
    open('y.txt', 'r') as yes_file,
    open('n.txt', 'r') as no_file
):
    for line1, line2 in zip(yes_file, no_file):
        print(line1.strip(), line2.strip())












25.4 Context managment is not just for files





Anytime you are managing resources (such as files, threads (manually acquiring and 
releasing locks), databases (opening and closing databases)
you can use "with" to automatically manage the resource, and close things.










25.5 Context manager with multiple files at once





This is often useful too:







def filter(txt, oldfile, newfile):
    '''\
   Read a list of names from a file line by line into an output file.
   If a line begins with a particular name, insert a string of text
   after the name before appending the line to the output file.
   '''

   with open(newfile, 'w') as outfile, open(oldfile, 'r', encoding='utf-8') as infile:
       for line in infile:
       if line.startswith(txt):
           line = line[0:len(txt)] + ' - Truly a great person!\n'
       outfile.write(line)

# input the name you want to check against
text = input('Please enter the name of a great person: ')    
letsgo = filter(text,'Spanish', 'Spanish2')















26 Changes to python 3.0 after 2.7






26.1 The print statement has been replaced with a print() function, with keyword





arguments to replace most of the special syntax of the old print statement
(PEP 3105). Examples:





Old: print "The answer is", 2*2
New: print("The answer is", 2*2)





Old: print x,           # Trailing comma suppresses newline
New: print(x, end=" ")  # Appends a space instead of a newline





Old: print              # Prints a newline
New: print()            # You must call the function!





Old: print >>sys.stderr, "fatal error"
New: print("fatal error", file=sys.stderr)





Old: print (x, y)       # prints repr((x, y))
New: print((x, y))      # Not the same as print(x, y)!
You can also customize the separator between items, e.g.:





print("There are <", 2**32, "> possibilities!", sep="")
which produces:





There are <4294967296> possibilities!





Note:





The print() function doesn’t support the “ the old print statement. 
For example, in Python 2.x, print "A\n", "B" would write "A\nB\n"; 
but in Python 3.0, print("A\n", "B") writes "A\n B\n".
Initially, you’ll be finding yourself typing the old print x a lot in interactive mode. 
Time to retrain your fingers to type print(x) instead!





When using the 2to3 source-to-source conversion tool, all print statements 
are automatically converted to print() function calls, so this is mostly a non-issue 
for larger projects.











26.2 Dictionary Views And Iterators Instead Of Lists





Some well-known APIs no longer return lists:
dict methods dict.keys(), dict.items() and dict.values() return “views” instead of lists.
For example, this no longer works: k = d.keys(); k.sort(). Use k = sorted(d) instead 
(this works in Python 2.5 too and is just as efficient).





Also, the dict.iterkeys(), dict.iteritems() and dict.itervalues()
methods are no longer supported.















old python construct
replaced by





in python2
in python3






dict.iterkeys()
dictkeys()





dict.iteritems()
dictitems()





dict.itervalues()
dictvalues()










d = {'name': 'Richard Feynman', 'iq': 195, 'Nobel_laureate': True}

d.keys()
dict_keys(['name', 'iq', 'Nobel_laureate'])

d.values()
dict_values(['Richard Feynman', 195, True])

d.items()
dict_items([('name', 'Richard Feynman'), ('iq', 195), ('Nobel_laureate', True)])









map() and filter() return iterators. If you really need a list and the 
input sequences are all of equal length, a quick fix is to wrap map() 
in list(), e.g. list(map(…)), but a better fix is often to use a list 
comprehension (especially when the original code uses lambda), or rewriting 
the code so it doesn’t need a list at all. Particularly tricky is map() 
invoked for the side effects of the function; The John transformation 
is to use a regular for loop (since creating a list would just be wasteful).





If the input sequences are not of equal length, map() will stop at the 
termination of the shortest of the sequences. For full compatibility with map() 
from python 2.x, also wrap the sequences in itertools.ziplongest(), 
e.g. map(func, *sequences) becomes list(map(func, itertools.ziplongest(*sequences))).





range() now behaves like xrange() used to behave in 2.7, except it works with values 
of arbitrary size. The latter no longer exists.  xrange goes through the range
1 item at a time, which is what range does as well now in python 3



    

  • same thing with generators and iteritems. With python 3, iteritems does not

    • 




exist, but items behaves like iteritems. 





zip() now returns an iterator.













27 Python help, dir, inspect





help and dir are built-in.  inspect must be imported first.








27.1 dir





You can run the built-in dir() function on any object in python.  It
will give you a list of methods and variables used by that object.





For this example I had to first import requests and import example1  







dir()
  dir(requests)
  dir(requests.model)
  dir(example1.doubler)
  help(requesets.mode1)






These are all good in the interactive Python interpreter.








import example1   # leave off the .py
import re quests
from datetime import datetime

dir(datetime)
dir(datetime.datetime)






Note that writing example1.py, under the def for "doubler" you can add the help text










27.2 help





For stuff you did with dir, you can also try help  for example






import secrets
print(secrets.token_hex(16))

# in interactive python session try:
help(secrets.token_hex)
help(datetime.weekday)
help(example1)
help(example1.doubler)   # where doubler is a function in example1







In my .bashrc I have added:
alias secret='python3 -c "import secrets; print(secrets.token_hex(16))"'





Actually it is in my ~/.config/zsh/zsh-aliases as well.










27.3 inspect





Inspect has many methods itself.  In general it is used to look at any object,
method, class, or module
.







import requests
import inspect

r = requests.get("http://developer.cisco.com/")

inspect.getmembers(r)













27.3.1 inspect.getsource((<any class or module





or object or method>) to retrieve source code






requests.Requests
print(inpsect.getsource(requests.Request))












27.3.2 inspect.getsourcelines





to get the source code line by line







print(inspce.getsourcelines(requests.Request))












27.3.3 inspect.getdoc(<any class or module





or object>)
You can always run request.Request.__doc__ to read the docstrings for any
method.





inspect.getdoc(request.Request) does the same but prettier ??










27.3.4 inspect.getfile(<any class or module





or object>)







inspect.getfile(requests)












27.3.5 inspect.getclasstree





If you have some nested classes, getclasstree= will help you understand the
relationships and heirarchy of these classes.







class top:
   pass

class middle(top):
   pass

class bottom(middle):
   pass






    

  • zptree = inspect.getclasstree([top, middle, bottom]) which takes a list of classes
as its argument


    
Then zptree[0] and zptree[1][1][0] and zptree[1][0], zptree[1][1][1][0] are things to explore.

    

    • 











27.3.6 inspect.signature





sig = inpsect.signature(requests.get)







sig = inpsect.signature(requests.get)
for param in sig.parameters.values():
   print(param.kind)






will show you all the kwargs for a method or class ??










27.3.7 inspect.getfullarpspec





This will to the same as above, but do it all in one shot for all
kwargs and args.











27.3.8 inspect.unwarp





@functools.wraps(func)











27.3.9 inpsect.getgeneratorstate





First lets get a generator defined






def values_I_generated():
    for x in range(10,2):
        yield x

mygen = values_I_generated()
 print(inspect.getgeneratorstate(mygen))

mygen
next(mygen)
next(mygen)
 print(inspect.getgeneratorstate(mygen))
next(mygen)
next(mygen)
 print(inspect.getgeneratorstate(mygen))







A generator can be in one of four states in order of creation to closure:



    

  • created

    • 

  • running

    • 

  • suspended

    • 

  • finished

    • 












27.3.10 inpsect.getgeneratorlocals





inspect.getgeneratorlocals(gen)
next(gen)
inspect.getgeneratorlocals(gen)
next(gen)
inspect.getgeneratorlocals(gen)
next(gen)
to see what is going on in the generator as it runs through its list.










27.3.11 inspect.trace()





Great for debugging complex errors





For example






 def divide(a,b):
    return a/b

 def do_something():
    divide(1,0)

do_something







<insert output here>






So a better approach is:






 def divide(a,b):
    return a/b

 def do_something():
    divide(1,0)

try:
    do_something
except:
    print(inspect.trace())






But this example uses a "bare except" clause, which catches any and all
errors the same way.  Better would be to use the actual exception that
will be thrown, i.e. ValueError: or AttributeError:  





Even better:






 def divide(a,b):
    return a/b

 def do_something():
    divide(1,0)

try:
    do_something
except:
    for frame in inspect.trace():
        print(frame)
        print(frame.code_context, frame.lineno)








Again this still does not specify the actual exception so this would be
best:






 def divide(a,b):
    return a/b

 def do_something():
    divide(1,0)

try:
    do_something
except ValueError:
    for frame in inspect.trace():
        print(frame)
        print(frame.code_context, frame.lineno)








With respect to the ValueError, here is a better example showcasing that:






def simple_except():
    while True:
        try:
            n = input("Input a number: ")
            x = int(n)
            break
        except ValueError:  # be cautious that C-c won't let you exit here
            print(f" You must enter a number.  {n} is not a number.")


















27.4 inspect from the command line





python3 -m inspect requests.Request






~











28 Ordering items in a function (*args before **kwargs)





The order is important when passing arguments in a function.  That is:



    

  1. standard arguments

    2. 

  2. *args arguments

    3. 

  3. **kwargs arguments

    4. 





So this is John:






def monty_function(x, y, *args, **kwargs)
   pass










28.1 use of / as a seperator with python 3.8 and later:





The order still matters, but now we can explicitly tell python where the
positional args end and the keyword kwargs begin like so:







 def calculate_income(rate, hours, /, *, is_contractor=False, is_student=False)
    #do your calculation

#then call the function later with:
calculate_income(65.50, 85, contrctor=True, student=False)








Here the * is used to accept other kwargs as needed, without the code failing.














29 * and ** as unpacking operators





As of python3.5 upacking operators are operators that unpack values from
iterable objects in Python.  The single * operator can be used on any iterable
and the ** operator can only be used on dictionaries.





A contrived example:






cast = ['Cleese', 'Palin', 'Idle', 'Jones', 'Gilliam', 'Chapman']
print(cast)
['Cleese', 'Palin', 'Idle', 'Jones', 'Gilliam', 'Chapman']
# that was still "packaged" as a list

print(*cast)
Cleese Palin Idle Jones Gilliam Chapman
# that was unpacked.  So the print statment takes all the list items
# as though they were single arguments

# see how that is much easier than having to do:
for actor in cast:
    print(actor)
# or even something on one line as:
for actor in cast:
    print(actor, end=" ")









Note that if your function requires a specific number of arguments, then the
iterable you unpack must have the same number of arguments, or you will get
a Type error: myfunction() takes 4 positional arguments, but 5 were given.









29.1 Flexiblity of *args





 *args can take any number of arguments, if your function is written
such that it can handle any number of arguments, you can pass multiple
lists to the function and it will work through all items in all lists.






def total_everything(*args):
    result = 0
    for x in args:
        result += x
    return result

list1 = [1, 2, 3]
list2 = [104, 105]
list3 = [216, 217, 218, 219]

print(total_everything(*list1, *list2, *list3))
# gives an output of 1085















30 Some good tips on installing and package management






30.1 From DevNet basic python programming course





seen here:  part2





import antigravity   # ha ha ha













31 Python with Cisco Networking





The code for net-prog-basics from Cisco DevNet is available from github





Libraries to Work with Data



    

  • XML   
xmltodict  pip install xmltodict
import xmltodict

    • 


  • JSON
import json

    • 


  • YAML
pip install PyYAML
import yam

    • 


  • CSV
import csv

    • 





API Libraries



    

  • REST APIs

    • 

  • NETCONF/YANG

    • 

  • CLI

    • 






Here is another list of 7 python libraries that can be used with networking:



    

  1. Netmiko

    2. 

  2. Paramiko

    3. 

  3. Nornir

    4. 

  4. Telnetlib

    5. 




5)- Pyntc



    

  1. Scapy

    2. 

  2. Subprocess

    3. 








31.1 Libraries to Work with Data






31.1.1 XML





xmltodict  pip install xmltodict
import xmltodict





This does all the heavy lifting, so you don't have to.  What you get is the easy reading of XML input
to create a python dictionary.*  You can also reverse that and take your dictionary, and create XML output.





Python also includes a native Markup (html/xml) interace.  This is more powerful, but also more complex.
*/technically to an OrderedDict/ 







import xmltodict
xml_example = open("xml_example.xml").read()

xml_dict = xmltodict.parse(xml_example)
int_name = xml_dict["interface"]["name"]

print (int_name)

output-file = xmltodict.unparse(xml_dict)













31.1.2 JSON





These commands you will always be using are : 1) import json, 2) json.load and json.loads
and 3) json.dump and json.dumps 








import json

json_example_file = open("json_example.json").read()
json_python = json.load(json_example_file)        # at this point the python dictionary is set up.

# we can then assign variables straight out of the python dictionary by referencing the [key][value] pair.
int_name = json_python["ietf-interfaces:interface"]["name"]

print (int_name)

# back to json for output:
out-file = json.dumps(json_pyton)








JSON looks very similar to what a Python dictionary looks like.  i.e. a series of key: value pairs. 
The keys are usually [2], but can be any data type.  The values of a dictionary are often of
a mixed data type, like [2], intergers, floats, even lists for example





Here's a snippet of JSON:






{
  "points": 500,
  "name": "Introduction",
  "description": "Learn what the Network Programmability Video Course is all about.",
  "mandatory": true,
  "completed": false,
  "lessons": ["Intro", "House Keeping", "Details", "Review", "Conclusion"],
  "cost": null,
  "resources":  {
      "computer": "Macbook Pro",
      "ide": "emacs v 26.2",
      "python": "version 3.6.5",
      "sandbox": "sandbox.cisco.com/intro"
   }







And here's a snippet of python code for the same dictionary:







{
  "": "00-intro",
  "name": "Introduction",
  "description": "Learn what the Network Programmability Video Course is all about.",
  "readme": "https://raw.githubusercontent.com/CiscoDevNet/netprog_basics/master/intro/intro/README.md",
  "image": "",
  "lessons": [
    {
      "lesson": "intro",
      "name": "How 

  #+END_EXAMP

{
  "points": 500,
  "name": "Introduction",
  "description": "Learn what the Network Programmability Video Course is all about.",
  "mandatory": True,
  "completed": False ,
  "lessons": ["Intro", "House Keeping", "Details", "Review", "Conclusion"],
  "cost": none,
  "resources":  {
      "computer": "Macbook Pro",
      "ide": "emacs v 26.2",
      "python": "version 3.6.5",
      "sandbox": "sandbox.cisco.com/intro"
   }







So reviewing, we can load json into a python dictionary using the json module
  (import json)
by using either a file with json, or a string with json code.  Here's how:





json.load(myfile):   Load JSON data from a file or file-like object.
json.loads(mystring): Load JSON data from a string, or variable holding a string.









31.1.3 json.load() or json.loads()  (from json to python dictionary)





json.loads() loads from a string whereas json.load() loads from a json file.






And we can produce json formatted output with the dump command:









31.1.4 json.dump() (from python dictionary to json file)





json.dump writes to a file.  json.dumps writes to a string.







json.dump(myjasonobject,myfile): # Writes JSON object to a file myfile (or file-like object)
json.dumps(myjasonobject): #  Outputs json object as a string

json_file = open("~/json_data/nexus9k-dc1.json","r")
9k-dc1= json.load(json_file)
json_file.close()
print(type(9k-dc1)) # returns json dictionary?  confirm this.









some good examples: ~/bin/python/netprogbasics/programmingfundamentals/dataformats










31.1.5 Some more examples:







print(json.dumps({"name": "John", "age": 30}))
print(json.dumps(["apple", "bananas"]))
print(json.dumps(("apple", "bananas")))
print(json.dumps("hello"))
print(json.dumps(42))
print(json.dumps(31.76))
print(json.dumps(True))
print(json.dumps(False))
print(json.dumps(None))








Converting from Python to JSON, these objects are correspondingly
converted:















PYTHON
JSON






dict
Object






list
Array






tuple
Array






str
String






int
Number






float
Number






True
true






False
false






None
null











   import json

   # this is my Python dictionary
   x = {
     "name": "John",
     "age": 30,
     "married": True,
     "divorced": False,
     "children": ("Ann","Billy"),
     "pets": None,
     "cars": [
       {"model": "BMW 230", "mpg": 27.5},
       {"model": "Ford Edge", "mpg": 24.1}
     ]
   }

   print(json.dumps(x))

   # Results in:
   # {"name": "John", "age": 30, "married": true, "divorced": false, "children": ["Ann", "Billy"], "pets": null, "cars": [{"model": "BMW 230", "mpg": 27.5}, {"model": "Ford Edge", "mpg": 24.1}]}


   print(json.dumps(x, indent=4))
   # Results in:
{
    "name": "John",
    "age": 30,
    "married": true,
    "divorced": false,
    "children": [
        "Ann",
        "Billy"
    ],
    "pets": null,
    "cars": [
        {
            "model": "BMW 230",
            "mpg": 27.5
        },
        {
            "model": "Ford Edge",
            "mpg": 24.1
        }
    ]
}

 print(json.dumps(x, indent=4, sort_keys=True))                                                                                                                                                               
{
    "age": 30,
    "cars": [
        {
            "model": "BMW 230",
            "mpg": 27.5
        },
        {
            "model": "Ford Edge",
            "mpg": 24.1
        }
    ],
    "children": [
        "Ann",
        "Billy"
    ],
    "divorced": false,
    "married": true,
    "name": "John",
    "pets": null
}













31.1.6 Printing json





Several good methods for printing json data.  Actually since first you
json load it into a python dictionary, these methods are how to print
dictionaries in various ways.




    

  1. pprint

    


    from pprint import pprint

# recipes.json is a list with each element a json formatted recipe
# interesting that json.load seems to not care that jsonfile is
# a list of json files, and not just a json file itself.

with open('recipes.json', 'r') as jsonfile:
json_recipes = json.load(jsonfile)

# work with just one recipe, so take the second last recipe
one_recipe = json_recipes[-2]

pprint(json_recipes)

print(json.dumps(json_recipes, indent=4, sort_keys=True))
print(json.dumps(json_recipes, indent=4, sort_keys=True))
print(json.dumps(one_recipe, indent=6))


for key in one_recipe:
    print(key, " : ", one_recipe[key])
    print("---------------------")


for key in one_recipe:
    pprint(key, " : ", one_recipe[key])
    print("---------------------")


    

    

    2. 










31.1.7 YAML0





pip install PyYAML to install first. 





import yaml








import yaml

yml_example = open("yaml_example.yaml").read()
yaml_python = yaml.load(yml_example)        # at this point the python dictionary is set up.

# we can then assign variables straight out of the python dictionary by referencing the [key][value] pair.
int_name = yaml_python["ietf-interfaces:interface"]["name"]

print (int_name)

# back to json for output:
out-file = yaml.dumps(yaml_pyton)













31.1.8 CSV





import csv 





Can import data from any spreadsheet csv output.  Python will treat the CSV data as lists.
You can efficiently process large files without memory issue, even lists of thousands of routers. 
You also get options for header rows, and different formats, so read the docs to 
get details on  importing excel spreadsheets, vs mac numbers, vs plain csv files.







import csv

csv_example = open("csv_example.csv").read()
print ( csv_example)
'"router1","10.1.0.1","New York"\n"router2","10.2.0.1","Denver"\n"router3","10.3.0.1","Austin"\n'

csv_python = csv.reader(csv_example)        # at this point the python dictionary is set up.

for row in csv_python:
    print ("{} is in {} and has IP {}".format(row[0], row[2], 

 router 1 is in New York and has IP 10.1.0.1
 router 2 is in New York and has IP 10.2.0.1
 router 3 is in New York and has IP 10.3.0.1

















31.2 API Libraries





REST APIS use requests
pip install requests





NETCONF APIs use ncclient
pip install ncclient





Network CLI use netmiko
pip install netmiko





YANG 
pip install pyang








31.2.1 REQUESTS (pip install requests, import requests)





Basically gives you a full HTTP client inside your python code.
Takes care of authentication, headers, ajnd response tracking.
Great for REST API calls, or any HTTP reuests.





Example:
myresponse = requests.get(u, headers = headers, auth=(router["user"], router["pass"]), verify=False)













31.3 dir(requests)







 >>> dir(requests)
 ['ConnectTimeout', 'ConnectionError', 'DependencyWarning', 'FileModeWarning', 'HTTPError', 'NullHandler',
 'PreparedRequest', 'ReadTimeout', 'Request', 'RequestException', 'RequestsDependencyWarning', 'Response',
 'Session', 'Timeout', 'TooManyRedirects', 'URLRequired', '__author__', '__author_email__', '__build__', 
 '__builtins__', '__cached__', '__cake__', '__copyright__', '__description__', '__doc__', '__file__', 
 '__license__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__title__', '__url__',
 '__version__', '_internal_utils', 'adapters', 'api', 'auth', 'certs', 'chardet', 'check_compatibility', 
 'codes', 'compat', 'cookies', 'delete', 'exceptions', 'get', 'head', 'hooks', 'logging', 'models',
 'options', 'packages', 'patch', 'post', 'put', 'request', 'session', 'sessions', 'status_codes',
 'structures', 'urllib3', 'utils', 'warnings']
 >>> import pprint
 >>> from pprint import pprint
 >>> 
 >>> router = {"ip": "172.20.20.1", "port": "443", "user": "root", "password": "c1scO123"}
 >>> headers = {"Accept": "application/yang-data+json"}
 >>> u = "https://{}:{}/restconf/data/interfaces/interface=GigabitEthernet1"

 >>> u = u.format(router["ip"], router["port"])

 >>> r = requests.get(u,
           headers = headers,
           auth=(router["user"], router["pass"]),
           verify=False)

pprint(r.text)







From help(requests.models)
Usage::















>>> import requests





>>> req = requests.Request('GET', 'https://httpbin.org/get')





>>> r = req.prepare()





>>> r





<PreparedRequest [GET]>











>>> s = requests.Session()





>>> s.send(r)





<Response [200]>














31.4 help(requests)





Straight from the built-in help() facility after importing requests






Requests is an HTTP library, written in Python, for human beings.
Basic GET usage:

   >>> import requests
   >>> r = requests.get('https://www.python.org')
   >>> r.status_code
   200
   >>> b'Python is a programming language' in r.content
   True

... or POST:

   >>> payload = dict(key1='value1', key2='value2')
   >>> r = requests.post('https://httpbin.org/post', data=payload)
   >>> print(r.text)
   {
     ...
     "form": {
       "key1": "value1",
       "key2": "value2"
     },
     ...
   }













31.4.1 NETCONF/YANG





This is the new model for network programmability that replaces snmp. 
It is a Full NETCONF Manager (i.e. client) in Python.  It handles all details including
authenticatioon, RPC and operations.





And it deals in raw XML.





Note that the port used by netconf is port 830.  The example below is missing the netconf filter
and the router identification dictionary.  See examples from course for this info.







from ncclient import manager
from pprint import pprint
import xmltodict
...
m = manager.connect(host=rotuer["ip"], port=router["port"], password=router["pass"], hostkey_verify=False)

interface_netconf = m.get_config("running", netconf_filter)   # check on the filter seetup too.)
interface_python = xmltodict.parse(interface_netconf.xml)["rpc-reply"]









As you can see, ncclient uses the manager, that needs multiple pieces to connect, i.e. ip address, port, user, passwd, and authenticaitonn










31.4.2 CLI











31.5 Built-in subprocess module





The built-in subprocess module is the preferred way to run system commands and
capture the output in python.  (vs the os module which works, but only for basic
commands, and without the flexibility of the subprocess module.





Some code examples for some of the most common features:






import subprocess
results = subprocess.run(["ls","-ltr"],
                         stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE)

print(results.stdout.decode('utf-8'))  # decode is needed to get the output
# on separate lines, like you would
# see on a command line.  # or if you add the text=True you won't need
# the decode(utf-8) on printing, just print(resultst.stdout)
results_t = subprocess.run(["ls","-ltr"],
                           stdout=subprocess.PIPE,
                           stderr=subprocess.PIPE,
                           text=True)
print(resultst.stdout)

# now outputting to a file:
with open('output.txt',w) as out:
    results_t = subprocess.run(["ls","-ltr"],
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE,
                               text=True)

# the output will be in the file.

# when processes take a long time, it is good to run them in the background
ping_results = subprocess.Popen(["ping", "-n", "15", "zintis.net"],
                           stdout=subprocess.PIPE,
                           stderr=subprocess.PIPE,
                           text=True)

results.poll()   # if returns none, the subprocess is still running
                 # if returns a number, the process has finished.
# afterwards you can look at:
print(results.stdout)















32 Coding Philosophy






32.1 everything in python is an object.





Objects are purpose-built groupings of variables (called attributes) and
functions (called methods) that work together to do something useful.





To access attributes or methods contained within an object, just remember to
always use the dot syntax i.e. mystring.bit_length() or "MeSsedupSTRING".lower()











32.2 Variables are NOT like boxes





A common notion in programming is that a variable is like a box that can hold 
a value,   s = 5  puts a value of "5" into a box labelled "s"
This is NOT TRUE in python.





In Python, it is better to think of the variable as a name/label of an
object itself, not the box holding the object.  In fact each object has an
object ID, that is unique.  See the following:







  >>> arthur_wilson1=('1943-03-29', 'mt.kilamanjaro', ['mountaineer', 'expedition-lead'])

  >>> arthur_wilson2=('1943-03-29', 'mt.kilamanjaro', ['mountaineer', 'expedition-lead'])

  >>> arthur_wilson1 == arthur_wilson2
  True
  >>> arthur_wilson1 is arthur_wilson2
  False
  >>> id(arthur_wilson1), id(arthur_wilson2)
  (4512010820, 4512010856)

# some more examples using sets


In [6]: x = {'A', 'B', 'C'}
In [7]: y = {'A', 'B', 'C'}
In [14]: z = {'C', 'B', 'A'}

In [15]: x
Out[15]: {'A', 'B', 'C'}
In [16]: y
Out[16]: {'A', 'B', 'C'}
In [17]: z
Out[17]: {'A', 'B', 'C'}

In [21]: x == z
Out[21]: True
In [22]: y == z
Out[22]: True
In [23]: x == y
Out[23]: True

In [24]: id(x)
Out[24]: 4415184384
In [25]: id(y)
Out[25]: 4415190208
In [26]: id(z)
Out[26]: 4415186400

# three DIFFERENT objects.
In [37]: x is y
Out[37]: False
In [38]: x is z
Out[38]: False
In [39]: y is z
Out[39]: False

# now I set x to be equal to y.  That will make y the SAME object as x
# it is another "label" to the one and only object with id 4415184384

In [42]: y = x

In [43]: id(x)
Out[43]: 4415184384
In [44]: id(y)
Out[44]: 4415184384









Back to aruther wilson 1 and 2.  They are two different objects that have the
same properties. So they are equal in the sense that the two objects have
identical properties, but they are not the same object.  There are two objects.







>>> eric_idle = arthur_wilson1
>>> eric_idle
('1943-03-29', 'mt.kilamanjaro', ['mountaineer','expedition-lead'])
>>> eric_idle == arthur_wilson1
True
>>> eric_idle == arthur_wilson2
True
>>> eric_idle is arthur_wilson1
True
>>> eric_idle is arthur_wilson2
False








The names eric_idle and arthur_wilson1 are aliases.  Aliases of the SAME
object.  Variables are NOT boxes but names we give to objects.





Without this understanding, the following would not make sense:







>>> skills = eric_idle[2]
>>> skills.append('singer-songwriter')
>>> eric_idle
('1943-03-29', 'mt.kilamanjaro', ['mountaineer', 'expedition-lead', 'singer-songwriter'])







You might think that we are NOT affecting ericidle.  We are just appending
to skills.  But you'd be wrong!!  







That is because the one and only object is ['mountaineer','expedition-lead']
And it had 2 labels: Both 




    

  • "skills"

    • 

  • "eric_idle[2]"

    • 





By changing one, we automatically change the other, because,….. 
there was only ever one.  "You can't bend the spoon because there is no spoon"







>>> eric_idle
('1943-03-29', 'mt.kilamanjaro', ['mountaineer', 'expedition-lead', 'singer-songwriter'])
>>> arthur_wilson1
('1943-03-29', 'mt.kilamanjaro', ['mountaineer', 'expedition-lead', 'singer-songwriter'])







Notice how not only skills and eric_idle[2] were changed, but even the third
label pointing to the same object, "arthur_wilson1" were changed.  Again,
because there was ONLY ONE OBJECT, with all these different labels.  When
"The Great One" scores 4 goals, and 2 assists, Wayne Gretzy stats show 6
more points.










32.3 Tuples are immutable, lists are changeable






32.3.1 Creating from a list







x = tuple(cast)
# where
cast = [eric, john, terry, graham, michael]














32.4 A tuple holds references to objects.





From the above example, the two tuples, eric_idle and arthur_wilson1 are just
references to the SAME objects.   cool-cool!








32.4.1 What is immutable in tuples is the physical content of a tuple





consisting of the object references only!!
The objects inside the tuple can change, as we have seen above, when the 
object that is the skills that Eric Idle brings to the Python team.
(Or ArthurWilson1)












32.5 Tuples that contain lists





More explicitly; a tuple that contains a list, can actually have changes made
to the list, for example, extended, sorted, pop'd  etc..  But you cannot delete
the list because doing so would change the tuple, and tuples are not mutable.











32.6 Python Language Reference (Data Model Chapter)





Not the whole chapter, just a reference:




    

  • "Every object has an identity, a type and a value. An object’s identity

    • 




never changes once it has been created; you may think of it as the object’s
address in memory. The ‘is’ operator compares the identity of two objects;
the id() function returns an integer representing its identity."





After arthurwilson1 became a song-writer, the twins were no longer equal!










32.7 Assignment never copies values.  It ONLY copies references.





By writing skills eric_idle[2], I did not copy the list of skills itself,
just a reference to object. (i.e  the skills that eric_idle[2] referenced)
That's why changing the eric_idle skills by appending to them, automatically
changed arthur_wilson1 skills, because they were really the same object.





Always be thinking variable "arthurwilson1" is assigned to the object.
Do NOT be thinking object is assigned to "arthurwilson1" 





Put simpler, variable "x" is assigned to the object, never the object is
assigned to variable "x".  That also highlights the fact that the object
is created before the assignment.  So,






y = x * 10







The x * 10 object is created first, then this object is assigned variable "y"











32.8 Include default values for functions





Python allows you to specify that a function argument is optional by 
providing a default value for it. While this is a great feature of
the language, it can lead to some confusion when the default value 
is mutable.  So, make sure that each time a function is called, the
default value has not changed and is what you expect it to be.





Python docs state "default value for a function argument is only
evaluated once, at the time that the function is defined."





So, if I define a fuction called "black" as so:
def black(knight[]):=
The list "knight" is set to the default empty list only the first
time black is called.





Subsequent calls to the function "black" that do not specify an
argument, will continue to use the same list to which night was
originally initialized.





Better to use the following…  not sure I understand why.







def black(knight=None):
   if knight is None:      # or if not bar:
      knight=[]
   knight.append("always triumphs")
   return knight













32.9 Examples of function definitions





These include default values, restricting types, and restricting return types


















32.10 Method Resolution Order (MRO)







   >>> class A(object):
...     x = 1
...
>>> class B(A):
...     pass
...
>>> class C(A):
...     pass
...
>>> print A.x, B.x, C.x
1 1 1







Makes sense.







>>> B.x = 2
>>> print A.x, B.x, C.x
1 2 1







Yup, again as expected.







>>> A.x = 3
>>> print A.x, B.x, C.x
3 2 3







What the $%#!&?? We only changed A.x. Why did C.x change too?





In Python, class variables are internally handled as dictionaries and follow
what is often referred to as Method Resolution Order (MRO). 





Since the attribute x is not found in class C, it will be looked up in
its base classes (only A in the above example, although Python supports
multiple inheritances). 





In other words, C doesn’t have its own x property, independent of A. 
Thus, references to C.x are in fact references to A.x. This causes a Python
problem unless it’s handled properly. 













33 MVC Model View Controller





Model-View-Controller






MVC prescribes a way of breaking an application or component into three parts



    

  1. model        # system function and state (on the server)

    2. 

  2. view         # visual feedback to the user

    3. 

  3. controller   # user input

    4. 






While not specificlly Python, it is related and a recommended approach to all
web development, not just for Python.
The term MVC is often encountered in statements such as “framework foo 
supports MVC”. MVC is more about the overall organization of code, rather
than any particular API. Many web frameworks use this model to help the
developer bring structure to their program. Bigger web applications can have
lots of code, so it is a good idea to have an effective structure right from
the beginning. That way, even users of other frameworks (or even other
languages, since MVC is not Python-specific) can easily understand the code,
given that they are already familiar with the MVC structure.





MVC stands for three components:




    

  1. The model. This is the data that will be displayed and modified. In Python
frameworks, this component is often represented by the classes used by an
object-relational mapper.

    2. 

  2. The view. This component’s job is to display the data of the model to the
user. Typically this component is implemented via templates.

    3. 

  3. The controller. This is the layer between the user and the model. The
 controller reacts to user actions (like opening some specific URL), tells
the model to modify the data if necessary, and tells the view code what to
display,

    4. 





While one might think that MVC is a complex design pattern, in fact it is not.
It is used in Python because it has turned out to be useful for creating
clean, maintainable web sites.





Note While not all Python frameworks explicitly support MVC, it is often
trivial to create a web site which uses the MVC pattern by separating the data
logic (the model) from the user interaction logic (the controller) and the
templates (the view). That’s why it is important not to write unnecessary
Python code in the templates – it works against the MVC model and creates chaos
in the code base, making it harder to understand and modify.





See also The English Wikipedia has an article about the Model-View-Controller
pattern. It includes a long list of web frameworks for various programming
languages.











34 Good Coding Conventions





This from several online links, including Hitchhiker's Guide to Python, 
Best Practices Guide for Python, RealPython, google Style Guide, Airbrake.io











35 Structure





In order of appearance in your code your program should have the following:







35.1 Shebang







#!/usr/bin/env python
#!/usr/bin/python







The "Shebang" Line: The first statement in many executable Python scripts
 isn't meant for the Python interpreter.The shebang is interpretted by the
 shell, not python.  Python sees this line as a comment. Ite tells the shell
 attempting to run the script what interpreter should be used to "execute"
 the script.











35.2 Module docstring







'''  string describing module
     as would appear to someone hitting
     help(modulename)
'''






The Module Docstring, the triple-quoted string at the beginning of this Python
script is a module docstring, which serves as a built-in mechanism for
documenting the purpose of and the functionality-provided by the module.





"""Module docstring."""  The interpreter will save this string as a special
_doc__ variable for the module.





Python interpreter stores this string in the __doc__ variable.










35.3 import statement





Import statements import other code into your script so that you can use their
functionality.  All of them up here at the top.  Some imported modules will
have their own subsequent import statements.  That is expected.  However all
classes, functions and methods that are included as part of this program
should put their import statements up here at the top.






When the interpreter encounters an import statement, it opens the module being
imported and (starting with the first line of that module) executes each of
the statements in that file. The contents of that module are then available to
your script through the module's name, using dot-syntax, to access the
variables, functions, and classes within the module.











35.4 constants





Module constants, named by convention using all-CAPS variable names, are
simple variable definitions. Nothing in the Python language makes these
"constant." Their values can be changed. As a community we recognize an
all-CAPS variable name as something that we probably shouldn't change.










35.5 Module-level "Global" Variables:





Every function and class within the module will have at least "read access" to
these variables as they exist at the top-level "global" scope within a module.











35.6 Module Functions and Classes:





As the interpreter encounters new function or class definitions, it creates
and stores them in memory to make them available to subsequent portions of
your code.








def main(*args):
"""My main script function.

Displays the full patch to this script, and a list of the arguments passed
to the script.  """ 

  print(START_MESSAGE) 
  print("Script Location:", location)
  print("Arguments Passed:", args)







Note that the statements within a function (or class) definition are NOT
"executed" when they are defined. They are "loaded" and made available for
future use in your script. You must _call_ a function (supplying any needed
arguments) to execute its block of statements.










35.7 global variables





within the module , these variables are available to all functions and classes,
at least on a read basis, because the exist at the top level of the code.








35.7.1 functions and classes





Each function and class that the interpretor encounters reading in a program
gets stored in memory, ready for use by the main program.  They are NOT
executed, unlike imports of modules.





See Classes are more than just dictionaries. org file for my notes on classes.












35.8 if _name__ == '_main__': block





Some Python files could serve as both a script (to be executed directly)
and, a module (which could be imported). 





Every python program has a built-in dunder variable called _name__ 
__name__ answers the question "what's my name?" which will be different
depending on how the program was run. Either



    

  • 1) directly as a script or

    • 

  • 2) imported as a module by another program.

    • 








35.8.1 Running scripts Directly From Command Line





So if I run any python script, directly from a command line, that script will
have __name__ set to __main__





So you can use a condition if _name__ == '_main__': statement that if it is
true only if the current program was called natively from the command line.






The if __name__ == '__main__' block: 






def main():
    pass

if __name__ == '__main__':
    main()
    # Ture if this program was run DIRECTLY













35.8.2 Running indirectly via the import command





When a Python script is imported by some calling script, the python
interpreter will see __name__ as their own module















runme.py is executed as a *script
muyscript.py is imported as a module






_name__  == 'main'
_name __ == 'myscript.py'





 
 









It does not matter who imported  me and how many times I was imported,
name will always be set to "myscript.py"




if __name__ == '__main__': 
   print(f" I was run directly, hence my name is  __main__ ")
else:
   print(f"  My name is {__name__} and I was imported by some calling script")












35.8.3 Why do I care if I was called directly or imported by another program?





It allows you to run a script directly, and have it run with local parameters,
but also allows the same script to be run as a module imported from some
other python program that could be passing other parameters into a fucnction.










35.8.4 load command line arguments when called directly





If this script is the one being executed, we can take actions like load up any
arguments that were passed to the script like so:







import 
if __name__ == '__main__':
  _, *script_args = sys.argv
  main(*script_args)














35.9 Sample programs mod1.py







#!/usr/bin/env python
'''Module docstring

        Demonstrate the way python uses __name__ and when it is __main__ .
        See also, mod1.py (this module), mod2.py and main-template.py

'''

# Imports
import os
import sys

# Module Constants
ENV_TITLE = "My Silly Walk Script"

# Module "Global" Variables
location = os.path.abspath(__file__)

# Module Functions and Classes
def main(*args):
        """My main script function.

        Displays the full patch to this script, and a list of the arguments passed
        to the script.  (note if this module is imported and run as a module
        ths main() function will never run, because __name__ will NOT be __main__
        so the if statement below will never call this __main__ function.

        """

    print(f"Full path for this script {location}")
    print(f"Arguments Passed {args}")


    # Check to see if this file is the "__main__" script being executed and
    # gather any command-line arguments that were entered with the script name

    if __name__ == '__main__':
            _, *script_args = sys.argv
            main(*script_args)

    else:
       print(f"  My name is {__name__} and I was run by import( {__name__} )")
















35.10 Breaking up long lines of code PEP8 recommendations.





Also follow the conventions in style specified by PEP8
There is more than one way to do it.




    

  1. A long statement:

    2. 







def print_something():
    print 'This is a really long line,', \
      'but we can make it across multiple lines.'







2). Using parenthesis:







def print_something():
    print ('Wow, this also works?',
           'I never knew!')

line1 = "\n This is a long line that would take up most fo the 80 characters"
line2 = "\n allowed on an 80 character code"
# breaking up an f-string that is long
body = ( f"The first line is very long and contains line1: {line1}"
         f"the second line does not have to be so long, but is line2: {line2}"
         )






3). Using \ again:







x = 10
if x == 10 or x > 0 or \
  x < 100:
    print 'True'






Quoting PEP8:





The preferred way of wrapping long lines is by using Python's implied line
continuation inside parentheses, brackets and braces. If necessary, you can
add an extra pair of parentheses around an expression, but sometimes using a
backslash looks better. Make sure to indent the continued line
appropriately. The preferred place to break around a binary operator is after
the operator, not before it.











35.11 Code Respository





When developing a project, a properly structured code repository is key:
For Python these components should all exist before serious dev occurs.







35.11.1 License [root]





Include one of your choice from choosealicense.com









35.11.2 README [root]





Always include one, if for no other reason than to refresh your own memory
when you revisit this years later.  Include:



    

  • describe the project

    • 

  • define its purpose

    • 

  • outline the basic functionality

    • 

  • point out where things don't follow the expected behaviour

    • 









35.11.3 Module Code [/module] or [/bin]





Suggestion is to always place actual code inside a properly name
sub-directory of for a small, single-file project, in the root dir itself.









35.11.4 setup.py [root]





A basic setup.py script is very common in Python projects.
setup.py allows Distutils to properly build and distribute the moduls
needed in your project
See official docs on details of setup.py









35.11.5 requirements.txt [root]





pip uses the requirements.txt file to install all the module





dependancies needed to properly setup a virtual environment.









35.11.6 Documentation [/docs]





Obviously any good projec has proper documentation









35.11.7 Tests [/tests]





Python has very good unit-test capabilities, and the results of those
tests should be kept in its own sub-directory.  See Socratica on Unit
Testing, as well as the file: Python unittests





A basic Python Repository structure might look like this:



    

  • docs/conf.py

    • 

  • docs/index.rst

    • 

  • module/_init_.py

    • 

  • module/core.py

    • 

  • tests/core.py

    • 

  • LICENSE

    • 

  • README.rst

    • 

  • requirements.txt

    • 

  • setup.py

    • 










35.11.8 docs/conf.py





Key component to a well-structured project.  typically placed in /docs
directory.










35.11.9 docs/index.rst







35.11.10 module/_init_.py







35.11.11 module/core.py





Best practices is to create your code under a sub-directory structure,
or under [root] for a single-file project.










35.11.12 tests/core.py







35.11.13 LICENSE





The first thing to add to a project.   see choosealicense.com










35.11.14 README.rst





Get this in place to help you describe your project, define its
purpose, outline its basic functionality










35.11.15 requirements.txt





Usually, if you have several non-standard modules needed by the
project, you can specify them all, one one each line, of this file.
You can specify all the dependencies (modules) needed, and the sub
modules they need, here in one file.  





Later, you can use  pip install -r /requirements.txt/
to install all the modules needed, at once.










35.11.16 setup.py





A common, simple setup script that helps Distutils to properly build
and distribute the modules your project needs.  setup.py is described
in the official documentation















36 Unit Tests or unittest





See the file  unittest.org for an detailed discussion of unittests as well
as the more flexible, modern, pytest module
 .  BTW Cisco's PyATS uses the
pytest module
 .










37 Tips






37.1 Method changing





When two statements can be chained into one.  For example







37.1.1 Two Statemenets





openedurl = urllib2.urlopen(request)
content = openedurl.read()









37.1.2 Equivalent Single Statement





content = urllib2.urlopen(request).read()












37.2 secret keys





To generate a random number or ascii char string of specified length you can use








import secrets
print(secrets.token_hex (16))
















38 Conditional Statements (Comparators)







if variable1 == "blue":
    return ("Yellow")
else:
    return("anycolor but yellow.")


















Equals
a == b





Not Equals
a != b





Less than
a < b





Less than or equal to
a <= b





Greater than
a > b





Greater than or equal to
a >= b





isinstance
a is a particular instance of a class type





is
checks for identity, not equality








Avoid explicitly stating what is Pythons default anyway:















works but awkward
proper and default






if bool(a):
if a





if len(x) != 0:
if x










elif (var1 > 65535) or ((var1 < 1024) and (var1 != 80) and (var1 != 443)):
will not work!!
this will:
elif var == 80 or var == 443 or 1024 <= var <= 65535:





As will:
elif not (var1 == 80 or var1 == 443 or (1024 <= var1 <= 65535)):
i.e. it is often easier to think about the positive case, then wrap it 
in a negative.





You could of course also go all out and be a bit more object-oriented:







class PortValidator(object):
  @staticmethod
  def port_allowed(p):
    if p == 80: return True
    if p == 443: return True
    if 1024 <= p <= 65535: return True
    return False

# .... bunch of code ...
elif not PortValidator.port_allowed(var1):
  # fail








In a conditional statment it looks like this:



    

  • if x == 5:

    • 

  • if x != 5:

    • 










38.1 conditional statements on boolean values.





If you are checking a condition against a boolean value, then the syntax
becomes easier:








38.1.1 boolean true







if myboolean:
  print(f' True is {myboolean}')













38.1.2 boolean false







if not myboolean:
  print(f' False is {myboolean}')















38.2 compound conditional statements





In python a simple and and a or between two conditions make a compound
conditional statement.  Logic rules apply:







import sympy.ntheory as nt

print(nt.isprime(31)
# returns True

print (" All the compound numbers (i.e. NOT prime) that end in 7 up to 500")
for x in range(1, 500, 2):
      # no need to look at even numbers
      if not nt.isprime(x) and x % 10 == 7:
          print(f'  {x:{3}} ends in 7 but is NOT prime')

          factorsof = [y for y in range(2,int(x/2)+1) if x%y==0]
          print(f'  {x:{3}} has these factors: \n {factorsof}')


          pfactorsof = [y for y in range(2,int(x/2)+1) if x%y==0 and nt.isprime(y)]      
          print(f'  {x:{3}} has these PRIME factors: \n {pfactorsof}')


# a cleaner approach:
compound_sevens = [z for z in range(1, 500, 2) if z % 10 ==7 and not nt.isprime(z)]
for i in compound_sevens:
      pfactorsof = [y for y in range(2,int(i/2)+1) if i%y==0 and nt.isprime(y)]
      # read this as the array of y's where y up to i whenever i evenly divides into y,...
      # and take only those y's that are prime.
      print(f'   {i:{4}} has these PRIME factors : {pfactorsof}')


Print (" All the compound numbers (i.e. NOT prime) that end in 7 up to 500")
for x in range(1, 500, 2):
      # no need to look at even numbers
      if not nt.isprime(x) and x % 10 == 7:
          print(f'  {x:{3}} ends in 7 but is NOT prime')
          print(f'  {x:{3}} has these PRIME factors:')
          factorsof = [y for y in range(2,int(x/2)+1) if x%y==0 and nt.isprime(y)]
          print(factorsof)



For x in range(1, 100, 2):
      if not nt.isprime(x):
          print(f'  {x:{3}} is not prime')


for x in range(1, 1000, 2):
      if nt.isprime(x):
           print(f'   {x:{3}} is prime')

for x in range(1, 1000, 2):
      if not nt.isprime(x):
           print(f'   {x:{3}} is prime')


















39 Ternary statements (x = 1 if condition else 0)





Ternary assigment statements are usually just as easy reading as if blocks.
For instance the statement x = 1 if condition else 0
Could be written:







if condition:
   x = 1
else:
   x = 0












40 Exception Handling





Best to start at the docs.python.org site, chpter 8, Errors and Exceiptions





In python all exceptions must be instances of a class that derives from
BaseException





The built-in exceptions listed below all have an "associated value" indicating
the cause of the error.





Users are encouraged to derive new exception classes (custom exceptions) from
the exception class or one of its subclasses.





Code:






try:
except:
finally:







WHen raising an exception, the implicit exception context can be supplemented
with an explicit cause by using from  i.e.



    

  • raise new_exc from original_exc
where the statement following "from" must be an exception, or None

    • 





Here is a list of built-in error types,





    

  • KeyboardInterrupt

    • 

  • GeneratorExit

    • 

  • ArithmeticError

    • 

  • AssertionError

    • 

  • BufferError

    • 

  • EnvironmentError

    • 

  • EOFErro

    • 

  • ImportError

    • 

  • IndexError

    • 

  • IOError

    • 

  • LookupError

      

    • KeyError

      • 

    • IndexError

      • 

    

    • 

  • MemoryError

    • 

  • ModuleNotFoundError

    • 

  • NameError

    • 

  • OSError (obsoletes IOError, socket.error, EnvironmentError, select.error)

    • 

  • RecursionError

    • 

  • ReferenceError

    • 

  • RuntimeError

    • 

  • SyntaxError

      

    • IndendtationError

      • 

    

    • 

  • SystemError

    • 

  • TypeError

    • 

  • ValueError

    • 

  • WindowsError

    • 

  • ZeroDivisionError

    • 








40.1 catching errors gracefully with try: except





First an example of good exception handling to catch type errors.







try:
    l = ["a", "b"]
    int(l[2])
except ValueError, IndexError:  # To catch both exceptions, right?
    pass








Another example showing difference when raising and not raising an error:
In the first examle, I raise the ValueError:






def cat(filename: str):
    try:
        with open(filename) as file:
            data = file.read()
        return data
    except FileNotFoundError:
        raise ValueError('filename cannot use special characters')
        pass        







This will cause the script to halt with the ValueError printed.



File "/Users/zintis/bin/python/bin/flask/flask_app.py", line 19, in cat
   raise ValueError('filename cannot use special characters')
ValueError: filename cannot use special characters





Now if we do NOT raise the error:






def cat(filename: str):
 try:
     with open(filename) as file:
         data = file.read()
     return data
 except FileNotFoundError
 pass     







The script does NOT terminate in an error, but is allowed to continue.
The result is that the cat function will simply return None












40.2 Reading a stack trace





The stack is all the modules and classes that were called at the place of
the error found.  Easiest is to read the last line first. Then move your way
back up and in so doing, travel back in who called me to get me to this
point.






  1. The last line: "What went wrong?"
    

    
Start here.

    

    

    2. 


  2. Revew the stack from the top to the bottom
    

    
Even though you read the "last" line, or the bottom to start off, hwen you
have done that, the next best approach is top down.

    


    
Read each level to see where the problem started.

    





    
Traceback (most recent call last):
  File "<stdin>", line 3, in <module
 >
  IndexError: list index out of range

    


    
The problem here is that the except statement does not take a list of
exceptions  specified in this manner. Rather, In Python 2.x, the syntax
except Exception, e is used to bind the exception to the optional
second parameter specified (in this case e), in order to make it available
for further inspection. As a result, in the above code, the IndexError
exception is not being caught by the except statement; rather, the
exception instead ends up being bound to a parameter named IndexError.

    



    
The proper way to catch multiple exceptions in an except statement is to
specify the first parameter as a tuple containing all exceptions to be caught.
Also, for maximum portability, use the as keyword, since that syntax is
supported by both Python 2 and Python 3:

    



    


    >>> try:
...     l = ["a", "b"]
...     int(l[2])
... except (ValueError, IndexError) as e:  
...     pass
...
>>>

    

    

    










41 Python exception handling, BaseException





Python has a built-in set of classes for error and exception handling.  They
are aptly named BaseException which include the popular TypeError and
ValueError.
See how these are utilized in Unit Tests section above.







 try:
    if datetime.strptime(to_date, "%Y-%m-%d") < datetime.strptime(
        from_date, "%Y-%m-%d"
    ):
        raise ValueError(
            "Error: The second date input is earlier than the first one"
        )
except ValueError:
    raise ValueError(
        "Error: InJohn format given for dates. They must be given like 'yyyy-mm-dd' (ex: '2016-10-01')."
    ) 










41.0.1 raise and except





raise errors can be used to create your own error if you want your code to
stop and raise an error if a condition arises:






if mycondition:
   raise Exception('My condition was met, so I am raising an exception')

if (a < 0.001):
   raise ValueError()


try:
   f = open('filename', 'r')
except IOError:
   # additional commands you want to run here
   # and only then pass the error on
   raise







More complex example showing raise and except, where you can use raise to
reraise the current exception in an exception handler so that it can be
handled further up the call stack.  - not sure what this means for now.






try:
    if datetime.strptime(to_date, "%Y-%m-%d") < datetime.strptime(
        from_date, "%Y-%m-%d"
    ):
        raise ValueError(
            "Error: The second date input is earlier than the first one"
        )
except ValueError:
    raise ValueError(
        "Error: InJohn format given for dates. They must be given like 'yyyy-mm-dd' (ex: '2016-10-01')."
    ) 















42 Python Scope Rules (LEGB)





Local, Enclosing, Global, Built-in  LEGB
 From developer.cisco.com:







#!/usr/bin/env python
"""Demonstrate module
 vs. locally scoped variables."""

# Create a module
 variable
 module_variable = "I am a module variable."

# Define a function that expects to receive a value for an argument variable
def my_function(argument_variable):
    """Showing how module, argument, and local variables are used."""
    # Create a local variable
    local_variable = "I am a local variable."

    print(module_variable, "...and I can be accessed inside a function.")
    print(argument_variable, "...and I can be passed to a function.")
    print(local_variable, "...and I can ONLY be accessed inside a function.")

# Call the function; supplying the value for the argument variable
my_function(argument_variable="I am a argument variable.")

# Let's try accessing that local variable here at module
 scope
print("\nTrying to access local_variable outside of its function...")
try:
    print(local_variable)
except NameError as error:
    print(error)











42.1 Local Scope





inside a class or function, and can only be accessed by statements executing
within the same function that created the variable.





Here is an example of a scoping error:






x = 10
def foo():
    x += 1
    print x

foo()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in foo
UnboundLocalError: local variable 'x' referenced before assignment







When you make an assignment to a variable in a scope, like x = x + 1, which
is what x += 1 is doing, this is considered to be local in the function foo
example, that variable is automatically considered by Python to be *local* to
   that scope and shadows any similarly named variable in any outer scope.





i.e. within foo, x is being assigned to x + 1, but foo does not yet know
what 'x' is to add 1 to.  it's NOT the x = 10, as that is in the global scope.





This example taken from toptal.com





It is particularly common for this to trip up developers when using lists. 
Consider the following example:







lst = [1, 2, 3]
def foo1():
    lst.append(5)   # This works ok...
    # because lst is not defined here in the function foo1, so
    # python simply uses the globally defined lst.

foo1()
print(lst)
# will print [1, 2, 3, 5]
###########################################

lst = [1, 2, 3]
def foo2():
    lst += [5]      # ... but this bombs!
    # because lst is being assigned to 'lst + 5', so within foo2
    # lst is a local variable.  By itself that is not yet the problem,
    # but the assignment does not yet have a value for lst to which
    # to add the list [5] to, as lst += [5] is really lst = lst + [5] 

foo2()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module
>
  File "<stdin>", line 2, in foo
UnboundLocalError: local variable 'lst' referenced before assignment








Huh? Why did foo2 bomb while foo1 ran fine?





The answer is the same as in the prior example problem but is admittedly more
subtle.




    

  • foo1 is not making an assignment to lst, whereas foo2 is.

    • 





Remembering that lst += [5] is really just shorthand for lst = lst + [5], we
see that we are attempting to assign a value to lst (therefore presumed by
Python to be in the local scope).





However, the value we are looking to assign to lst is based on lst itself
(again, now presumed to be in the local scope), which has not yet been
defined
Boom.











42.2 Enclosing scope







42.3 Global scope a.k.a. module





Defined at the top of the module , and is available to every function and
class created within this module.










42.4 Built-in scope





A no brainer.  Read-only, never changes.













43 quit() sys.exit() exit() os.exit(0) break





Code:






from sys import exit
exit()







A example in an if clause






if "ATG" in my_DNA:
   ## do something
else:
   print("Start codon is missing.  Check your DNA sequence!"_
   exit()







break only exits a loop, and the script continues to run.





An interactive loop that needs a keyboard interupt to continue uses break.
For example:






from time import sleep
while True:
  try:
     print("If you have enough of this, hit ctrl-c  ")
     sleep(1)
  except KeyboardInterrupt:
     break









sys.exit() exits the code with 0 as the exit code (normal exit)
sys.exit(1) exits the code with 1 as the exit code (error exception exit)






These all kill the python interpreter.  Therefore, if they appear in a script
called from another script by execfile(), it stops execution of both scripts.





While sys.exit() is preferred, as it is more 'friendly' to other code, it does
raise an exception.  Also this only kills the running thread.





os._exit()  terminates immediately and does NOT perform any normal tear-down
and kills the whole process.










44 List Comprehensions (see also Dictionary Comprehensions)





Before going on note that in most cases comprehensions obsolete map, reduce,
filter functions.





Python incorporates a number of elegant programming paradigms which, when used
properly, can result in significantly simplified and streamlined code. A side
benefit of this is that simpler code is less likely to be bitten by the
accidental-deletion-of-a-list-item-while-iterating-over-it bug. One such
paradigm is that of list comprehensions. Moreover, list comprehensions are
particularly useful for avoiding this specific problem.





They are lists.  So, they use square brackets. [ ] 
But instead of listing the elements one by one and separated by commas like
python = [Cleese, Jones, Palin, Gilliam, Chapman, Idle]





You have the elements created automatically in a for loop within the list itself.








44.1 List Comprehension syntax




    

  • [ /expr/ for /val/ in /collection/ ]

    • 

  • [ /expr/ for /val/ in /collection/ if <test> ]

    • 

  • [ /expr/ for /val/ in /collection/ if <test> and <test> ]

    • 

  • [ /expr/ for /val1/ in /collection1/ and  /val2/ in /collection2/ ]

    • 






The expr generates elements of the list
The for loop evaluates this express over a collection of data, however formed.
i.e. could be a range, could be another list that holds the collection.





If you want to create the list from a subset of elements in the collection,
you can add an if clause whose condtions will need to be met for the element
to be added to the list.








[ expr for val in collection if < test > ]
[ expr for val in collection if < test > and < test2 > ]
[ expr for val in collection and val2 in collection2 ]









>>> odd = lambda x : bool(x % 2)
>>> numbers = [n for n in range(10)]
>>> numbers[:] = [n for n in numbers if not odd(n)]  # ahh, the beauty of it all
>>> numbers
[0, 2, 4, 6, 8]







List comprehensions and generator functions, explained below, are
powerful enough that the python built-in map, reduce, filter
functions are not really needed anymore!  They are still explained
below as well, but consider list comprehensions, as they are typically 
more readalbe.










44.2 Create a list using List Comprehensions





This is also known as a "constructor"  see Constructors below.  It makes
creating lists very easy and fast.







squares-to-100 = [ expression for val in range ]
squares-to-100 = [i**2 for i in range(1,101)]
[(i, i**2) for i in range(0,201)] 







So to create a list of tuples where the first element of each tuple is i
and the second element is i**2 you can do this, (and similar things)







sqr_dict = [(i, i**2) for i in range(0,201)]
sqr_dict_mod5 = [(i, i**2%5) for i in range(0,201)]

coolsquares = *[i**2 for i in range(1, 101)]*
coolersquares = *[i**2 for i in range(1, 101) if i**2 < 1000]*
coolestsquares = *[{i, i**2} for i in range(1, 101) if i**2 < 1000]*













44.3 Scalar multiplication of a vector





If a vector is a list of three elements, say v = [3, 5, -9]
Then if you try v * 4 it will give you [3, 5, -9, 3, 5, -9, 3, 5, -9, 3, 5, -9]
that is typically not what you want.   Instead if you say:






v_times_4 =  [4*i for i in v]






will result in  [12, 20, -36]










44.4 Cartesian Product





if w = [-2, -3, 0] the v X w (cartesian product) is:






cart_prod = [(i,j) for i in v for j in w] 













44.5 Generator expressions when lists are very large





So while list comprehensions are usually the preferred approach,
when the lists are very large, they can use lots of memory as the
entire list is generated for each value in the iterable.





Here is a list comprehension again:






coolsquares = [i**2 for i in range(1, 10_000_000_001)]







Could result in a Out-of-Memory error.  So replace the [ with ( and you have
a generator instead.







coolsquaregen = (i**2 for i in range(1, 10_000_000_001))







You can then access every subsequent element of the generator with the next
function or by a for loop.






coolsquaregen = (i**2 for i in range(1, 10_000_000_001))
print(next(coolsquaregen))
print(next(coolsquaregen))
print(next(coolsquaregen))

for x in coolsquaregen:
    print(x)







Just remember that generators will remember where you were, so you can't go
back. i.e. generators are stateful.










44.6 When range() is a mistake





If you have an iterable data structure, like a list, avoid range(len(mylist))
So printing each element of a list using and an index, say i,  is awkward.
Don't do this:






mylist = ["Holy Grail", "Life of Brian", "Meaning of Life"]
for i in range(len(mylist)):
    print(mylist[i])







Python lets you just say:






for i in mylist
    print(i)






The result is identical, but the later is much easier.








44.6.1 enumerate





A slight addition to this is when you want to include the index for some
reason, and not just use it to loop through the list, you can use enumerate.
Put in another way, python lets you access a counter whenever you iterate
over an iterable i.e. whenever you loop.  





Don't do this:






for i in range(len(mylist)):
    print("Index: ", i, "Value ", mylist[i])







Use enumerate instead:






mylist = ["a", "b", "c", "d"]
for i, v in enumerate(mylist):
    print("Index: ", i, "Value ", v)






results in:



Index:  0 Value  a
Index:  1 Value  b
Index:  2 Value  c
Index:  3 Value  d















44.7 List built-in function, filter(), map(), reduce()





Might be deprecated in the future, in favour of comprehensions
From docs.python.org :










44.8 filter(function, sequence)





filter(function, sequence) returns a sequence consisting of those items from
the sequence for which function(item) is true. If sequence is a str, unicode
or tuple, the result will be of the same type; otherwise, it is always a list.





For example, to compute a sequence of numbers divisible by 3 or 5:







>>> def f(x): return x % 3 == 0 or x % 5 == 0
...
>>> filter(f, range(2, 25))
[3, 5, 6, 9, 10, 12, 15, 18, 20, 21, 24]









list(filter(lambda x: x>2, n))   *this is a different example 













44.9 map(function, sequence)





Given an iterable (i.e. some list, or tuple, or other that is iterable) and
 funciton, you can map a function on this iterable to get a new object that is
 iterable.  This new object can be made into a list with list(object)





if the sequence is a list [x1, x2, x3]
and your function os "f"
you get back [f(x1), f(x2), f(x3)]   well you get an object that when iterated
will give you f(x1), f(x2), f(x3).











44.9.1 map and filter have been deprecated with list comprehensions since Python3





Apply some function to each element of a list?  Then use the map function.
list, [m,n,p] function, f()  ==> map ==> Newlist, [f(m), f(n), f(p)]





The syntax is map(f, data)





map(function, sequence) calls function(item) for each of the sequence’s items
and returns a list of the return values. For example, to compute some cubes:







>>> def cube(x): return x*x*x    * this could be a lambda function here
...
>>> map(cube, range(1, 11))
[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]







More than one sequence may be passed; the function must then have as many
arguments as there are sequences and is called with the corresponding item
from each sequence (or None if some sequence is shorter than another). 
For example:







>>> seq = range(8)
>>> def add(x, y): return x+y
...
>>> map(add, seq, seq)
[0, 2, 4, 6, 8, 10, 12, 14]







I like to think of it in terms of math:
for a function f and data a1, a2, ..., an





map(f, data):





returns: f(a1), f(a2), f(a3), ..., f(an)












44.10 map(function, sequence) with lambdas:





The above examples repeated now, using lambdas







map(lambda x: x**3, range(1,11))
map(lambda x,y: x+y, range(8), range(8))











44.11 <map object at 0x1103e2110>





If you expected to see a list printed, but instead got a message similar
to this, then the map was done Johnly.  However, since python 3, the
map function was changed to return a map object instead of a list.  Thus
to get a list, you have to wrap it in a list() call.







cubedn = list(map(lambda x: x**3, range(1,11)))

def factorial(n):
   '''Returns n!'''
   return 1 if n < 2 else n * factorial(n-1)

list(map(factorial, range(11))
#  results in:
#  [1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800]

list(map(factorial, range(1-10)))
#  results in:
#  [1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800]













44.12 Examples of map functions replaced by list comprehensions:







import math
list(map(factorial, range(11))
[1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800]

[factorial(n) for n in range(11)]
[1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800]


list(map(factorial, filter(lambda n: n%2, range(11))))
[1, 6, 120, 5040, 362880]

[factorial(n) for n in range(11) if n % 2]
[1, 6, 120, 5040, 362880]

# list comprehensions seem easier as in:
y =  ["A", "B", "C"]
[x.lower() for x in ["A", "B", "C"]]
[x.upper() for x in ["a", "b", "c"]]
[x.lower() for x in y]   

# same thing as:
list(map(lambda x: x.lower(), ["A", "B", "C"]))
list(map(lambda x: x.upper(), ["a", "b", "c"]))















44.13 reduce(function, sequence)






44.13.1 reduce has been depracted with sum(), any() and all() since Python3





Reduce has the idea to apply some operation to successive items in a
sequence, accumulating previous results, thus reducing a sequence of
values to a single value.





reduce(function, sequence) returns a single value constructed by calling the
binary function function on the first two items of the sequence, then on the
result and the next item, and so on. For example, to compute the sum of the
numbers 1 through 10:







>>> def add(x,y): return x+y
...
>>> reduce(add, range(1, 11))
55








If there’s only one item in the sequence, its value is returned; if the
sequence is empty, an exception is raised.





A third argument can be passed to indicate the starting value. In this case
the starting value is returned for an empty sequence, and the function is
first applied to the starting value and the first sequence item, then to the
result and the next item, and so on. For example,







>>> def sum(seq):
...     def add(x,y): return x+y
...     return reduce(add, seq, 0)
...
>>> sum(range(1, 11))
55
>>> sum([])
0








Don’t use this example’s definition of sum(): since summing numbers is such
a common need, a built-in function sum(sequence) is already provided, and
works exactly like this.












44.14 Replacing reduce with built-in higher order functions sum(), all(), any()





In fact, since Python 3.0 reduce has been demoted to the functools and must
be imported first if you insist on using reduce.  Because the most common
usage of reduce was to sum up values, Python 3 now includes the built-in
sum() funciton.  Similarily with all() and any().





An iterable is anything over which one can loop, like lists, ranges, 
dictionaries, any others?








44.14.1 sum(iterable)





from functools import reduce
from operator import add
reduce(add, range(100)
4950





That can be replaced with:
sum(range(100))
4950










44.14.2 all(iterable)





Returns True if every element of th e itrable is truthy. all([]) returns
True.  Similar to boolean "and".




all([0,1,2]) returns False
all([1,2,3]) returns True
all("string1", "") returns False
all("string1", "string2") returns True










44.14.3 any(iterable)





Returns True if any element of the iteralbe is truthy. any([]) returns
False.  Similar to boolean "or".





any([0,1,2]) returns True
any([0,0,0]) returns False
any([]) returns False

















45 Dictionary Comprehensions





Similar to list comprehensions, but as of python3.8 and "ordered dicts" which
are now also iterable, you can now have dictionary comprehensions.





So if these are list comprehensions:



    

  • coolsquares = [i**2 for i in range(1, 101)]

    • 

  • coolersquares = [i**2 for i in range(1, 101) if i**2 < 1000]

    • 

  • coolestsquares = [{i, i**2} for i in range(1, 101) if i**2 < 1000]

    • 




Then these are dictionary comprehensions:



    

  • coolerstofall  = {i: i * i for i in range(10)}

    • 





Rather than showing you coolestsquares and coolestofall, best to run these
yourself in an interactive python interpreter, such as ipython.











46 Set Comprehensions





See Set comprehensions   










47 Comprehensions (A Review)





Simple syntax summary of all the comprehensions.






dict_comp = {i: i*i for i in range(16)}
list_comp = [i*i for i in range(16)]
set_comp  = {i*i for i in range(16)}
gen_comp  = (i*i for i in range(16))













48 Other Resources.





The Network Programming Basics Devnet course had a good document on getting your
workstation / developer environment set up properly.  There is lots of good advice 
in the workstation setup link on github.








48.1 Computerphile





I got this example from youtube:





def nat(n) :
   yield n
   yield from nats(n+1)





s = nats(2)





next(s)
next(s)
next(s)
…






def sieve(s):
   n = next(s)
   yield n
   yield from sieve(i for i in s if i%n!=0)





prime = sieve(nats(2))





next(prime)
next(prime)
next(prime)
next(prime)
…

















49 Upgrade python on Linux:





Upgrading on Mac I used brew.  See Upgrading python used by a particular venv
section above.  Upgrading on my Centos8 host I used dnf.  No suprise here.





Mostly got my instructions from linuxize.com.




    

  1. mkdir build

    2. 

  2. cd build

    3. 

  3. wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tgz

    4. 

  4. dnf -y groupinstall "Development Tools"

    5. 

  5. dnf install bzip2-devel expat-devel gdbm-devel ncurses-devel openssl-devel readline-devel wget sqlite-devel tk-devel xz-devel zlib-devel libffi-devel

    6. 

  6. tar -xzf Python-3.8.5.tgz

    7. 

  7. cd Python-3.8.5

    8. 

  8. ./configure --enable-optimizations

    9. 

  9. nproc  # to check how many processors I have.  then I can make -j #ofprocs.
> for me it was 6

    10. 

  10. make -j 6

    11. 

  11. make altinstall   ( you could also just try make install ) See altinstall.

    12. 

  12. which pip38

    13. 

  13. which python38

    14. 

  14. python38 –version  #(3.8.3)

    15. 

  15. python3.8 –version #(3.8.5)

    16. 

  16. cd etc/alternatives

    17. 

  17. ls -l python3

    18. 

  18. update-alternatives –install /usr/bin/python3 python3 /usr/local/bin/python3.8 1
this did not work (no error, but just did not seem to make any changes

    19. 

  19. ln -s /usr/local/bin/python3.8 python3

    20. 

  20. ln -s /usr/local/bin/python3.8 python3

    21. 

  21. python3 –version  # now shows 3.8.5'

    22. 





Regarding 18) above, I did get this to work as follows;





#+BEGINEXAMPLE_
root@c8host /etc/alternatives[1059]$
update-alternatives –config python3





There are 2 programs which provide 'python3'.





Selection    Command





    

  • 1           /usr/bin/python3.6

      

    • 2           /usr/local/bin/python3.8

      • 

    

    • 





Enter to keep the current selection[+], or type selection number: 1       
root@c8host /etc/alternatives[1059]$
update-alternatives –config python3





There are 2 programs which provide 'python3'.





Selection    Command






*+ 1           /usr/bin/python3.6
   2           /usr/local/bin/python3.8





Enter to keep the current selection[+], or type selection number: 2   
root@c8host /etc/alternatives[1059]$
update-alternatives –config python3





There are 2 programs which provide 'python3'.





Selection    Command





    

  • 1           /usr/bin/python3.6

      

    • 2           /usr/local/bin/python3.8

      • 

    

    • 





Enter to keep the current selection[+], or type selection number: 
root@c8host /etc/alternatives[1059]$





#+ENDEXAMPLE








49.1 altinstall vs install





from stackoverflow.com





install:         altinstall bininstall maninstall
It does everything altinstall does, along with bininstall and maninstall





Here's bininstall; it just creates the python and other symbolic links.







# Install the interpreter by creating a symlink chain:
#  $(PYTHON) -> python2 -> python$(VERSION))
# Also create equivalent chains for other installed files
bininstall:     altbininstall
     -if test -f $(DESTDIR)$(BINDIR)/$(PYTHON) -o -h $(DESTDIR)$(BINDIR)/$(PYTHON); \
     then rm -f $(DESTDIR)$(BINDIR)/$(PYTHON); \
     else true; \
     fi
     (cd $(DESTDIR)$(BINDIR); $(LN) -s python2$(EXE) $(PYTHON))
     -rm -f $(DESTDIR)$(BINDIR)/python2$(EXE)
     (cd $(DESTDIR)$(BINDIR); $(LN) -s python$(VERSION)$(EXE) python2$(EXE))
     ... (More links created)







And here's maninstall, it just creates "unversioned" links to the Python manual pages.







# Install the unversioned manual pages
maninstall:     altmaninstall
     -rm -f $(DESTDIR)$(MANDIR)/man1/python2.1
     (cd $(DESTDIR)$(MANDIR)/man1; $(LN) -s python$(VERSION).1 python2.1)
     -rm -f $(DESTDIR)$(MANDIR)/man1/python.1
     (cd $(DESTDIR)$(MANDIR)/man1; $(LN) -s python2.1 python.1)








TLDR: altinstall skips creating the python link and the manual pages linksg,
install will hide the system binaries and manual pages.














50 Upgrading python within a venv > pytyon3.3





With python as of version 3.3, python includes the ability to upgrade within
a previously created venv.  simply put:



    

  • python3 -m venv --upgrade ENV_DIR

    • 




But you might have to specify a specific version of python, like:



    

  • python3.8 -m venv --upgrade ENV_DIR

    • 





Some have had to manually update symlinks to the John python version. Also
remember to backup your existing venv with pip freeze > requirements.txt





Someone on Stackoverflow created this workflow:




If you're using Homebrew Python on OS X, first deactivate all virtualenv, then upgrade Python:

brew update && brew upgrade python
Run the following commands (<EXISTING_ENV_PATH> is path of your virtual environment):

cd <EXISTING_ENV_PATH>
rm .Python
rm bin/pip{,2,2.7}
rm bin/python{,2,2.7}
rm -r include/python2.7
rm lib/python2.7/*
rm -r lib/python2.7/distutils
rm lib/python2.7/site-packages/easy_install.*
rm -r lib/python2.7/site-packages/pip
rm -r lib/python2.7/site-packages/pip-*.dist-info
rm -r lib/python2.7/site-packages/setuptools
rm -r lib/python2.7/site-packages/setuptools-*.dist-info
Finally, re-create your virtual environment:

virtualenv <EXISTING_ENV_PATH>
By doing so, old Python core files and standard libraries (plus setuptools and
pip) are removed, while the custom libraries installed in site-packages are
preserved and working, as soon as they are in pure Python









50.1 Safest and Cleanest method of upgrading python in a venv





By far the safest method that also allows you to backout at any time to a
fully working older environement is to not upgrade, but create a new venv.





First you would activate the old venv, and create a requirements.txt from
a pip freeze:   pip freeze > requirements.txt





Then deactivate the current venv (old venv) and create the new venv with
the newer version of python.  Then activate the new venv, and install all
the modules from the requirements.txt file you had saved within the old
venv.  So: pip install -r requirements.txt










50.2 venv –upgrade





With newer python version, you could try: python3.8 -m venv --upgrade <path>
So for example python3.8 -m venv --upgrade ~/bin/python/.venv-3.7  See the
python3.8 -m venv --help  Of course that assumes python has been upgraded in
place.












51 5 common mistakes to avoid






51.1 1. mixing tabs and spaces i.e. indentation errors.





A proper IDE, such as emacs, or VS, will prevent this error.  Text editors
could be the source of these problems.  You will get an indentation error





pylint could also be used to fix these.










51.2 2. naming modules the same as imported modules.





Which one will be picked? So use different names for your own created modules.
So, if you want to import a module
 called math, don't name your own module
'math.py' You will get an error "Cannot import radians from math.py".





This is more common when you use modules such as flask.  Always check your
path, but as explained here, always check what you called your own module
.





Same is true for variables, that you might inadvertently pick a name that
matches a variable, or a module
 from an imported





Error is "float object is not callable" or similar errors.










51.3 3. mutable default arguments





So, rather than
creating a new empty list each time we addemployee using the function below,
we only created the new empty list the first time, then added to the same
list on subsequent calls.








def add_to_cast(actor, actor_list=[]):
    actor_list.append(actor)
    print(actor_list)

add_to_cast('Cleese')
print(actor_list)
['Cleese']

add_to_cast('Palin')
print(actor_list)
['Cleese', 'Palin']

add_to_cast('Chapman')
print(actor_list)
['Cleese', 'Palin', 'Chapman']

add_to_cast('Idle')
print(actor_list)
['Cleese', 'Palin', 'Chapman', 'Idle']








This will show all four actors, even though each time you called addtocast
you did without specifying the list to add to, so you might think that
you are adding to a new empty list each time. Not so!






A fix to that problem is to use "None" and then if "none, add to empty list.







def add_t
    if actor_list is None
        actor_list = []
    actor_list.append(actor)

add_to_cast('Cleese')
print(actor_list)
['Cleese']


add_to_cast('Palin')
print(actor_list)
['Palin']

add_to_cast('Chapman')
print(actor_list)
['Chapman']

add_to_cast('Idle')
print(actor_list)
['Idle']










51.3.1 Same issue with time.





You will notice that even though I am calling the whattime function
multiple times, the returned value is the SAME time as the first time I 
called the function.  







import time
from datetime import datetime

def whattime(time=datetime.now()):
    print(time.strftime('%B %D, %Y %H:%M:%S'))

whattime()
time.sleep(3)
whattime()
time.sleep(3)
whattime()







This will display the SAME time all three calls. 





The fix for this is like above, set the default function to "None"
the within the body of the function, check for none, and then set the 
time to "now"







def whattime(time=None):
    if time is None:
       time = datetime.now()
    print(time.strftime('%B %D, %Y %H:%M:%S'))

whattime()
time.sleep(3)
whattime()
time.sleep(3)
whattime()








Now you will see three times, one for each call, (when you do NOT specify
an argument.












51.4 4. More to be added here










52 Itertools module





  (builtin)
Useful tools as follows:








52.1 count()





Returns an iterator that counts





Will start at zero, and go to infinity, unless stopped.
This could freeze your computer:






import itertools

counter = itertools.count()

print("ready to hit ctrl-c to halt this infinite loop? ")

for num in counter:
   print(num)








A safer way, use next






print(next(counter))








Let's say you have a list of numbers, 
To associate each item in the list with a count, you this count itertool:







data = [100, 200, 300, 400] 
daily_data = zip(itertools.count(), data)

# or optionally
daily_data = list(zip(itertools.count(), data))

print(daily_data) 
# results in:
[(0, 100), (1, 200), (2, 300), (3, 400)]








counter = itertools.count(start=5, step=-2.5)  i.e. lots of options.










52.2 itertools.ziplongest(range(10), data))





If data has more than 10 items, it will continue until data is exhausted, and
simply put "None" as the value for range, or "None" as the value for data, 
for whatever runs out first.










52.3 itertools.cycle([1, 2, 3])







print(next(counter))
print(next(counter))
print(next(counter))
print(next(counter))
print(next(counter))

will show 
1
2
3
1
2







itertools.cycle(('on', 'off'))





will show
on
off
on
off
on










52.4 itertools.repeat(2)







print(next(counter))
print(next(counter))
print(next(counter))
print(next(counter))

2
2
2
2













52.5 itertools.repeat(2, times=3)







print(next(counter))
print(next(counter))
print(next(counter))
print(next(counter))

2
2
2
Error Stopiteration












52.6 map







squares = map(pow, range(10), itertools.repeat(2))
print(squares)
# <map at 0x109d0ea40>
print(list(squares)
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]







This is just to illustrate the repeat function.  THere are easier ways 
to get a list of squares.










52.7 itertools.starmap





squares = itertools.starmap(pow, [(0,2), (1,2), (2,2)])







letters = ['a', 'b', 'c', 'd']
numbers = [0, 1, 2, 3] 
names = ["Graham", "Terry"]

resultc = itertools.combinations(letters, 2)
for item in result:
    print(item)

('a', 'b')
('a', 'c')
('a', 'd')
('b', 'c')
('b', 'd')
('c', 'd')


resultp = itertools.permutations(letters, 2)
for item in result:
    print(item)

('a', 'b')
('a', 'c')
('a', 'd')
('b', 'a')
('b', 'c')
('b', 'd')
('c', 'a')
('c', 'b')
('c', 'd')
('d', 'a')
('d', 'b')
('d', 'c')


resultcp = itertools.product(numbers, repeat=4)
for item in result:
    print(item)

('a', 'b')
('a', 'c')
('a', 'd')
('b', 'a')
('b', 'c')
('b', 'd')
('c', 'a')
('c', 'b')
('c', 'd')
('d', 'a')
('d', 'b')
('d', 'c')












52.8 chain





Will iterate through the first iterable, then chain it to the next iterable





combine = letters + numbers + names







combine = itertools.chain(letters, numbers, names)

for item in combined:
    print(item)

# results in 
   ...:                                                                                                                                     
(0, 0, 0, 0)
(0, 0, 0, 1)
(0, 0, 0, 2)
(0, 0, 0, 3)
(0, 0, 1, 0)
(0, 0, 1, 1)
(0, 0, 1, 2)
(0, 0, 1, 3)
(0, 0, 2, 0)
(0, 0, 2, 1)
(0, 0, 2, 2)
(0, 0, 2, 3)
(0, 0, 3, 0)
(0, 0, 3, 1)
(0, 0, 3, 2)
(0, 0, 3, 3)
(0, 1, 0, 0)
(0, 1, 0, 1)
(0, 1, 0, 2)
(0, 1, 0, 3)
(0, 1, 1, 0)
(0, 1, 1, 1)
(0, 1, 1, 2)
(0, 1, 1, 3)
(0, 1, 2, 0)
(0, 1, 2, 1)
(0, 1, 2, 2)
(0, 1, 2, 3)
(0, 1, 3, 0)
(0, 1, 3, 1)
(0, 1, 3, 2)
(0, 1, 3, 3)
(0, 2, 0, 0)
(0, 2, 0, 1)
(0, 2, 0, 2)
(0, 2, 0, 3)
(0, 2, 1, 0)
(0, 2, 1, 1)
(0, 2, 1, 2)
(0, 2, 1, 3)
(0, 2, 2, 0)
(0, 2, 2, 1)
(0, 2, 2, 2)
(0, 2, 2, 3)
(0, 2, 3, 0)
(0, 2, 3, 1)
(0, 2, 3, 2)
(0, 2, 3, 3)
(0, 3, 0, 0)
(0, 3, 0, 1)
(0, 3, 0, 2)
(0, 3, 0, 3)
(0, 3, 1, 0)
(0, 3, 1, 1)
(0, 3, 1, 2)
(0, 3, 1, 3)
(0, 3, 2, 0)
(0, 3, 2, 1)
(0, 3, 2, 2)
(0, 3, 2, 3)
(0, 3, 3, 0)
(0, 3, 3, 1)
(0, 3, 3, 2)
(0, 3, 3, 3)
(1, 0, 0, 0)
(1, 0, 0, 1)
(1, 0, 0, 2)
(1, 0, 0, 3)
(1, 0, 1, 0)
(1, 0, 1, 1)
(1, 0, 1, 2)
(1, 0, 1, 3)
(1, 0, 2, 0)
(1, 0, 2, 1)
(1, 0, 2, 2)
(1, 0, 2, 3)
(1, 0, 3, 0)
(1, 0, 3, 1)
(1, 0, 3, 2)
(1, 0, 3, 3)
(1, 1, 0, 0)
(1, 1, 0, 1)
(1, 1, 0, 2)
(1, 1, 0, 3)
(1, 1, 1, 0)
(1, 1, 1, 1)
(1, 1, 1, 2)
(1, 1, 1, 3)
(1, 1, 2, 0)
(1, 1, 2, 1)
(1, 1, 2, 2)
(1, 1, 2, 3)
(1, 1, 3, 0)
(1, 1, 3, 1)
(1, 1, 3, 2)
(1, 1, 3, 3)
(1, 2, 0, 0)
(1, 2, 0, 1)
(1, 2, 0, 2)
(1, 2, 0, 3)
(1, 2, 1, 0)
(1, 2, 1, 1)
(1, 2, 1, 2)
(1, 2, 1, 3)
(1, 2, 2, 0)
(1, 2, 2, 1)
(1, 2, 2, 2)
(1, 2, 2, 3)
(1, 2, 3, 0)
(1, 2, 3, 1)
(1, 2, 3, 2)
(1, 2, 3, 3)
(1, 3, 0, 0)
(1, 3, 0, 1)
(1, 3, 0, 2)
(1, 3, 0, 3)
(1, 3, 1, 0)
(1, 3, 1, 1)
(1, 3, 1, 2)
(1, 3, 1, 3)
(1, 3, 2, 0)
(1, 3, 2, 1)
(1, 3, 2, 2)
(1, 3, 2, 3)
(1, 3, 3, 0)
(1, 3, 3, 1)
(1, 3, 3, 2)
(1, 3, 3, 3)
(2, 0, 0, 0)
(2, 0, 0, 1)
(2, 0, 0, 2)
(2, 0, 0, 3)
(2, 0, 1, 0)
(2, 0, 1, 1)
(2, 0, 1, 2)
(2, 0, 1, 3)
(2, 0, 2, 0)
(2, 0, 2, 1)
(2, 0, 2, 2)
(2, 0, 2, 3)
(2, 0, 3, 0)
(2, 0, 3, 1)
(2, 0, 3, 2)
(2, 0, 3, 3)
(2, 1, 0, 0)
(2, 1, 0, 1)
(2, 1, 0, 2)
(2, 1, 0, 3)
(2, 1, 1, 0)
(2, 1, 1, 1)
(2, 1, 1, 2)
(2, 1, 1, 3)
(2, 1, 2, 0)
(2, 1, 2, 1)
(2, 1, 2, 2)
(2, 1, 2, 3)
(2, 1, 3, 0)
(2, 1, 3, 1)
(2, 1, 3, 2)
(2, 1, 3, 3)
(2, 2, 0, 0)
(2, 2, 0, 1)
(2, 2, 0, 2)
(2, 2, 0, 3)
(2, 2, 1, 0)
(2, 2, 1, 1)
(2, 2, 1, 2)
(2, 2, 1, 3)
(2, 2, 2, 0)
(2, 2, 2, 1)
(2, 2, 2, 2)
(2, 2, 2, 3)
(2, 2, 3, 0)
(2, 2, 3, 1)
(2, 2, 3, 2)
(2, 2, 3, 3)
(2, 3, 0, 0)
(2, 3, 0, 1)
(2, 3, 0, 2)
(2, 3, 0, 3)
(2, 3, 1, 0)
(2, 3, 1, 1)
(2, 3, 1, 2)
(2, 3, 1, 3)
(2, 3, 2, 0)
(2, 3, 2, 1)
(2, 3, 2, 2)
(2, 3, 2, 3)
(2, 3, 3, 0)
(2, 3, 3, 1)
(2, 3, 3, 2)
(2, 3, 3, 3)
(3, 0, 0, 0)
(3, 0, 0, 1)
(3, 0, 0, 2)
(3, 0, 0, 3)
(3, 0, 1, 0)
(3, 0, 1, 1)
(3, 0, 1, 2)
(3, 0, 1, 3)
(3, 0, 2, 0)
(3, 0, 2, 1)
(3, 0, 2, 2)
(3, 0, 2, 3)
(3, 0, 3, 0)
(3, 0, 3, 1)
(3, 0, 3, 2)
(3, 0, 3, 3)
(3, 1, 0, 0)
(3, 1, 0, 1)
(3, 1, 0, 2)
(3, 1, 0, 3)
(3, 1, 1, 0)
(3, 1, 1, 1)
(3, 1, 1, 2)
(3, 1, 1, 3)
(3, 1, 2, 0)
(3, 1, 2, 1)
(3, 1, 2, 2)
(3, 1, 2, 3)
(3, 1, 3, 0)
(3, 1, 3, 1)
(3, 1, 3, 2)
(3, 1, 3, 3)
(3, 2, 0, 0)
(3, 2, 0, 1)
(3, 2, 0, 2)
(3, 2, 0, 3)
(3, 2, 1, 0)
(3, 2, 1, 1)
(3, 2, 1, 2)
(3, 2, 1, 3)
(3, 2, 2, 0)
(3, 2, 2, 1)
(3, 2, 2, 2)
(3, 2, 2, 3)
(3, 2, 3, 0)
(3, 2, 3, 1)
(3, 2, 3, 2)
(3, 2, 3, 3)
(3, 3, 0, 0)
(3, 3, 0, 1)
(3, 3, 0, 2)
(3, 3, 0, 3)
(3, 3, 1, 0)
(3, 3, 1, 1)
(3, 3, 1, 2)
(3, 3, 1, 3)
(3, 3, 2, 0)
(3, 3, 2, 1)
(3, 3, 2, 2)
(3, 3, 2, 3)
(3, 3, 3, 0)
(3, 3, 3, 1)
(3, 3, 3, 2)
(3, 3, 3, 3)












52.9 itertools.islice(range(10), 1, 5, 2)





Really this is islice(iterator, starting, ending, step)





So the above results in 






result = itertools.islice(range(20), 1, 5, 2)

1
3
5
7
9
11
13







Here is islice that is similar to head -6.  This works because a file
becomes an iterator in python as well.







with open('/var/log/messages', 'r') as f:

   head6 = itertools.islice(f, 6)

   for line in header:
      print(line)


   for line in header:
      print(line, end='')













52.10 itertools.compress







selectors = [True, True, False, True]

result = itertools.compress(letters, selectors)

for item in result:
   print(item)
# this returns an iterator as well.

a
b
d







Similar to filter function.  








52.10.1 Filter funciton





this uses a function to determine if something is true or false.







 def lt_2(n):
   if n < 2:
      return True
   return Fales


result = filter(lt_2, numbers)   # notice filter is not a itertools function.

for item in result:
   print(item)

0
1












52.10.2 Filter funciton





this uses a function to determine if something is true or false.







def lt_2(n):
   if n < 2:
      return True
   return Fales

result = itertools.filterfalse(lt_2, numbers)

for item in result:
   print(item)

2
3














52.11 itertools.dropwhile





These stop filtering once a result is true





Will filter along, until you get a true, and after that it just returns
all the remaining iterator values.







result = itertools.dropwhile(lt_2, numbers)
# dropwhile(filterfunction, iterator)

for item in result:
   print(item)







For an example of this, see the youtube video by Corey Schafer











52.12 itertools.accumulate





Returns an accumulated sum of all items seen.
Can also change the "sum" function to another function.







numbers = [0, 1, 2, 3, 2, 1, 0]

result = itertools.accumulate(numbers)

for item in result:
   print(item)

0
1
3
6
8

numbers = [1, 2, 3, 2, 1, 0]

=itertools.accumulate(numbers, operator.mul)=

1
2
6
12
12
0













52.13 itertools.groupby





First tell itertools, what to group by, which is our key.
So we need a function, that will return a single item from our iterable.







def get_state(person):
   return person['state']


person_group = itertools.groupby(people, get_state)







Each item will be a tupole of two things, the key (state in this case) and
the person.







for key, group in person_group
   print(key, group)








This example, as the whole itertools section is all from a Corey Shaefer
youtube video, and github page: github








def get_state(person):
    return person['state']


people = [
    {
        'name': 'John Doe',
        'city': 'Gotham',
        'state': 'NY'
    },
    {
        'name': 'Jane Doe',
        'city': 'Kings Landing',
        'state': 'NY'
    },
    {
        'name': 'Corey Schafer',
        'city': 'Boulder',
        'state': 'CO'
    },
    {
        'name': 'Al Einstein',
        'city': 'Denver',
        'state': 'CO'
    },
    {
        'name': 'John Henry',
        'city': 'Hinton',
        'state': 'WV'
    },
    {
        'name': 'Randy Moss',
        'city': 'Rand',
        'state': 'WV'
    },
    {
        'name': 'Nicole K',
        'city': 'Asheville',
        'state': 'NC'
    },
    {
        'name': 'Jim Doe',
        'city': 'Charlotte',
        'state': 'NC'
    },
    {
        'name': 'Jane Taylor',
        'city': 'Faketown',
        'state': 'NC'
    }
]

person_group = itertools.groupby(people, get_state)

copy1, copy2 = itertools.tee(person_group)

for key, group in person_group:
    print(key, len(list(group)))
    for person in group:
        print(person)
    print()












52.14 zip





zip is easiest to see in a simple example






cast = ['John Cleese',
        'Terry Jones',
        'Eric Idle',
        'Terry Gilliam',
        'Michael Palin',
        'Graham Chapman']
roles = ['Sir Lancalot',
         'Mother of Brian',
         'Sir Robin',
         'Deaf and Mad Guard',
         'Catholic Dad',
         'Brian']
movies = ['Holy Grail',
          'Life of Brian',
          'Holy Grail',
          'Life of Brian',
          'Meaning of Life ',
          'Life of Brian']

for actor, role, movie in zip(cast, roles, movies):
    print(f'  {role} is played by {actor} in film {movie}')








Results in this output:



Sir Lancalot is played by John Cleese in film Holy Grail
Mother of Brian is played by Terry Jones in film Life of Brian
Sir Robin is played by Eric Idle in film Holy Grail
Deaf and Mad Guard is played by Terry Gilliam in film Life of Brian
Catholic Dad is played by Michael Palin in film Meaning of Life 
Brian is played by Graham Chapman in film Life of Brian





An example with three lists zipped together.






import string

myzip = zip([1,2,3],['a','b','c'],['x','y','z'])

myalphabet = list(string.ascii_lowercase)
# myalphabet = ["a", "b", "c", ... "z"]

myAlphabet = list(string.ascii_uppercase)

mynums = list(range(1,27))

for i, j, k in zip(mynums, myalphabet, myAlphabet):
    print(f'  {i}: {j}  upper case: {k}')







Resulst in this ouput:



1: a  upper case: A
2: b  upper case: B
3: c  upper case: C
4: d  upper case: D
5: e  upper case: E
6: f  upper case: F
7: g  upper case: G
8: h  upper case: H
9: i  upper case: I
10: j  upper case: J
11: k  upper case: K
12: l  upper case: L
13: m  upper case: M
14: n  upper case: N
15: o  upper case: O
16: p  upper case: P
17: q  upper case: Q
18: r  upper case: R
19: s  upper case: S
20: t  upper case: T
21: u  upper case: U
22: v  upper case: V
23: w  upper case: W
24: x  upper case: X
25: y  upper case: Y
26: z  upper case: Z












52.15 zip in printing lists in columns





A very good use of zip is a pythonic way of printing a list into columns.
Lets say you have a long list, and you want to print them into three columns
of 10 each.  Here's what you can do:







# print out all the python 'printable' ascii characters:
import string
alonglist = list(string.printable)

for col1, col2, col3, col4 in zip(alonglist[::4], alonglist[1::4], alonglist[2::4], alonglist[3::4]):
    print(f' {col1:<8}{col2:<8}{col3:<8}{col4:<}')







Results in this output:



0       1       2       3
4       5       6       7
8       9       a       b
c       d       e       f
g       h       i       j
k       l       m       n
o       p       q       r
s       t       u       v
w       x       y       z
A       B       C       D
E       F       G       H
I       J       K       L
M       N       O       P
Q       R       S       T
U       V       W       X
Y       Z       !       "
#       $       %       &
'       (       )       *
+       ,       -       .
/       :       ;       <
=       >       ?       @
[       \       ]       ^
_       `       {       |
}       ~               	






From above section on  Slicing [start: end+1: step]​ you will recall that
python starts counting at 0, and if start is omitted, the default is 0.
which means the first element has index 0.





so [1::4] starts at the second elements ("1" is the second element) and
counts by four.





Also recall that if end is omitted, slicing will go to the end of the list.








x = [1, 2, 3, 4, 5, 6 ,7 ,8, 9, 10]

print(x[::4])
# Gives you a list itself with 3 elements [1, 5, 9]
# the 

print(x[1::4])
# Gives you a list itself with 3 elements [2, 6, 10]

print(x[2::4])
# Gives you a list itself with 3 elements [3, 7]

print(x[3::4])
# Gives you a list itself with 3 elements [4, 8]








So finally, zipping these four lists gets you the first line of 1, 2, 3, 4















53 Python debugger, pdb, breakpoint()





pdb has been deprecated by breakpoint(), after Python 3.7





Traditionally print statements peppered through-out your code to dump
vales at given spots in your code is still done, but use this sparringly.





A better approach allows you to leave behind the ubiquitous print statement
when use the python debugger, pdb.  With pdb, the execution of a python script
can be stopped anywhere, and then the state of all variables and code can be
  examined interactively.  Any surprises or unexpected states can be easily
determined, and then fixed.  The code can be restarted at any time.





pdb is included in standard libraries when you install python.  ipdb is not
included, so you must pip install ipbd if you prefer the interactive version.
-who wouldn't?








53.1 interactive python





An excellent approach is to first test your code or code snippets in
interactive python interpretor.  Could be ipython or just python.












53.2 stopping execution (breaking)





breakpoint() is the new pdb





Prior to python 3.7 you would have to:



    

  1. import pdb

    2. 

  2. pdb.set_trace()  somewhere in your code where you wanted to check.

    3. 





Alternatively, if you prefer ipdb (need to pip install it) it would be:



    

  1. import ipdb

    2. 

  2. ipdb.settrace()

    3. 










53.3 Post 3.7 just use breakpoint()





You do NOT even need to import pbd.  breakpoint() is always available.










53.4 pdb commands





The three most common pdb commands to know are ~
next~, step, and continue. 
You can type them out, or you can use the short forms:



    

  • n  next

    • 

  • s  step into a function.  continue with s.  n will finish the function

    • 

  • c  continue

    • 








53.4.1 Once stopped, these commands can be issued to the python script/pdb.




    

  • q quit

    • 

  • C-d quit (EOF)

    • 

  • h help

    • 










53.4.2 Manual breakpoints





You can clear breakpoints, add new breakpoints, and list them.




    

  • b   show all breakpoints

    • 

  • b7 set a breakpoint at line 7

    • 

  • b7, condition   set a breakpoint at line 7 if condition is met

    • 

  • cl  clears all breakpoints

    • 










53.4.3 Stepping




    

  • n next step (step over)

    • 

  • s step into a function (n would otherwise run the whole function as 1 step)

    • 

  • r continue execution to current function returns

    • 

  • c continue to next breakpoint

    • 

  • j8 jump to line 8 (use to break out of loops)

    • 










53.4.4 Frame navigation




    

  • u   up   one level in the stack trace

    • 

  • d   down one level in the stack trace

    • 










53.4.5 Display




    

  • p expr  print the value of expression

    • 

  • pp expr pretty print the value of the expression

    • 

  • w       print the current position where you are and the stack trace

    • 

  • l       list the lines of code around the current line

    • 

  • ll      list the lines of code around the current line (long listing)

    • 

  • a       print args of the current function

    • 












53.5 typical workflow





$ export PYTHONBREAKPOINT=ipdb.settrace
$ 




    

  • add a breakpoint() line in your code.

    • 


  • if your code is waiting for a web client to connect, do so on another
terminal with curl, so that the program runs, and will 

    • 





    

  • s (step) start the next line of code  (repeatedly)

    • 

  • n (next)

    • 

  • r (will return.  ? from what? the next function?

    • 


  • myvariablename   # if l shows you the line where you set myvariablename,

    • 


  • l list the code around this spot 

    • 












54 reading from and writing to a file





You should import os, and maybe import sys I think





But here is the simplest form:







f = open("demofile2.txt", "a")
f.writexb("Now the file has more content!")
f.close()

#open and read the file after the appending:
f = open("demofile2.txt", "r")
print(f.read())







Alt example






outfile = open(fname, mode='w', newline='\n')









f = open("guru99.txt","w+")









The summary is that the OS module is responsible for interacting with the
operating system, providing access to the underlying interface of the
operating system, and the SYS module is responsible for the interaction
between the program and the Python interpreter, providing a series of
functions and variables for manipulating the Python runtime environment.










55 os and sys modules





import os and import sys gives you these methods:







55.1 Os Common methods




    

  • Os.remove (' path/filename ') Delete file

    • 

  • Os.rename (Oldname, newname) renaming files

    • 

  • Os.walk () Generate all file names under the directory tree

    • 

  • Os.chdir (' dirname ') Change directory

    • 

  • Os.mkdir/makedirs (' dirname ') create a directory/multi-level directory

    • 

  • Os.rmdir/removedirs (' dirname ') Delete directory/multi-level directory

    • 

  • Os.listdir (' dirname ') lists the files for the specified directory

    • 

  • OS.GETCWD () Get the current working directory

    • 

  • Os.chmod () Changing directory permissions

    • 

  • Os.path.basename (' path/filename ') remove directory path, return file name

    • 

  • Os.path.dirname (' path/filename ') remove file name, return directory path

    • 

  • Os.path.join (path1[,path2[,…]]) combines the parts of the separation into one path name

    • 

  • Os.path.split (' path ') returns (DirName (), basename ()) tuple

    • 

  • Os.path.splitext () return (filename, extension) tuple

    • 

  • Os.path.getatime\ctime\mtime returns the last access, creation, modification time, respectively

    • 

  • Os.path.getsize () returns the file size

    • 

  • Os.path.exists () is present

    • 

  • Os.path.isabs () is an absolute path

    • 

  • Os.path.isdir () is a directory

    • 

  • Os.path.isfile () is a file

    • 










55.2 SYS Common Methods





SYS.ARGV command line argument list, the first element is the path of the program itself
Sys.modules.keys () returns the list of all modules that have been imported
Sys.excinfo () Gets the exception class currently being processed, Exctype, Excvalue, exctraceback the exception details currently handled
Sys.exit (n) exit program, Exit normally (0)
Sys.hexversion gets the version value of the Python interpreter, 16 binary format such as: 0x020403f0
Sys.version get version information for Python interpreter
Sys.maxint the largest int value
Sys.maxunicode the largest Unicode value
Sys.modules returns the module
 field of the system import, key is the module
 name, value is the module





Sys.path returns the search path for the module
, using the value of the PYTHONPATH environment variable when initializing
Sys.platform returns the operating system platform name
Sys.stdout Standard Output
Sys.stdin Standard input
Sys.stderr Error Output
Sys.excclear () to clear current or recent error messages that are present on the current thread
Sys.execprefix returns the location of the platform standalone Python file installation
Sys.byteorder The local byte rule indicator, the value of the Big-endian platform is ' big ', the value of the Little-endian platform is ' little '
Sys.copyright record python copyright-related things





API version of C for the Sys.apiversion interpreter





Sys.stdin,sys.stdout,sys.stderr





stdin, stdout, and stderr variables contain stream objects that correspond to standard I/O streams. If you need more control over the output, and print does not meet your requirements, they are what you need. You can also replace them, so you can redirect output and input to other devices, or handle them in a non-standard way.We often use print and rawinput to input and print, then How does print and rawinput relate to standard input/output streams?
In fact, the Python program's standard input/output/error stream is defined in the SYS module
, respectively: Sys.stdin,sys.stdout, Sys.stderr
The following programs can also be used to input and output the same:
import sys





sys.stdout.write(‘HelloWorld!‘)





print ‘Please enter yourname:‘,
name=sys.stdin.readline()[:-1]
print ‘Hi, %s!‘ % name





So Sys.stdin, Sys.stdout, stderr exactly what is it? We enter the following code in the Python Runtime environment:
import sys
for f in (sys.stdin,sys.stdout, sys.stderr): print f
output：
<open file‘<stdin>‘, mode ‘r‘ at 892210>
<open file‘<stdout>‘, mode ‘w‘ at 892270>
<open file‘<stderr>‘, mode ‘w at 8922d0>





It can be seen that stdin, stdout, stderr in Python are nothing more than file attributes, they are automatically associated with the standard input, output, and error in the shell environment when Python starts.
The I/O redirection of the Python program in the shell is exactly the same as the DOS command at the beginning of this article, which is actually provided by the shell and is not related to python itself. So can we redirect Stdin,stdout,stderr Read and write operations to an internal object inside a python program? The answer is yes.
Python provides a Stringio module
 to complete this idea, such as:
From Stringio import Stringio
Import Sys
Buff =stringio ()





Temp =sys.stdout #Save standard I/o Flow
Sys.stdout =buff #Redirect the standard I/o flow to Buff object
Print, 'hello', 0.001





Sys.stdout=temp #Restore standard I/O Flow





Print Buff.getvalue ()
The difference between OS and sys two modules in Python













56 Doc[2]





This is taken straight from nmpydoc.readthedocs.io





A documentation string (docstring) is a string that describes a module
,
function, class, or method definition. The docstring is a special attribute
of the object (object._doc__) and, for consistency, is surrounded by triple
double quotes, i.e.:





"""This is the form of a docstring.





It can be spread over several lines.





""" NumPy, SciPy, and the scikits follow a common convention for doc[2] that
provides for consistency, while also allowing our toolchain to produce
well-formatted reference guides. This document describes the current
community consensus for such a standard. If you have suggestions for
improvements, post them on the numpy-discussion list.





Our docstring standard uses re-structured text (reST) syntax and is rendered
using Sphinx (a pre-processor that understands the particular documentation
style we are using). While a rich set of markup is available, we limit
ourselves to a very basic subset, in order to provide doc[2] that are easy to
read on text-only terminals.





A guiding principle is that human readers of the text are given precedence
over contorting doc[2] so our tools produce nice output. Rather than
sacrificing the readability of the doc[2], we have written pre-processors to
assist Sphinx in its task.





The length of docstring lines should be kept to 75 characters to facilitate
reading the doc[2] in text terminals.





Sections The docstring consists of a number of sections separated by headings
(except for the deprecation warning). Each heading should be underlined in
hyphens, and the section ordering should be consistent with the description
below.





The sections of a function’s docstring are:





Short summary





A one-line summary that does not use variable names or the function name,
e.g.





def add(a, b): """The sum of two numbers.





   """ The function signature is normally found by introspection and
displayed by the help function. For some functions (notably those written in
C) the signature is not available, so we have to specify it as the first line
of the docstring:





""" add(a, b)





The sum of two numbers.





""" Deprecation warning





A section (use if applicable) to warn users that the object is
deprecated. Section contents should include:





In what NumPy version the object was deprecated, and when it will be removed.





Reason for deprecation if this is useful information (e.g., object is
superseded, duplicates functionality found elsewhere,  etc..).





New recommended way of obtaining the same functionality.





This section should use the deprecated Sphinx directive instead of an
underlined section header.





.. deprecated:: 1.6.0 `ndobjold` will be removed in NumPy 2.0.0, it is
	  replaced by `ndobjnew` because the latter works also with array
	  subclasses.  Extended Summary





A few sentences giving an extended description. This section should be used
to clarify functionality, not to discuss implementation detail or background
theory, which should rather be explored in the Notes section below. You may
refer to the parameters and the function name, but parameter descriptions
still belong in the Parameters section.





Parameters





Description of the function arguments, keywords and their respective types.





Parameters






x : type Description of parameter `x`.  y Description of parameter `y` (with
    type not specified).  Enclose variables in single backticks. The colon
    must be preceded by a space, or omitted if the type is absent.





For the parameter types, be as precise as possible. Below are a few examples
of parameters and their types.





Parameters






filename : str copy : bool dtype : data-type iterable : iterable object
shape : int or tuple of int files : list of str If it is not necessary to
specify a keyword argument, use optional:





x : int, optional Optional keyword parameters have default values, which are
displayed as part of the function signature. They can also be detailed in the
description:





Description of parameter `x` (the default is -1, which implies summation over
all axes).  or as part of the type, instead of optional. If the default value
would not be used as a value, optional is preferred. These are all
equivalent:





copy : bool, default True copy : bool, default=True copy : bool, default:
True When a parameter can only assume one of a fixed set of values, those
values can be listed in braces, with the default appearing first:





order : {'C', 'F', 'A'} Description of `order`.  When two or more input
    parameters have exactly the same type, shape and description, they can be
    combined:





x1, x2 : arraylike Input arrays, description of `x1`, `x2`.  Returns





Explanation of the returned values and their types. Similar to the Parameters
section, except the name of each return value is optional. The type of each
return value is always required:





Returns






int Description of anonymous integer return value.  If both the name and type
    are specified, the Returns section takes the same form as the Parameters
    section:





Returns






errcode : int Non-zero value indicates error code, or zero on success.
    errmsg : str or None Human readable error message, or None on success.
    Yields





Explanation of the yielded values and their types. This is relevant to
generators only. Similar to the Returns section in that the name of each
value is optional, but the type of each value is always required:





Yields






int Description of the anonymous integer return value.  If both the name and
    type are specified, the Yields section takes the same form as the Returns
    section:





Yields






errcode : int Non-zero value indicates error code, or zero on success.
    errmsg : str or None Human readable error message, or None on success.
    Support for the Yields section was added in numpydoc version 0.6.





Receives





Explanation of parameters passed to a generator’s .send() method, formatted
as for Parameters, above. Since, like for Yields and Returns, a single object
is always passed to the method, this may describe either the single
parameter, or positional arguments passed as a tuple. If a docstring includes
Receives it must also include Yields.





Other Parameters





An optional section used to describe infrequently used parameters. It should
only be used if a function has a large number of keyword parameters, to
prevent cluttering the Parameters section.





Raises





An optional section detailing which errors get raised and under what
conditions:





Raises






LinAlgException If the matrix is not numerically invertible.  This section
    should be used judiciously, i.e., only for errors that are non-obvious or
    have a large chance of getting raised.





Warns





An optional section detailing which warnings get raised and under what
conditions, formatted similarly to Raises.





Warnings





An optional section with cautions to the user in free text/reST.





See Also





An optional section used to refer to related code. This section can be very
useful, but should be used judiciously. The goal is to direct users to other
functions they may not be aware of, or have easy means of discovering (by
looking at the module
 docstring, for example). Routines whose doc[2] further
explain parameters used by this function are good candidates.





As an example, for numpy.mean we would have:





See Also






average : Weighted average.  When referring to functions in the same
sub-module, no prefix is needed, and the tree is searched upwards for a
match.





Prefix functions from other sub-modules appropriately. E.g., whilst
documenting the random module
, refer to a function in fft by





fft.fft2 : 2-D fast discrete Fourier transform.  When referring to an
entirely different module










scipy.random.norm : Random variates, PDFs,  etc..  Functions may be listed
without descriptions, and this is preferable if the functionality is clear
from the function name:





See Also






funca : Function a with its description.  funcb, funcc_, funcd funce If
the combination of the function name and the description creates a line that
is too long, the entry may be written as two lines, with the function name
and colon on the first line, and the description on the next line, indented
four spaces:





See Also






package.module
.submodule.funca : A somewhat long description of the
    function.  Notes





An optional section that provides additional information about the code,
possibly including a discussion of the algorithm. This section may include
mathematical equations, written in LaTeX format:





Notes






The FFT is a fast implementation of the discrete Fourier transform:





.. math:: X(ejω  ) = x(n)e - jω n Equations can also be
typeset underneath the math directive:





The discrete-time Fourier time-convolution property states that





.. math::





x(n) * y(n) ⇔ X(ejω  )Y(ejω  )\\ another
equation here Math can furthermore be used inline, i.e.





The value of :math:`ω` is larger than 5.  Variable names are displayed
in typewriter font, obtained by using \mathtt{var}:





We square the input parameter `alpha` to obtain :math:`\mathtt{alpha}2`.
Note that LaTeX is not particularly easy to read, so use equations sparingly.





Images are allowed, but should not be central to the explanation; users
viewing the docstring as text must be able to comprehend its meaning without
resorting to an image viewer. These additional illustrations are included
using:





.. image:: filename where filename is a path relative to the reference guide
source directory.





References





References cited in the notes section may be listed here, e.g. if you cited
the article below using the text [1]_, include it as in the list as follows:





.. [1] O. McNoleg, "The integration of GIS, remote sensing, expert systems
   and adaptive co-kriging for environmental habitat modelling of the
   Highland Haggis using object-oriented, fuzzy-logic and neural-network
   techniques," Computers & Geosciences, vol. 22, pp. 585-588, 1996.  which
   renders as 1:





1 O. McNoleg, “The integration of GIS, remote sensing, expert systems and
adaptive co-kriging for environmental habitat modelling of the Highland
Haggis using object-oriented, fuzzy-logic and neural-network techniques,”
Computers & Geosciences, vol. 22, pp. 585-588, 1996.





Referencing sources of a temporary nature, like web pages, is
discouraged. References are meant to augment the docstring, but should not be
required to understand it. References are numbered, starting from one, in the
order in which they are cited.





Warning





References will break tables





Where references like [1] appear in a tables within a numpydoc docstring, the
table markup will be broken by numpydoc processing. See numpydoc issue #130





Examples





An optional section for examples, using the doctest format. This section is
meant to illustrate usage, not to provide a testing framework – for that, use
the tests/ directory. While optional, this section is very strongly
encouraged.





When multiple examples are provided, they should be separated by blank
lines. Comments explaining the examples should have blank lines both above
and below them:





Examples






>>> np.add(1, 2) 3





Comment explaining the second example.





>>> np.add([1, 2], [3, 4]) array([4, 6]) The example code may be split across
multiple lines, with each line after the first starting with ‘… ‘:





>>> np.add([[1, 2], [3, 4]], …  [[5, 6], [7, 8]]) array([[ 6, 8], [10,
12]]) For tests with a result that is random or platform-dependent, mark the
output as such:





>>> import numpy.random np.random.rand(2) array([ 0.35773152, 0.38568979])
>>> #random You can run examples as doctests using:





>>> np.test(doctests=True) np.linalg.test(doctests=True) # for a single
>>> module
 In IPython it is also possible to run individual examples simply
>>> by copy-pasting them in doctest mode:





In [1]: %doctestmode Exception reporting mode: Plain Doctest mode is: ON >>>
%paste import numpy.random np.random.rand(2) ## – End pasted text – array([
0.8519522 , 0.15492887]) It is not necessary to use the doctest markup
<BLANKLINE> to indicate empty lines in the output. Note that the option to
run the examples through numpy.test is provided for checking if the examples
work, not for making the examples part of the testing framework.





The examples may assume that import numpy as np is executed before the
example code in numpy. Additional examples may make use of matplotlib for
plotting, but should import it explicitly, e.g., import matplotlib.pyplot as
plt. All other imports, including the demonstrated function, must be
explicit.





When matplotlib is imported in the example, the Example code will be wrapped
in matplotlib’s Sphinx `plot` directive. When matplotlib is not explicitly
imported, .. plot:: can be used directly if
matplotlib.sphinxext.plotdirective is loaded as a Sphinx extension in
conf.py.





Documenting classes Class docstring Use the same sections as outlined above
(all except Returns are applicable). The constructor (init) should also be
documented here, the Parameters section of the docstring details the
constructors parameters.





An Attributes section, located below the Parameters section, may be used to
describe non-method attributes of the class:





Attributes






x : float The X coordinate.  y : float The Y coordinate.  Attributes that are
    properties and have their own doc[2] can be simply listed by name:





Attributes






real imag x : float The X coordinate.  y : float The Y coordinate.  In
general, it is not necessary to list class methods. Those that are not part
of the public API have names that start with an underscore. In some cases,
however, a class may have a great many methods, of which only a few are
relevant (e.g., subclasses of ndarray). Then, it becomes useful to have an
additional Methods section:





class Photo(ndarray): """ Array with associated photographic information.





…





Attributes






exposure : float Exposure in seconds.





Methods






colorspace(c='rgb') Represent the photo in the given colorspace.
    gamma(n=1.0) Change the photo's gamma exposure.





    """ If it is necessary to explain a private method (use with care!), it
can be referred to in the Extended Summary or the Notes section. Do not list
private methods in the methods section.





Note that self is not listed as the first parameter of methods.





Method doc[2] Document these as you would any other function. Do not include
self in the list of parameters. If a method has an equivalent function (which
is the case for many ndarray methods for example), the function docstring
should contain the detailed documentation, and the method docstring should
refer to it. Only put brief summary and See Also sections in the method
docstring. The method should use a Returns or Yields section, as appropriate.





Documenting class instances Instances of classes that are part of the NumPy
API (for example np.r_ np.c_, np.indexexp,  etc..) may require some care. To
give these instances a useful docstring, we do the following:





Single instance: If only a single instance of a class is exposed, document
the class. Examples can use the instance name.





Multiple instances: If multiple instances are exposed, doc[2] for each
instance are written and assigned to the instances’ doc attributes at run
time. The class is documented as usual, and the exposed instances can be
mentioned in the Notes and See Also sections.





Documenting generators Generators should be documented just as functions are
documented. The only difference is that one should use the Yields section
instead of the Returns section. Support for the Yields section was added in
numpydoc version 0.6.





Documenting constants Use the same sections as outlined for functions where
applicable:




    

  1. summary

    2. 

  2. extended summary (optional)

    3. 

  3. see also (optional)

    4. 

  4. references (optional)

    5. 

  5. examples (optional)

    6. 




Doc[2] for constants will not be visible in text terminals (constants are of
immutable type, so doc[2] can not be assigned to them like for for class
instances), but will appear in the documentation built with Sphinx.





Documenting modules Each module
 should have a docstring with at least a
summary line. Other sections are optional, and should be used in the same
order as for documenting functions when they are appropriate:




    

  1. summary

    2. 

  2. extended summary

    3. 

  3. routine listings

    4. 

  4. see also

    5. 

  5. notes

    6. 

  6. references

    7. 

  7. examples

    8. 




Routine listings are encouraged, especially for large modules, for which it
is hard to get a good overview of all functionality provided by looking at
the source file(s) or the all dict.





Note that license and author info, while often included in source files, do
not belong in doc[2].





Other points to keep in mind Equations : as discussed in the Notes section
above, LaTeX formatting should be kept to a minimum. Often it’s possible to
show equations as Python code or pseudo-code instead, which is much more
readable in a terminal. For inline display use double backticks (like y =
np.sin(x)). For display with blank lines above and below, use a double colon
and indent the code, like:





end of previous sentence::





    y = np.sin(x) Notes and Warnings : If there are points in the docstring
that deserve special emphasis, the reST directives for a note or warning can
be used in the vicinity of the context of the warning (inside a
section). Syntax:





.. warning:: Warning text.





.. note:: Note text.  Use these sparingly, as they do not look very good in
text terminals and are not often necessary. One situation in which a warning
can be useful is for marking a known bug that is not yet fixed.





arraylike : For functions that take arguments which can have not only a type
ndarray, but also types that can be converted to an ndarray (i.e. scalar
types, sequence types), those arguments can be documented with type
arraylike.





Links : If you need to include hyperlinks in your docstring, note that some
docstring sections are not parsed as standard reST, and in these sections,
numpydoc may become confused by hyperlink targets such as:





.. _Example: http://www.example.com If the Sphinx build issues a warning of
the form WARNING: Unknown target name: "example", then that is what is
happening. To avoid this problem, use the inline hyperlink form:





`Example http://www.example.com`_ Common reST concepts For paragraphs,
indentation is significant and indicates indentation in the output. New
paragraphs are marked with a blank line.





Use italics, bold and ``monospace`` if needed in any explanations (but not
for variable names and doctest code or multi-line code). Variable, module
,
function, and class names should be written between single back-ticks
(`numpy`).





A more extensive example of reST markup can be found in this example
document; the quick reference is useful while editing.





Line spacing and indentation are significant and should be carefully
followed.











57 Anti-patterns





Code that should be refactored into better code.  These are patterns that are
awkward and/or inefficient and should be avoided.








57.1 ITM





Initialize Then Modify is very common.  Take this sample code:






myarray = []
i = 0

for element in elements:
   i += 1
   interests_me = element.[2]
   print(i, interests_me)
   myarray.append((interests_me, []))








Replace that with enumerate







myarray = []

for i, element in enumerate(elements):
   interests_me = element.[2]
   print(i, interests_me)
   myarray.append((interests_me, []))













57.2 Use conditional statements.





Rather than explicit if this else that use conditional statements







myarray = []

for i, element in enumerate(elements):
  interests_me = element.[2]
  try:
     interests_me = int(interests_me)
  else:
     pass
  # while appending, convert numeric thingies to integers
  myarray.append((interests_me, []))








This can be more concise as well as more readable at the same time:






myarray = []

for i, element in enumerate(elements):
  interests_me = element.[2]
  interests_me = int(interests_me) if interests_me.isnumeric else interests_me
  myarray.append((interests_me, []))













57.3 map() and filter() on Iterable Objects





Replace these with List Comprehensions and
Examples of map functions replaced by list comprehensions:







from math import factorial
from math import factorial
list(map(factorial, range(10))
# should become:
[factorial(n) for n in range(10)]


list(map(factorial, filter(lambda n: n%2, range(10))))
#should become
[factorial(n) for n in range(10) if n % 2]













57.4 Unneccessary generators





list, dict and set have comprehensions, so no need to use list/dict/set
around a generator expression.  





Bad:






squares = dict((i, i**2) for i in range(1, 100))







Good:






squares = {i: i**2 for i in range(1, 100)}












57.5 Raise execptions in a function if it does not return a single function type





Rather than have the code that calls the function have to check on the type
of the return and call an exception there, do it in the function itself.
That also means of course that a function should ONLY return a single type,
and not be mixing types of values returned.





Bad:






def get_actor_age(montian):
   castmember = db.get_actor(montian)
   if castmember:
      return castmember.age
# returns None if castmember not found








Good:






def get_actor_age(montian):
   castmember = db.get_actor(montian)
   if not castmember:
      raise Exception(f" Excuse me but, {montian} was not part of Monty Python")
   return castmember.age







Copying straight out of an example in deepsource.io:




"This anti-pattern affects the readability of code. Often we see code to
create a variable, assign a default value to it, and then checking the
dictionary for a certain key. If the key exists, then the value of the key is
assigned into the value for the variable. Although there is nothing wrong in
doing this, it is verbose and inefficient as it queries the dictionary twice,
while it can be easily done using the get() method for the dictionary."






So bad code is:






currency_map = {'usd': 'US Dollar'}

if 'inr' in currency_map:
   indian_currency_name = currency_map['inr']
else:
   indian_currency_name = 'undefined'








The better replacment is:






currency_map = {'usd': 'US Dollar'}
indian_currency_name = currency_map.get('inr', 'undefined')














57.6 Use items() to iterate over a dictionary





items() method on a dictionary returns an iterable with (key, value) tuples
which can be unpacked in a for loop.  However bad practice is :







for actor in cast:
    name = cast[actor]
    # some code that manipulates actor









for actor, name in cast.items()
   # some code that manipulates actor












57.7 Use iteral syntax to initialize empty iterables





So for lists, dictionaries and tuples initializatin: be specific as clarity
over brevity should be preferred.





Good:






spanish_inquisition_feared_items = []


















58 Python on Windows (.i.e not on MAC and not on Linux)















unix
equivalent Windows command






python -m venv newenv
py -m venv newenv





. newenv/bin/activate
.\newenv\Scripts\activate





 
 














59 Decorators @





Decorators are used lots by flask but you should learn to use them in your
code too.





Simply put, a decorator function is a function that you can add to another
function, typically an existing built-in function that you want to add
additionaly functionality.





It is really a syntax short cut for the following.






import datetime
def decorator_function(original_function):
    def wrapper_function(*args, **kwargs):
        # the *args, **kwargs if your built-in function passes some args
        print('wrapper function adds some extra functionality, like')
        print(f'reminding you that of a date {datetime.datetime.()}')
        # more code here to add functionality to the original function.
        return original_function()

    return wrapper_function

def builtin_display():
    print('built-in display does the normal thing you would expect')
    print(' --->  that is, display something like this message')


# old school approach:
# create an object that is a decorated function call of a built-in function:
decorated_display = decorator_function(builtin_display)
decorated_display()
# so this little chunk is replaced with the @ decorator_funtion line.









This is essentially the same as writing:







import datetime
def decorator_function(original_function):
    def wrapper_function(*args, **kwargs ):
        # the *args, **kwargs if your built-in function passes some args
        print('  wrapper function adds some extra functionality, like')
        print(f"  decorating output with today's date: {datetime.datetime.now()}")
         # more code here to add functionality to the original function.
        return original_function(*args, **kwargs)

    return wrapper_function

@ decorator_function
def builtin_display():
    print('1)     built-in display ONLY displays something,')
    print(' --->  that is, display something like this message\n\n')

# this syntax so far is just functions calling functions with nothing special.
# now you can understand what the @ decorator syntax will do for you:

@ decorator_function
def builtin_add_sig():
    ''' add my signature to a string (just to learn about decorators).'''
    print("\n2)          Your's sincerely")
    print("            Bob (your uncle)")


@ decorator_function
def builtin_add_sigargs(name, date):
    ''' add my signature to a string (just to learn about decorators).'''
    print("\n\n3)           Your's sincerely")
    print(f"             {name}")
    print(f"             {date}\n\n\n")


@ decorator_function
def fibo():
    '''prints first 7 fibonacci sequence numbers'''
    print("0, 1, 1, 2, 3, 5, 8, 13, 21, 34")


############ main program flow #####################
builtin_display()
builtin_add_sig()
print("\n\n  -------------- Now with args:  ----------")
builtin_add_sigargs("John Cleese", datetime.datetime.now())
fibo()














60 Iterators







for i in range(1,11)
    print(i)

# gets you 1 to 10 without having to store all items in memory like:
x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

y = map(lambda i: i**2, x)
# the map function does not need to store all the squares form 1 to 10
# but it still gets the job done.   y is a map function, not a list
# in storage.

for i in y:
    print(i)

# but still, y, was not something stored in memory, it was "generated"
# one at a time as the for loop when through hte "iterator"








So what is an iterator actually doing.  It runs a function called "next()"
For example I can call next(y) all the way through the "iterator"







print(next(y))
print(next(y))
print(next(y))
print(next(y))
print(next(y))

# in essence what the for loop is doing is calling the next(y) each
# time through the loop, AND, keeping track of where it is.  Actually
# the iterator itself keeps track of that so if I was to put the for
# loop in the code here: it will start at the number 6.

for i in y:
    print(i)








from an ipython session:




In [5]: y = map(lambda i: i**2, x)

In [6]: for i in y:print(i)
1
4
9
16
25
36
49
64
81
100

In [7]: y = map(lambda i: i**2, x)

In [8]: print(y.__next__())
1
In [9]: print(next(y))
4
In [10]: print(y.__next__())
9
In [11]: for i in y:print(i)
16
25
36
49
64
81
100

In [12]:








60.1 for loop as what it is actually doing:







x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
y = map(lambda i: i**2, x)

while True:
    try:
        value = next(y)
        print(value)
    except StopIteration:
        print('Done')
        break

# Now examining the range function
x = range(1,11)
print(x)   # will output this string: "range(1,11)"  not what we wanted.

next(x)   # will crash with TypeError: 'range' object is not an iterator

print(next(iter(x)))  # will now print "1"
# for i in x:  can just as well be written for i in iter(x):
# in words:
# start by calling the iter method
# that iter method returns the iterator which has a next method
# then you call the next method on the iterator.
# So now, lets create a custom iterator:

import sys
class Iter:
    def __inti__(self, n):
         self.n = n

    def __iter__(self):
          self.current = -1
          return self

    def __next__(self):
          self.current +=1

          if self.current >= self.n:
              raise StopIteration

          return self.current

x = Iter(5)

for i in x:
    print(i)

# gives you want you'd expect, 1,2,3,4
















61 Security with Bandit





pip install bandit will let you analyze your python scripts in a directory
with the command bandit -r . while in the root directory of your project.





You can read the results in stdout.  Run bandit --help for more info, or
check out the docs in bandit.readthedocs.io











62 Python logging





Rather than just using print, or even break as methods to debug your python
script, you can add logging that will record progress and encountered problems
of your script in a separate file.





You first need to import logging which is part of the standard python library,
and examine the available methods with dir(logging)  This module
  follows
these conventions:



    

  • ALLCAPS are constants, for example DEBUG

    • 

  • Capitalized entries are classes, for example LogRecord

    • 

  • lower case letters are methods, for example captureWarnings

    • 








62.1 5 Logging Levels





By default python supports 5 logging levels:



    

  1. DEBUG

    2. 

  2. INFO

    3. 

  3. WARNING

    4. 

  4. ERROR

    5. 

  5. CRITICAL

    6. 





You can customize additional logging levels if the need arises.
You select which level you need to use by passing the level parameter
to the basicConfig class.







logging.basicConfig(filename = "~/pythyon/logs/zp_logger.log",
      level = logging.DEBUG)














62.2 basicConfig and getLogger classes.







import logging

# Call the basicConfig class to create a logger object for a file
logging.basicConfig(filename = "~/pythyon/logs/zp_logger.log")
# if the file already exists, it will append to that file.

# next, call the getLogger class on this newly created basicConfig
# object
logger = logging.getLogger()   # not adding a name will default to the
                               # root logger.
# at this point you would expect to see a message to the log file.
# but the logging level by default is WARNING, so you won't see an .info
# level message.
logger.info("This is an info level message sent to the logger")












62.3 Logging levels.





If your logger is set to Debug, this line will NOT add anything
to the log file.  If it is set to Info or higher, you will see
that line added to the log file.  To set the logger level, you change
the numeric value according to this table:















Level
Interger Value






NOTSET
0





DEBUG
10





INFO
20





WARNING
30





ERROR
40





CRITICAL
50








print(logger.level) to see the current setting.







import logging

# add settings to the basicConfig method call:
logging.basicConfig(filename = "~/pythyon/logs/zp_logger.log",
                    level = logging.DEBUG,
                    )

logger = logging.getLogger()
# to actually print a message to the log file, use logger.level("mess")
logger.debug("This is an DEBUG level message sent to the logger")
logger.info("This is an INFO level message sent to the logger")
logger.warning("This is an WARNING level message sent to the logger")
logger.error("This is an ERROR level message sent to the logger")
logger.critical("This is a CRITICAL level message sent to the logger")
logger.debug("This is another DEBUG level message sent to the logger")

# This time you will see a message to the log file as follows
# DEBUG:root:This is an info level message sent to the logger

# if level = logging.error, you will only see the error & critical messages









Then to add a time stamp it would look like this:






import logging
MYLOGFORMAT = "wordle.py %(levelname)s %(asctime)s %(message)s"

# add settings to the basicConfig method call:
logging.basicConfig(filename = "zp_script_logs.log",
                    level = logging.DEBUG,
                    format = MYLOGFORMAT)

logger = logging.getLogger()
logger.info("This is a INFO level message sent to the logger")

# This time you will see a message to the log file as follows
# INFO:root:This is an info level message sent to the logger









For additional logging messages you can add exceptions, etc.
help(logging.exception) for built-in help while using interactive pythyon







import logging

MYLOGFORMAT = "%(levelname)s, %(asctime)s, %(message)s"

# add settings to the basicConfig method call:
logging.basicConfig(filename = "~/pythyon/logs/zp_logger.log",
                    level = logging.DEBUG,
                    filemode = 'a', # the default is 'a'
                    format = MYLOGFORMAT)

# filemodes options are 'a' for append, 'w' for overwrite 

logger = logging.getLogger()

try:
    solution = x / y
except ZeroDivisionError:
    logger.exception("Tried to divide by zero.  Duh!!")
else:
    return solution









To use another logger, that is not the default 'root' logger you pass the
logger name to getLogger, i.e. logger = logging.getLogger(___name___)





This is especially import when you are importing modules that might have
there own logging setup.  As you know imports are done first, and when an
module
 is imported it is also run, so that that imported module
 may have
already claimed and configured the root logger.








62.3.1 _name__ as logger





Using dundername fixes the above issue as follows.
If the script is run directly, the getLogger(___name___) will be changed to
_ _ main _ _





 If the script is imported by another script getLogger(_name__) will be
 changed to the name of the imported module
.





So it is best practice to use logger = logging.etLogger(__name__) almost always.










62.3.2 loggers are heirarchical





If you have not defined a logger in a script, python will fall back to using
the root logger.













62.4 Advanced logger settings (to not always use root)





So as to be more clear in logging and be able to distinguish logging from
various imported modules it is a good idea to use this chunk of code for all
your scripts.  Basically you will be setting up a unique logger for each
script and module







import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

formatter = logging.Formatter('%(levelname)s,%(name)s,%(asctime)s %(message)s')

file_handler = logging.FileHandler("zp_script_logs.log")
file_handler.setLevel(logging.ERROR)
file_handler.setFormatter(formatter)

logger.addHandler(file_handler)

# With all that set, you do not need the basicConfig anymore
# logging.basicConfig(filename = "zp_script_logs.log",
#                        level = logging.DEBUG,
#                        filemode = 'a', # the default is 'a'
#                        format = MYLOGFORMAT)

logger.DEBUG('this message will appear in whatever file specifed by file_handler')
try:
    solution = x / y
except ZeroDivisionError:
    logger.exception("Tried to divide by zero.  Duh!!")
else:
    return solution












62.5 Adding a stream handler as well as a file handler





The filehandler will capture the messages to a file, but we may also want to
see the errors in the interactive session where we ran the programs.  Python
supports adding a second (or more) handlers, in this case a streamhandler








import logging
import zp_env_user

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

formatter = logging.Formatter('%(levelname)s,%(name)s,%(asctime)s %(message)s')

file_handler = logging.FileHandler("zp_script_logs.log")
file_handler.setLevel(logging.ERROR)
file_handler.setFormatter(formatter)

stream_handler = logging.StreamHandler()
stream_handler.setFormatter(formatter)

logger.addHandler(file_handler)
logger.addHandler(stream_handler)

# With all that set, you do not need the basicConfig anymore
# logging.basicConfig(filename = "zp_script_logs.log",
#                        level = logging.DEBUG,
#                        filemode = 'a', # the default is 'a'
#                        format = MYLOGFORMAT)

logger.DEBUG('this message will appear in whatever file specifed by file_handler')
try:
    solution = x / y
except ZeroDivisionError:
    logger.exception("Tried to divide by zero.  Duh!!")
else:
    return solution













62.6 logging vs print in crontab jobs





I have run weekly meraki reports using cron tables where I the python script
print statements have been collected using bash redirects.  That means that
any print statements in the python script would be redirected to a file
as specified using bash in the cron job.





crontab -l sample:






0 14 * * 6 /home/zintis/bin/meraki-weekly-reports.sh 1>> /home/zintis/cron-output.txt 2>> /home/zintis/cron-errors.txt
59 3 * * 0  /home/zintis/bin/meraki-weekly-zibens.sh 1>> /home/zintis/cron-output.txt 2>> /home/zintis/cron-errors.txt
0 14 * * 5 /home/zintis/bin/meraki-weekly-precheck.sh 1>> /home/zintis/cron-output.txt 2>> /home/zintis/cron-errors.txt
1 0 23 * * /home/zintis/bin/meraki-monthly-reports.sh 1>> /home/zintis/cron-output.txt 2>> /home/zintis/cron-errors.txt







That means that any print output would be appended to the file cron-output.txt.





This was the case for all the print statements in all the imported modules
as well.  However, doing it this way you could not distinguish where each
print statement originated.  Changing the python scripts to use logging will
give better info, as well as eliminate the need for the bash redirect in
the cron job.  I would still keep the error redirection though, 2>>













63 Misc. bash script comparing python packages in venvs





Can ignore this…  will document later.






for line in sight-packs
do
    echo "$line is this line"
    echo " folder $line being checked out.  Should be full name... hmmm "
    ls -l $line >> venvpackages
    echo $folder

    ec
    ho ""
done
grep "test" venvpackages










63.1 STart a python http server in one line:





In a folder with some html files:






python -m http.server







If there is an index.html file in the current directory, python will server
that up as a web server.












64 Convert string of currency to a value







from re import sub
from decimal import Decimal

money = '$6,150,593.22'
value = Decimal(sub(r'[^\d.]', '', money))







This does not care what the currency symbol is, $, or E or lbs.










64.1 Home








65 Installing Cisco ACI Cobra tools





I did this while in the python 2.7.16 virtual environment.





(controllers-venv2.7) /Users/zintis/bin/controllers-venv2.7/cobra[185] % python setup.py install
running install
running bdist_egg
running egg_info
writing requirements to acicobra.egg-info/requires.txt
writing acicobra.egg-info/PKG-INFO
writing namespace_packages to acicobra.egg-info/namespace_packages.txt
writing top-level names to acicobra.egg-info/top_level.txt
writing dependency_links to acicobra.egg-info/dependency_links.txt
reading manifest file 'acicobra.egg-info/SOURCES.txt'
writing manifest file 'acicobra.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.14-x86_64/egg
running install_lib
running build_py
creating build/bdist.macosx-10.14-x86_64/egg
creating build/bdist.macosx-10.14-x86_64/egg/cobra
copying build/lib/cobra/services.py -> build/bdist.macosx-10.
.
.
.
Installed /Users/zintis/bin/controllers-venv2.7/lib/python2.7/site-packages/certifi-2019.3.9-py2.7.egg
Finished processing dependencies for acicobra==0.1

(controllers-venv2.7) /Users/zintis/bin/controllers-venv2.7/cobra[187] % pip list
Package    Version 
---------- --------
acicobra   0.1     
certifi    2019.3.9
chardet    3.0.4   
future     0.14.3  
idna       2.8     
pip        19.1.1  
requests   2.21.0  
setuptools 41.0.1  
urllib3    1.24.3  
wheel      0.33.4  
(controllers-venv2.7) /Users/zintis/bin/controllers-venv2.7/cobra[188] % 






Where before this install my 2.7.16 virtualenv had this:




 Package    Version
---------- -------
pip        19.1.1 
setuptools 41.0.1 
wheel      0.33.4 
(controllers-venv2.7) /Users/zintis/bin[109] % 







Possible approaches:






#+ENDEXAMPLE





>>> None of theese worked.  I just had to run setup.py within cobra. <<<





For more info check out python.org documentation on packages.














66 insert where appropriate





From askubunto.com
This explains how to have a python script execute some sudo commands
safely.




  The correct way to do it to setup sudo such that only the one specific command you need, i.e. echo date... > rtc..., is allowed to run WITHOUT needing the password.

Step 1. Create a shell script with just that command
Open up gedit (or your favorite editor), and create the script e.g. pydatertc.sh
Insert only this line, and save it to, e.g. your home directory:
echo date \'+%s\' -d \'+ 24 hours\' > /sys/class/rtc/rtc0/wakealarm
Quit the editor, and from the terminal, make the script executable and change its ownership to root, otherwise another user with access to your system could possibly edit it and execute whatever commands they want as root without needing your password:
sudo chown root:root /home/username/pydatertc.sh
sudo chmod 700 /home/username/pydatertc.sh
Step 2. Set up sudo to allow pydatertc.sh to execute without requiring a password
Type sudo visudo at the terminal to open the sudo permissions (sudoers) file
Around line 25, you'll see this line: %sudo   ALL=(ALL:ALL) ALL
Below that line, insert the following line, where username is your username:
username  ALL=(ALL) NOPASSWD: /home/username/pydatertc.sh
Exit the editor (Ctrl+X if nano)
Step 3. Modify your python script to call pydatertc.sh
Change the line to:
os.system('sudo /home/username/pydatertc.sh')
Now your script should run without requiring a password AND without compromising the security of your account, your data or your system!