my cheat sheet on pip in virtual environments

Home

1 Python Virtual Environment Choices

There are many, among which are:

  • pyenv
  • pyvenv => became venv as of Python 3.6
  • pipenv
  • venv * the one I use the most, and since version 3.7, the default.
  • virtualenv

This cheat tries to get some clarity here.

2 venv

venv is a package shipped with Python 3, which you can run using python3 -m venv (although for some reason some distros separate it out into a separate distro package, such as python3-venv on Ubuntu/Debian). It serves the same purpose as virtualenv, but only has a subset of its features (see a comparison here). virtualenv continues to be more popular than venv, especially since virtualenv supports both Python 2 and 3.

However, more and more with later python versions, like 3.10 and 3.11 venv will be the way to go.

python-virtualenv-pyenv-pipenv.png

Figure 1: Python Virtua Environments

3 pyenv is really just venv since 3.4

C A U T I O N

Do not get caught up in the hype. pyenv is deprecated by the built-in module venv as of python 3.4 and later. So, I really should clean up the old shims versions I have lying around in /.pyenv which, as of Jan2020, are: ~$ pyenv versions

3.1 overview tp pyenv

pyenv: version `3.9' is not installed (set by PYENVVERSION environment variable)

  • system
  • 2.7.16
  • 3.7.5
  • 3.7.7

See this link for an indepth examination of .pyenv, so that you will know what to remove (in favour of the python3 -m venv approach) realpython.com on pyenv.

On Jan2021 I ran the following:

  • pyenv unintall 2.7.16
  • pyenv unintall 3.7.5
  • pyenv unintall 3.7.7

My aim is to remove the .shims directories as well, and just stick with the built-in python3 -m venv approach.

Another good link to see what pyenv had done is ianmaddaus.com

And another one: towardsdatascience.com where I got his image:

And finally, on stackoverflow:

3.2 A good review of pyenv-virtualenv

freecodecamp.org has a good overview of how to install each. But know that python3 -m venv is the way to go forward, since venv is an included module as of python3.4

3.3 pyenv, activating virtual environment

To activate this project's virtualenv, run pipenv shell Nope: the new command is pyenv activate 3.9 where pipenv shell merely shows you or sets the shell-specific python version you will run.

Alternatively, run a command inside the virtualenv with pipenv run. (For clarification and more detail see the pipenv section below.)

Of course simply running pyenv will show a list of available commands to run.

4 pyvenv

pyvenv is a script shipped with Python 3 but deprecated in Python 3.6+ as it had problems (not to mention the confusing name).

Indepth of how shims work in pyvenv dasblinkenlichten.com

5 pipenv

5.1 Pipfile (used by pipenv)

You can see from above, that pipenv uses a new file, called Pipfile which is in essence the same thing that a requirements.txt file is. i.e. a list of packages in this current virtual environment.urw

My pipfile is in /bin/python/mynewpythonproject/Pipfile and the virtual env that was created is in ~/.local/share/virtualenvs/mynewproject-F9bljqZVdddddddddddddddddddddddddddddddddd 

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"

[dev-packages]

[requires]
python_version = "3.9"

5.2 Pipfile.lock (used by pipenv)

Has a list of actual versions of packages that you know work. This file can be updated using some pipenv lock commands

5.3 pipenv -> A final challenger to venv?

pipenv might be the new pip virtual environment tool that can handle multiple python version, if you don't need the latest version of python. Currently pipenv is supported in versions: 2.7, 3.4, 3,5, 3.6, 3.7 as of June 2021

So, pipenv not yet available for 3.8 or 3.9. Remember that -m venv is great for virtual environments when you are all on the same version of python, but just need different groups of packages and dependencies for different projects)

pipenv does that and more. First off some links:

This is a pro pipenv and the author has some valid arguments. https://www.activestate.com/blog/why-pipenv-venv/ But I have not switched to pipenv (after trying it briefly)

5.4 DevNet Associate labs use pipenv

Since they are ok. with python version 3.7, the labs in the DevNet study group use pipenv to make it easier for their labs to get built in a flash.

5.5 pipenv run python mypythonscript.py

This command runs mypythonscript.py but first creates a virtual environement and then runs the script within that virtual environment. If you have a pipfile in this current directory that specifies python3.5, then the virtual env that will be created will use that 3.5 version.

5.6 Setting up vscode to use the correct python executable

You edi .vscode/settings.json and put in the path for python

5.7 first attempt at installing virtual environment using pipenv

Since I wanted to use my latest (at that time) python, which was 3.9, but I only had 3.9 which was itself in a virtual enviroment I used a hack (my guess) which was to find out where my 3.9 python was located (while in my 3.9 virtaul enviroment). Then deactivate out of that virutual environment. But then use a directed python3 command to, and pipenv from, that directory to create a new pipenv virtual environment that would use 3.9 Unfortunately as I now know, pipenv does not yet support 3.9, so this is why I failed trying to do this.

when I ran this I was confused between pipenv and pyenv. However it is good to see what pipenv did vs pyenv: This is what I did: /bin/python/venv-webdev/bin/python -m pipenv install requests 

/Users/zintis/bin/python/mynewpythproject[532]:
$ ~/bin/python/venv-webdev/bin/python -m pipenv install  requests
Creating a virtualenv for this project...
Pipfile: /Users/zintis/bin/python/mynewpythproject/Pipfile
Using /usr/local/bin/python3 (3.9.0) to create virtualenv...
β ‹ Creating virtual environment...created virtuapyel environment CPython3.9.0.final.0-64 in 726ms
  creator CPython3Posix(dest=/Users/zintis/.local/share/virtualenvs/mynewpythproject-F9bljqZV, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/zintis/Library/Application Support/virtualenv)
    added seed packages: pip==20.3.1, setuptools==51.0.0, wheel==0.36.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

βœ” Successfully created virtual environment! 
Virtualenv location: /Users/zintis/.local/share/virtualenvs/mynewpythproject-F9bljqZV
Creating a Pipfile for this project...
Installing requests...
Adding requests to Pipfile's [packages]...
βœ” Installation Succeeded 
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
βœ” Success! 
Updated Pipfile.lock (fe5a22)!
Installing dependencies from Pipfile.lock (fe5a22)...
  🐍   β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰ 0/0 β€” 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
/Users/zintis/bin/python/mynewpythproject[533]:
$

As you can see, it looks like 3.9 was used, but directly, through

  • /usr/local/bin/python3 which I checked afterward was a symlink to
  • /usr/local/Cellar/python@3.9/3.0.01/bin/python3

What the /bin/python/venv-webdev/bin/python -m pipenv install requests command created: 

/Users/zintis/bin/python/mynewpythproject[535]:
$ tree
.
β”œβ”€β”€ Pipfile
└── Pipfile.lock

0 directories, 2 files
/Users/zintis/bin/python/mynewpythproject[536]:

It turns out doing this within a virtual environment doesn't work, as pipenv recognizes that it is in a virtual environment and so just switches to a subshell within that virtual environment. It does NOT create a new virtual environment. So….

I had to deactivate, then fix my path settings so that python3.9 was run, in the directory /usr/local/bin

Then pyenv shell did not work, as it said I needed to run pyenv init first. Ok, so running pyenv init got simply told me to append this line to my .bashprofile file: eval "$(pyenv init -)"

which I did, and then sourced .bashprofile

But then running pyenv shell did nothing. (no errors either but nada)

6 Candidate for the future. Is it venv?

Ok, it seems that the built-in venv module might be the simplest way forward. pyenv seems like yet another wrapper, like virtualenv and others. While they do add functionality, there is more to keep track of. Now, <2020-12-26 Sat>, I am thinking I can stick to using python3 -m venv mynewvenv followed by source bin/activate in that new env. That also means I would be using good ol' pip on a python version that I upgraded using brew.

6.1 When python -m venv isn't enough

Except, in specific circumstances when you need more a version of python that is different from what the main python3 version is. For example: In Jan 2021 I went to install pyATS on a -m venv virtual environment that was established while my main python was at 3.9.1 It turns out that pyATS at that time supported python version 3.6, 3.7 and 3.8 but NOT 3.9

So, I was required to install a virtual environment for pyATS that is based on 3.8 Did I use virtualenv venv-pyATS ??? No. I hacked some symlinks and manually got venv pyats on a 3.8 system.

7 As of January 2021 new approach

I had installed python3.9 using brew, but then found that ACI-API needed python3.7 (I was getting some openssl errors with python3.9.

Therefore, I installed to 3.7 using https://www.python.org/downloads/, and then followed the instructions here: [[https://www.python.org/downloads/release/python-379/ and documented in Fix for multi-version python venvs.

As documented there, I simply used the system library 3.7 to create a virtual environment using -m venv and that worked well. Just remember that PATH settings need to be kept tidy.

7.1 Difference between venv and pyenv and virtualenv

The only two tools that are true alternatives are venv and virtualenv pyenv will work with both. I first installed virtualenv and have used that as per the Cisco DevNet suggestions/instructions.

In Nov 2019, I checked brew list and it appears I have been using pyenv the whole time. Probably pyenv with virtualenv.

As of Nov, 2019 virtualenv and virtualenvwrapper are obsolete. Use pyenv and maybe try poetry for python version management and python package/venv management. (I have been doing it manaully using pyenv), OR are they saying the virtualenv with virtualevnwrapper are obsolete? but virutalenv with pyenv are still the way to go?? circa 2018

Follow up: apparently virtualenv and pyenv do NOT perform the same function Also: apparently pipenv is not recommended.

As of python 3.6 and later, can use python3.8 -m venv venv-project1 to install a virtual environment called venv-project1. As of summer of 2020, I have been using python -m venv venv-nameofproject Seems easy and it works the same way. i.e. you run source venv-nameofproject/bin/activate followed by deactivate when you are done.

  1. From stackoverflow:

    difference between venv, pyvenv, pyenv, virtualev, virtualwrapper venv is a package shipped with Python 3 which can be run using python3 -m venv It serves the same purpose as virtualenv but has only a subset of its features. But, it deprecates virtualenv, pyenv, pyenv-virtualenv. i.e. all of them. True after Python 3.4. So there's your answer for python3.4 and later. If, however, you need a virtualenv with python 2.7, you MUST use virtualenv. So, good to know both.

    virtualenv Have to install in from PyPi. It is a very popular tool that creates isolated Python environments for Python libraries. If you're not familiar with this tool, I highly recommend learning it, as it is a very useful tool, and I'll be making comparisons to it for the rest of this answer.

    It works by installing a bunch of files in a directory (eg: env/), and then modifying the PATH environment variable to prefix it with a custom bin directory (eg: env/bin/). An exact copy of the python or python3 binary is placed in this directory, but Python is programmed to look for libraries relative to its path first, in the environment directory. It's not part of Python's standard library, but is officially blessed by the PyPA (Python Packaging Authority). Once activated, you can install packages in the virtual environment using pip.

    virtualenv is older than the built-in python module, venv which has been added to the main python library since pyton 3.4

    pyenv (deprecated by venv module, since python 3.4+). pyenv was used to isolate Python versions. For example, you may want to test your code against Python 2.7, 3.6, 3.7 and 3.8, so you'll need a way to switch between them. Once activated, it prefixes the PATH environment variable with ~/.pyenv/shims, where there are special files matching the Python commands (python, pip). These are not copies of the Python-shipped commands; they are special scripts that decide on the fly which version of Python to run based on the PYENV_VERSION environment variable, or the .python-version file, or the ~/.pyenv/version file. pyenv also makes the process of downloading and installing multiple Python versions easier, using the command pyenv install.

    pyenv-virtualenv (deprecated by venv module) is a plugin for pyenv by the same author as pyenv, to allow you to use pyenv and virtualenv at the same time conveniently. However, if you're using Python 3.3 or later, pyenv-virtualenv will try to run python -m venv if it is available, instead of virtualenv. You can use virtualenv and pyenv together without pyenv-virtualenv, if you don't want the convenience features.

    python3 -m venv venv-ansible Will create a directory called venv-ansible in your current directory. source ~/venv-ansible/bin/activate This will activate the virtual env deactivate will go back to your standard setup.

    1. Comparing virtualenv and -m venv i.e. subset of the features

      An excellent source of information is pypa.io which is quoted here: Virtualenv is a tool to create isolated Python environments. Since Python 3.3, a subset of it has been integrated into the standard library under the venv module. The venv module does not offer all features of this library, to name just a few more prominent:

      1. venv is slower (by not having the app-data seed method),
      2. is not as extendable,
      3. cannot create virtual environments for arbitrarily installed python

      versions (and automatically discover these),

      1. is not upgrade-able via pip,
      2. does not have as rich programmatic API (describe virtual environments

      without creating them).

      Other than that, they are the same thing.

      The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these libraries? If you install everything into your host python (e.g. python3.8) it’s easy to end up in a situation where two packages have conflicting requirements.

      Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application. Also, what if you can’t install packages into the global site-packages directory, due to not having permissions to change the host python environment?

      In all these cases, virtualenv can help you. It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments (and optionally doesn’t access the globally installed libraries either).

      But…… for most of my cases, venv has been working fine. Even upgrading a venv can be done as per. Once this is confirmed to work…. My upgrade venv-webex did not give me errors, but it did NOT work as Nov 1, 2020 because (webdev) python –version still said 3.7.7 

      (myvenv) $ python3.7 -m venv --upgrade /home/username/path/to/myvenv/
(webdev) $ python3.8 -m venv --upgrade venv-webdev     # from the right cd

      As of <2020-12-26 Sat> I have upgraded my python3 to python3.9 using brew. I will be cleaning up all .shims directories, and creating a clean 3.9 environment with virtualenv created strictly with the use of venv. i.e. python3 -m venv mynewvenv

      virtualenvwrapper is a set of extensions to virtualenv (see docs). It gives you commands like mkvirtualenv, lssitepackages, and especially workon for switching between different virtualenv directories. This tool is especially useful if you want multiple virtualenv directories. I am not using this.

      pyenv-virtualenvwrapper is a plugin for pyenv by the same author as pyenv, to conveniently integrate virtualenvwrapper into pyenv. I am not using this.

      pipenv aims to combine Pipfile, pip and virtualenv into one command on the command-line. The virtualenv directory typically gets placed in ~/.local/share/virtualenvs/XXX, with XXX being a hash of the path of the project directory. This is different from virtualenv, where the directory is typically in the current working directory. pipenv is meant to be used when developing Python applications (as opposed to libraries). There are alternatives to pipenv, such as poetry, which I won't list here since this question is only about the packages that are similarly named. I am not using this.

7.2 Prerequisites for Pyenv (brew etc) Note: pyenv is depracted by -m venv

I got the following online from this link: insert link here. If you want to install Pyenv, you’ll want to use Homebrew.

brew install pyenv.

You’ll also need readline, a standard GNU command-line management library that’s not included by default in OS X:

brew install readline

Now, you want to download your python of choice with pyenv, and turn it on. Easy! Just run pyenv install 2.7.16 (or any other version number).

Sadly, this won’t work: ERROR: The Python zlib extension was not compiled. Missing the zlib? This is documented in the pyenv FAQ however, which recommends setting

CFLAGS="-I$(xcrun --show-sdk-path)/usr/include" pyenv install 2.7.16 when you’re installing.

This sets options used by the Python compiler to build your new Python install, adding the XCode developer tools path first, so that the zlib libraries can be found.

This should now successfully install.

Undocumented though, is that this is not enough to give you a truly working Python install, especially if you want to install packages later on that require native compilation too.

I ended up with a whole stream of errors trying to install Jupyter and it’s dependencies, particularly trying to get Sqlite extensions working happily. You’ll quickly start hitting more errors like:

Symbol not found: _PyUnicodeUCS2_DecodeUTF8

and

Undefined symbols for architecture x86_64: β€œ_sqlite3_enable_load_extension”

It look quite a bit of googling to get a working answer, and I’ll happily admit this is somewhat cargo-culted from a whole series of StackOverflow answers and incomplete Github issues. C compilation flags are not my strong suit.

On the other hand though, for me, it now works nicely: end of quotation!

OK, so this worked, but I followed the link below and expanded on the above with the longer CLFAGS and LDFLAGS option below. 


CFLAGS="-I$(brew --prefix readline)/include -I$(brew --prefix openssl)/include -I$(xcrun --show-sdk-path)/usr/include" \
LDFLAGS="-L$(brew --prefix readline)/lib -L$(brew --prefix openssl)/lib" \
PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs2 pyenv intall -v 2.7.16

CFLAGS="-I$(brew --prefix readline)/include -I$(brew --prefix openssl)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix readline)/lib -L$(brew --prefix openssl)/lib" PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs2 \
pyenv install -v 2.7.16

Note the slashes at the end of each line: this is one single command. To explain a little of how I understand this to be working:

  1. CFlags -I is ensuring that readline header files, the correct openssl header files, and the full XCode development environment libraries are available to the compiler.
  2. LDFlags is then ensuring readline and the correct openssl actual libraries are available to the linker too.
  3. Python configure opts is telling the compiler to build a version of Python that uses UCS2 (not UCS4), so that it works with pre-compiler binaries pip installs that are expecting this.
  4. For SQLite specifically, I also needed to install SQLite through Homebrew, and the built-in OS X version doesn’t allow extension loading. That, plus the above pyenv setup seems to have got me back on the straight and narrow.

Phew! Again, I’m not an expert in this, but I have just spent a few hours fudging at it to get my Python environment working, and hopefully this’ll give you enough pointers to get there yourself too. Good luck!

Ok, back to my own notes: (Zintis)

7.2.1 pyenv Needs bash.(or tcsh read on…)

pyenv is used to isolate Python versions. It is a bash extension, that intercepts your calls to python, pip, to direct them to one of several of the system python tool-chains.

It works very well, but ONLY works in Bash. ( not true… read futher for tcsh )

Once activated it prefixes the PATH environment variable with ~/.pyenv/shims. That directory has special files that match the Python commands, python and pip. These are actually scripts that figure out on the fly which python and which pip to run.

That decision is based on the PYENVVERSION environement variable or the .pyenv-version file. or the ~/.pyenv/version file.

pynev also makes downloading and installing multiple Python versions easier using the pyenv install command. I used this in early 2019, but not since.

I ran pyenv install -v 2.7.16

pyenv versions to list the installed versions of python in the virutal env.

The * indicates which version is the currently active one.

  • system (set by Users/zintis.pyenv/version)

2.7.11 2.7.16 3.6.5

Although I am not using this in 2020, it is still around. Look at my output: 


/Users/zintis/[532]:
$ pyenv versions
  system
  2.7.16
  3.7.5
* 3.7.7 (set by PYENV_VERSION environment variable)
/Users/zintis/bin/python/bin/first-steps[533]:

7.2.2 after setting in bash, tcsh is NOT changed.

Will have to figure out how to manage this; Either use manual changing of paths or try to do the equivalent in .csrhc …. ? Why can't I just set the path variable to have ~/.pyenv/shims at the beginning of the path list? I know that the shims are all written in bash, so maybe that is the reason. I could possibly create a "shims-c" directory with re-written scripts ? Or maybe I don't have to. 


Solution: /May 15th, 2019/
Added this to my .cshrc
set path = ( ~/.pyenv/shims $path )
alias 2.7 'setenv PYENV_VERSION 2.7.16'
alias 3.6 'setenv PYENV_VERSION 3.6.5'
alias 3.7 'setenv PYENV_VERSION 3.7.4'

After which I can swap python versions used by typing 2.7 or 3.7 at the tcsh command prompt.

Update: As of 2020, I am using bash more, so here is my aliases in .bashprofile 


Solution: /June, 2020/
Added this to my .bash_profile

alias 27='export PYENV_VERSION=2.7.16'
alias 37='export PYENV_VERSION=3.7.7'

7.3 Sample changing python version on my MacBook.

I did this using pyenv in 2018. I use -m venv for python3.4 and later. 


> pyenv shell
pyenv: shell integration not enabled. Run `pyenv init' for instructions.
________________________________________________________________________________
~/.pyenv/shims (zintis) 
> pyenv init
# Load pyenv automatically by appending
# the following to ~/.bash_profile:
eval "$(pyenv init -)"
________________________________________________________________________________
~/.pyenv/shims (zintis)  I had edited .bashrc but not yet refreshed my session
> pyenv shell
pyenv: shell integration not enabled. Run `pyenv init' for instructions.
________________________________________________________________________________
~/.pyenv/shims (zintis)    
> eval "$(pyenv init -)"
________________________________________________________________________________
~/.pyenv/shims (zintis)  pyenv shell needs you to specify a python version.
> pyenv shell
pyenv: no shell-specific version configured
________________________________________________________________________________
~/.pyenv/shims (zintis) 
> pyenv shell 2.7.16
________________________________________________________________________________
What this does, is change the version file to specify 2.7.16 which is then 
referenced every time the shim directory intercepts the commands pip or python.
It also changes the PYENV_VERSION variable.  - test this.
~/.pyenv/shims (zintis) 

> printenv  # only pertinent variables are shown here:
TERM_PROGRAM=Apple_Terminal
PYENV_ROOT=/Users/zintis/.pyenv
PYENV_VERSION=2.7.16
GROUP=staff
USER=zintis
PATH=/Users/zintis/.pyenv/shims:/Users/zintis/.pyenv/versions/3.6.5/bin:\
     /usr/lcoal/bin:/usr/bin:/bin:/usr/sbin:/Users/zintis/.local/bin:\
     /Users/zintis/.pyenv/versions/3.6.5/bin:/usr/local/bin:/usr/bin:/bin:\
     /usr/sbin:/sbin:/Applications/VMware:Fusion.app/Contents/Public:\
     /opt/X11/bin:/usr/local/bin/jamf:/Users/zintis/.local/bin:\
     /usr/local/sbin:/usr/local/bin/
PYENV_SHELL=bash

________________________________________________________________________________
~/.pyenv/shims (zintis) 
> which python
/Users/zintis/.pyenv/shims/python
________________________________________________________________________________
~/.pyenv/shims (zintis) 
> python
Python 2.7.16 (default, May 14 2019, 13:26:11) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
________________________________________________________________________________
| ~/.pyenv/shims (zintis) 
> pyenv versions
  system
  2.7.11
  *2.7.16 (set by PYENV_VERSION environment variable)
  3.6.5
________________________________________________________________________________
| ~/.pyenv/shims (zintis) 
   > pyenv shell 3.6.5
   ________________________________________________________________________________
| ~/.pyenv/shims (zintis) 
   > python
   Python 3.6.5 (default, Nov 29 2018, 02:22:11) 
   [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>> quit()
   ________________________________________________________________________________
| ~/.pyenv/shims (zintis)

7.3.1 Recapping

pyenv shell 3.6.5
printenv PYENVVERSION
which python
python –version
pyenv versions
pyenv shell 2.7.16
printenv PYENVVERSION
which python
python –version
pyenv versions
pyenv shell 2.7.11
pyenv versions

7.3.2 switching back and forth in bash

  • python –version
  • pyenv shell 2.7.11
  • python –version
  • which python
  • pyenv shell 2.7.16
  • python –version
  • printenv PYENVVERSION
  • pyenv shell 3.6.5
  • printenv PYENVVERSION
  • python –version

7.3.3 switching back and forth in tcsh

  • 3.7
  • python –version
  • which python
  • pyenv versions
  • printenv PYENVVERSION
  • 2.7
  • python –version
  • which python
  • pyenv versions
  • printenv PYENVVERSION

Lots of good insights in this github.io post.

7.4 More pyenv commands

$ pyenv commands

will give you a list of all available pyenv commands, including itself, duh.

  • activate
  • commands
  • completions
  • deactivate …
  • virtualenvs
  • whence

7.5 Automatically setting pyenv version in a directory:

You can also use pyenv to define a project-specific, or local, version of Python: 


~$ pyenv global system

~$ mkdir cuting_edge

~$ cd cuting_edge/

~/cutting_edge$ pyenv local 3.4.0
~/cutting_edge$ python -V
Python 3.4.0

~/cutting_edge$ cd ..
~$ python -V
Python 2.7.6

7.6 cd Users/zintis.pyenv/versions

Can also just run pyenv versions to see which versions of python you have installed. Not to be confused with pyenv --version to see what version of pyenv you have.

8 virtualenv

This is the tool of choice as of May 2019. is pure python so works everywhere, it makes a copy of, optionally a specific version of, python and pip local to the activate environment which may or may not include links to the current system tool-chain, if it does not you can install just a known subset of libraries into that environment. As such it is almost certainly much better for testing and deployment as you know exactly which libraries, at which versions, are used and a global change will not impact other libraries.

virtualenv is deprecated. It is augmented by pyenv. However, in my environment, virtualenv is really pyenv. There is still command "virtualenv" but it is within python itself. To see it run pip list or pip freeze. My virtualenv version as of Nov 2019 is 16.7.7.

8.1 Setting up Virtual Environments "virtualenv ansible" (spring 2020)

virtualenv ansible (in the directory where you want to install this virt. env) Virtual enevironments are installed under the directory where the command virtualenv name is run. So one should cd to the directory first, before running these commands. My convention has been to have ~/bin as my root directory for python environment which means that running virtualenv newname will create a directory ~/bin/newname with all the links and added environment files below this directory.

8.1.1 First install (or upgrade) virtualenv using pip

  • pip install virtualenv # Installs the virtualenv library
  • pip install --upgrade --user virtualenv # if you have virtualenv already….

8.1.2 First install (or upgrade virtualenv NOT using pip

On Linux CentOS, I can install pip using:

  • sudo dnf install --upgrade --user virtualenv
  • sudo dnf install --upgrade virtualenv

8.1.3 Second create the virtual environment (under ~/bin directory)

  • virtualenv name # This creates the virtual environment named 'name'
  • virtualenv name --python=python3 # Specify the Python verion in this named environment
  • virtualenv name --python=python3.8 # Specify a specific version in this named environ.

8.1.4 Second create the virutal environment using the modern venv module:

  • python3 -m venv name # This creates the virtual environment named 'name'
  • python2 -m venv name # This creates the virtual environment named 'name' for python2

8.1.5 Once created activate and deactive the virtual environment (under ~/bin directory)

  • source name/bin/activate # Active this virtual environment (if you work in bash)
  • source name/bin/activate.csh # (if you work in tcsh or csh, which of course I do.)
  • deactivate # Deactive it. Does this delete the files? I'm guessing NO

As of May 31st, 2019 I have three virtual environments set up: pyenv versions

  1. ~/bin/python/venv-controllers-2.7
  2. ~/bin/python/venv-prog-basics
  3. ~/bin/python/venv-xpress

    Note: the programs used for venv-xpress are under ~/bin/dnav3 directory

As of Nov 12, 2019 my new mbp has a different 3 virtual environments: pyenv versions

  1. system
  2. 2.7.16
  3. 3.7.4
  4. *3.7.5 (set by PYENVVERSION environment variable)

>>>>>>>>> left off here Jan 30th <<<<<<<<<<<<<, 

$ ./zintis-figure-it-out.sh 

/usr/bin/python
../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Python 2.7.16

/usr/bin/python2
../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Python 2.7.16

/usr/local/bin/python3
../../../Library/Frameworks/Python.framework/Versions/3.7/bin/python3
Python 3.7.9

9 Pyenv upgrade

  • If you installed pyenv via pyenv installer: pyenv update
  • If you installed pyenv via Homebrew brew upgrade pyenv
  • If you installed pyenv via Git: cd $(pyenv root) && git pull

10 Logs of pyenv install 2.7.16

10.1 pyenv install 2.7.16

That is the command to use to install this particular version into the .pyenv versions folder. You can see all the versions you have using pyenv versions

I had some issues with zlib not found, so after some research, found that this worked: Taken from this pyenv on OSX with homebrew This github page also was a great resource in helping troubleshoot zlib issues. 


________________________________________________________________________________

> CFLAGS="-I$(xcrun --show-sdk-path)/usr/include" pyenv install 2.7.16
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-2.7.16.tar.xz...
-> https://www.python.org/ftp/python/2.7.16/Python-2.7.16.tar.xz
Installing Python-2.7.16...
python-build: use readline from homebrew   
Installed Python-2.7.16 to /Users/zintis/.pyenv/versions/2.7.16

10.2 Here's Zintis' pyenv Install log in five sections

10.2.1 Install-log1


> pyenv install -v 2.7.16
~ (zintis) 
> CFLAGS="-I$(brew --prefix readline)/include -I$(brew --prefix openssl)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix readline)/lib -L$(brew --prefix openssl)/lib" PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs2 \
python-build: use openssl from homebrew
python-build: use readline from homebrew
/var/folders/m3/8gyyp5wj2z95xdpp8lvqr08w0000gn/T/python-build.20190514130604.7108 ~
Downloading Python-2.7.16.tar.xz...
-> https://www.python.org/ftp/python/2.7.16/Python-2.7.16.tar.xz
/var/folders/m3/8gyyp5wj2z95xdpp8lvqr08w0000gn/T/python-build.20190514130604.7108/Python-2.7.16 /var/folders/m3/8gyyp5wj2z95xdpp8lvqr08w0000gn/T/python-build.20190514130604.7108 ~
Installing Python-2.7.16...
python-build: use readline from homebrew
checking build system type... x86_64-apple-darwin18.5.0
checking host system type... x86_64-apple-darwin18.5.0
checking for --enable-universalsdk... no
checking for --with-universal-archs... 32-bit
checking MACHDEP... darwin
checking EXTRAPLATDIR... $(PLATMACDIRS)
checking for --without-gcc... no
checking for gcc... clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether clang accepts -g... yes
checking for clang option to accept ISO C89... none needed
checking for --with-cxx-main=<compiler>... no
checking for c++... c++
configure: WARNING:

  By default, distutils will build C++ extension modules with "c++".
  If this is not intended, then set CXX on the configure command line.
  
checking how to run the C preprocessor... clang -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... y
.
.
.

10.2.2 Install-log2

/usr/bin/install -c -m 644 ./LICENSE /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/LICENSE.txt
if test -d /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/distutils/tests; then \
		/usr/bin/install -c -m 644 ./Modules/xxmodule.c \
			/Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/distutils/tests ; \
	fi
PYTHONPATH=/Users/zintis/.pyenv/versions/2.7.16/lib/python2.7   \
		./python.exe -E -Wi -tt /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/compileall.py \
		-d /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7 -f \
		-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
		/Users/zintis/.pyenv/versions/2.7.16/lib/python2.7
Listing /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7 ...
Compiling /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/BaseHTTPServer.py ...
Compiling /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/Bastion.py ...
Compiling /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/CGIHTTPServer.py ...
Compiling /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/ConfigParser.py ...
.
.
.
Compiling /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/xml/sax/xmlreader.py ...
Compiling /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/xmllib.py ...
Compiling /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/xmlrpclib.py ...
Compiling /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/zipfile.py ...
PYTHONPATH=/Users/zintis/.pyenv/versions/2.7.16/lib/python2.7   \
		./python.exe -E -Wi -t /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/compileall.py \
		-d /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/site-packages -f \
		-x badsyntax /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/site-packages
Listing /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/site-packages ...
PYTHONPATH=/Users/zintis/.pyenv/versions/2.7.16/lib/python2.7  \
		./python.exe -E -Wi -t -O /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/compileall.py \
		-d /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/site-packages -f \
		-x badsyntax /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/site-packages
Listing /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/site-packages ...
PYTHONPATH=/Users/zintis/.pyenv/versions/2.7.16/lib/python2.7  \
		./python.exe -E -m lib2to3.pgen2.driver /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/lib2to3/Grammar.txt
Generating grammar tables from /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/lib2to3/Grammar.txt
Writing grammar tables to /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/lib2to3/Grammar2.7.16.final.0.pickle
PYTHONPATH=/Users/zintis/.pyenv/versions/2.7.16/lib/python2.7  \
		./python.exe -E -m lib2to3.pgen2.driver /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/lib2to3/PatternGrammar.txt
Generating grammar tables from /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/lib2to3/PatternGrammar.txt
Writing grammar tables to /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/lib2to3/PatternGrammar2.7.16.final.0.pickle
Creating directory /Users/zintis/.pyenv/versions/2.7.16/include
Creating directory /Users/zintis/.pyenv/versions/2.7.16/include/python2.7
/usr/bin/install -c -m 644 ./Include/Python-ast.h /Users/zintis/.pyenv/versions/2.7.16/include/python2.7
/usr/bin/install -c -m 644 ./Include/Python.h /Users/zintis/.pyenv/versions/2.7.16/include/python2.7
/usr/bin/install -c -m 644 ./Include/abstract.h /Users/zintis/.pyenv/versions/2.7.16/include/python2.7
.
.
.

10.2.3 Install-log3

   
/usr/bin/install -c ./install-sh /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/config/install-sh
/usr/bin/install -c python-config /Users/zintis/.pyenv/versions/2.7.16/bin/python2.7-config
rm python-config
./python.exe -E ./setup.py install \
	   	--prefix=/Users/zintis/.pyenv/versions/2.7.16 \
		--install-scripts=/Users/zintis/.pyenv/versions/2.7.16/bin \
		--install-platlib=/Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/lib-dynload \
		--root=/
running install
running build
running build_ext
building dbm using ndbm
building '_Qt' extension
clang -DNDEBUG -g -fwr.....
.
.
.

10.2.4 Install-log4

   
Failed to build these modules:
_Qt                _tkinter                           

running build_scripts
running install_lib
creating /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/lib-dynload
.
.
.

10.2.5 Install-log5

   
changing mode of /Users/zintis/.pyenv/versions/2.7.16/bin/pydoc to 755
changing mode of /Users/zintis/.pyenv/versions/2.7.16/bin/smtpd.py to 755
running install_egg_info
Writing /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/lib-dynload/Python-2.7.16-py2.7.egg-info
rm /Users/zintis/.pyenv/versions/2.7.16/lib/python2.7/lib-dynload/_sysconfigdata.py*
Creating directory /Users/zintis/.pyenv/versions/2.7.16/share/man
Creating directory /Users/zintis/.pyenv/versions/2.7.16/share/man/man1
/usr/bin/install -c -m 644 ./Misc/python.man \
		/Users/zintis/.pyenv/versions/2.7.16/share/man/man1/python2.7.1
if test ! -d /Users/zintis/.pyenv/versions/2.7.16/lib/pkgconfig; then \
		echo "Creating directory /Users/zintis/.pyenv/versions/2.7.16/lib/pkgconfig"; \
		/usr/bin/install -c -d -m 755 /Users/zintis/.pyenv/versions/2.7.16/lib/pkgconfig; \
	fi
if test -f /Users/zintis/.pyenv/versions/2.7.16/bin/python -o -h /Users/zintis/.pyenv/versions/2.7.16/bin/python; \
	then rm -f /Users/zintis/.pyenv/versions/2.7.16/bin/python; \
	else true; \
	fi
(cd /Users/zintis/.pyenv/versions/2.7.16/bin; ln -s python2 python)
rm -f /Users/zintis/.pyenv/versions/2.7.16/bin/python2
(cd /Users/zintis/.pyenv/versions/2.7.16/bin; ln -s python2.7 python2)
rm -f /Users/zintis/.pyenv/versions/2.7.16/bin/python2-config
(cd /Users/zintis/.pyenv/versions/2.7.16/bin; ln -s python2.7-config python2-config)
rm -f /Users/zintis/.pyenv/versions/2.7.16/bin/python-config
(cd /Users/zintis/.pyenv/versions/2.7.16/bin; ln -s python2-config python-config)
test -d /Users/zintis/.pyenv/versions/2.7.16/lib/pkgconfig || /usr/bin/install -c -d -m 755 /Users/zintis/.pyenv/versions/2.7.16/lib/pkgconfig
rm -f /Users/zintis/.pyenv/versions/2.7.16/lib/pkgconfig/python2.pc
(cd /Users/zintis/.pyenv/versions/2.7.16/lib/pkgconfig; ln -s python-2.7.pc python2.pc)
rm -f /Users/zintis/.pyenv/versions/2.7.16/lib/pkgconfig/python.pc
(cd /Users/zintis/.pyenv/versions/2.7.16/lib/pkgconfig; ln -s python2.pc python.pc)
rm -f /Users/zintis/.pyenv/versions/2.7.16/share/man/man1/python2.1
(cd /Users/zintis/.pyenv/versions/2.7.16/share/man/man1; ln -s python2.7.1 python2.1)
rm -f /Users/zintis/.pyenv/versions/2.7.16/share/man/man1/python.1
(cd /Users/zintis/.pyenv/versions/2.7.16/share/man/man1; ln -s python2.1 python.1)
if test "xno" != "xno"  ; then \
		case no in \
			upgrade) ensurepip="--upgrade" ;; \
			install|*) ensurepip="" ;; \
		esac; \
		 ./python.exe -E -m ensurepip \
			$ensurepip --root=/ ; \
	fi
/var/folders/m3/8gyyp5wj2z95xdpp8lvqr08w0000gn/T/python-build.20190514131847.23248 ~
~
Installed Python-2.7.16 to /Users/zintis/.pyenv/versions/2.7.16

________________________________________________________________________________

10.2.6 Example workflow

From stackoverflow.com 


pyenv will handle everything you need:

My workflow (for one project to make it more readable) would be the following:

pyenv install 3.5.1
cd python_projects
mkdir myproject
cd myproject
pyenv virtualenv 3.5.1 venv_myproject  # I'm not sure I have been doing this... check it.
After that you can simply activate the virtualenv created by pyenv using

pyenv activate venv_myproject
which will open your distinct environment. Here you can do all the things you want, e.g. install your packages using pip etc. After you completed setting up the environment, you can freeze the environment and create a requirements file:

pip freeze > requirements.txt

11 Overlap/Conflicts between Brew, pyenv, after activate.

11.1 Issue when brew upgrade clobbered 3.7.4

I issued a brew upgrade sqlite which, as a side-effect, upgraded the python dependency from 3.7.4 to 3.7.5. By itself that was not an issue, however my pyenv still was pointing to 3.7.4 and that was no longer there.

Result was that python 3.7.5 was installed in /usr/local/bin which python pointed to /usr/local/Cellar/python/3.7.5/bin

11.1.1 Before:

/usr/local/bin[9] % lst | grep python
lrwxr-xr-x   1 zintis  admin    39 14 Aug 21:31 python-build -> ../Cellar/pyenv/1.2.13/bin/python-build
lrwxr-xr-x   1 zintis  admin    31 15 Aug 22:51 2to3 -> ../Cellar/python/3.7.4/bin/2to3
lrwxr-xr-x   1 zintis  admin    35 15 Aug 22:51 2to3-3.7 -> ../Cellar/python/3.7.4/bin/2to3-3.7
lrwxr-xr-x   1 zintis  admin    43 15 Aug 22:51 easy_install-3.7 -> ../Cellar/python/3.7.4/bin/easy_install-3.7
lrwxr-xr-x   1 zintis  admin    32 15 Aug 22:51 idle3 -> ../Cellar/python/3.7.4/bin/idle3
lrwxr-xr-x   1 zintis  admin    34 15 Aug 22:51 idle3.7 -> ../Cellar/python/3.7.4/bin/idle3.7
lrwxr-xr-x   1 zintis  admin    31 15 Aug 22:51 pip3 -> ../Cellar/python/3.7.4/bin/pip3
lrwxr-xr-x   1 zintis  admin    33 15 Aug 22:51 pip3.7 -> ../Cellar/python/3.7.4/bin/pip3.7
lrwxr-xr-x   1 zintis  admin    33 15 Aug 22:51 pydoc3 -> ../Cellar/python/3.7.4/bin/pydoc3
lrwxr-xr-x   1 zintis  admin    35 15 Aug 22:51 pydoc3.7 -> ../Cellar/python/3.7.4/bin/pydoc3.7
lrwxr-xr-x   1 zintis  admin    34 15 Aug 22:51 python3 -> ../Cellar/python/3.7.4/bin/python3
lrwxr-xr-x   1 zintis  admin    41 15 Aug 22:51 python3-config -> ../Cellar/python/3.7.4/bin/python3-config
lrwxr-xr-x   1 zintis  admin    36 15 Aug 22:51 python3.7 -> ../Cellar/python/3.7.4/bin/python3.7
lrwxr-xr-x   1 zintis  admin    43 15 Aug 22:51 python3.7-config -> ../Cellar/python/3.7.4/bin/python3.7-config
lrwxr-xr-x   1 zintis  admin    37 15 Aug 22:51 python3.7m -> ../Cellar/python/3.7.4/bin/python3.7m
lrwxr-xr-x   1 zintis  admin    44 15 Aug 22:51 python3.7m-config -> ../Cellar/python/3.7.4/bin/python3.7m-config
lrwxr-xr-x   1 zintis  admin    33 15 Aug 22:51 pyvenv -> ../Cellar/python/3.7.4/bin/pyvenv
lrwxr-xr-x   1 zintis  admin    37 15 Aug 22:51 pyvenv-3.7 -> ../Cellar/python/3.7.4/bin/pyvenv-3.7
lrwxr-xr-x   1 zintis  admin    33 15 Aug 22:51 wheel3 -> ../Cellar/python/3.7.4/bin/wheel3

11.1.2 After

/usr/local/bin[10] % lst | grep python
lrwxr-xr-x   1 zintis  admin    31 12 Nov 10:01 2to3 -> ../Cellar/python/3.7.5/bin/2to3
lrwxr-xr-x   1 zintis  admin    35 12 Nov 10:01 2to3-3.7 -> ../Cellar/python/3.7.5/bin/2to3-3.7
lrwxr-xr-x   1 zintis  admin    32 12 Nov 10:01 idle3 -> ../Cellar/python/3.7.5/bin/idle3
lrwxr-xr-x   1 zintis  admin    34 12 Nov 10:01 idle3.7 -> ../Cellar/python/3.7.5/bin/idle3.7
lrwxr-xr-x   1 zintis  admin    33 12 Nov 10:01 pydoc3 -> ../Cellar/python/3.7.5/bin/pydoc3
lrwxr-xr-x   1 zintis  admin    35 12 Nov 10:01 pydoc3.7 -> ../Cellar/python/3.7.5/bin/pydoc3.7
lrwxr-xr-x   1 zintis  admin    34 12 Nov 10:01 python3 -> ../Cellar/python/3.7.5/bin/python3
lrwxr-xr-x   1 zintis  admin    41 12 Nov 10:01 python3-config -> ../Cellar/python/3.7.5/bin/python3-config
lrwxr-xr-x   1 zintis  admin    36 12 Nov 10:01 python3.7 -> ../Cellar/python/3.7.5/bin/python3.7
lrwxr-xr-x   1 zintis  admin    43 12 Nov 10:01 python3.7-config -> ../Cellar/python/3.7.5/bin/python3.7-config
lrwxr-xr-x   1 zintis  admin    37 12 Nov 10:01 python3.7m -> ../Cellar/python/3.7.5/bin/python3.7m
lrwxr-xr-x   1 zintis  admin    44 12 Nov 10:01 python3.7m-config -> ../Cellar/python/3.7.5/bin/python3.7m-config
lrwxr-xr-x   1 zintis  admin    33 12 Nov 10:01 pyvenv -> ../Cellar/python/3.7.5/bin/pyvenv
lrwxr-xr-x   1 zintis  admin    37 12 Nov 10:01 pyvenv-3.7 -> ../Cellar/python/3.7.5/bin/pyvenv-3.7
lrwxr-xr-x   1 zintis  admin    31 12 Nov 10:01 pip3 -> ../Cellar/python/3.7.5/bin/pip3
lrwxr-xr-x   1 zintis  admin    33 12 Nov 10:01 pip3.7 -> ../Cellar/python/3.7.5/bin/pip3.7
lrwxr-xr-x   1 zintis  admin    43 12 Nov 10:01 easy_install-3.7 -> ../Cellar/python/3.7.5/bin/easy_install-3.7
lrwxr-xr-x   1 zintis  admin    33 12 Nov 10:01 wheel3 -> ../Cellar/python/3.7.5/bin/wheel3
lrwxr-xr-x   1 zintis  admin    39 12 Nov 10:08 python-build -> ../Cellar/pyenv/1.2.15/bin/python-build
/usr/local/bin[11] %

Also notice that ../Cellar/python had directory 3.7.4 deleted, and replaced with 3.7.5

pyenv still scripting to 3.7.4

11.2 Fix

/Users/zintis[87] % pyenv install 3.7.5
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.5.tar.xz...
-> https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tar.xz
Installing Python-3.7.5...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.7.5 to /Users/zintis/.pyenv/versions/3.7.5

/Users/zintis[88] %

11.3 Did I just install 2 copies of python 3.7.5 ???

I think I did because I now can run both 3.7.4 and 3.7.5 even though xx still only has the single 3.7.5 directory. So I must be running 3.7.4 from some place else.

11.3.1 pyenv runs Users/zintis.pyenv/versions/3.7.5/bin/python3.7

-rwxr-xr-x 2 zintis staff 3319792 12 Nov 10:29 python3.7

11.3.2 brew runs /usr/local/Cellar/python/3.7.5/bin/python3.7

which is a symlink to :

  • ../Frameworks/Python.framework/Versions/3.7/bin/pyvenv-3.7

i.e.

  • /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/bin

But listing python3.7 shows that it is too small to be actual python:

  • -rwxr-xr-x 1 zintis staff 13620 12 Nov 10:01 python3.7

11.3.3 What is the framework python really?

file python 3.7 gives back:

  • python3.7: Mach-O 64-bit executable x8664

And running it directly gives: 


/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/bin[30] 
% ./python3.7
Python 3.7.5 (default, Nov  1 2019, 02:16:32) 
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

And this is different from the pyenv version: 


/Users/zintis[37] % setenv PYENV_VERSION 3.7.5
/Users/zintis[38] % python
Python 3.7.5 (default, Nov 12 2019, 10:27:05) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

So, I am not sure I ever use the framework version, as I am always using #!/usr/bin/env python

11.4 pyenv install is what I should be running. (for python < 3.4)

Here's what I have just from pyenv (after the pyenv install 3.7.5 command) 

/Users/zintis/.pyenv/versions[34] % setenv PYENV_VERSION 3.7.4
/Users/zintis/.pyenv/versions[31] % which python
/Users/zintis/.pyenv/shims/python

/Users/zintis/.pyenv/versions[32] % python --version
Python 3.7.4

/Users/zintis/.pyenv/versions[33] % python
Python 3.7.4 (default, Aug 15 2019, 16:54:06) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

/Users/zintis/.pyenv/versions[34] % setenv PYENV_VERSION 3.7.5
/Users/zintis/.pyenv/versions[31] % which python
/Users/zintis/.pyenv/shims/python

/Users/zintis/.pyenv/versions[35] % python --version
Python 3.7.5

/Users/zintis/.pyenv/versions[36] % python
Python 3.7.5 (default, Nov 12 2019, 10:27:05) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

/Users/zintis/.pyenv/versions[37] %

11.4.1 Status as of Sept 2020

  • which python points to /Users/zintis/.pyenv/shims/python
  • python version shows 3.7.7
  • /usr/bin/env python also shows 3.7.7

11.5 Remember to upgrade the site packages as well! i.e. using pip.

Well,… the ~/.pyenv/versions shims now have all three versions:

  • 2.7.16
  • 3.7.4
  • 3.7.5

And they all have their own site packages, in ./lib/python3.7/sitepackages which means that I have to pip install all the packages I want again. i.e. 3.7.5/lib/python3.7/sitepackages did NOT contain all the packages.

11.5.1 Check that pip freeze hasn't been upgraded:


/Users/zintis/.pyenv/versions[29] % setenv PYENV_VERSION 3.7.5
/Users/zintis/.pyenv/versions[27] % pip freeze
virtualenv==16.7.7
/Users/zintis/.pyenv/versions[29] % setenv PYENV_VERSION 3.7.4
/Users/zintis/.pyenv/versions[30] % pip freeze
appnope==0.1.0
autopep8==1.4.4
backcall==0.1.0
beautifulsoup4==4.8.1
bs4==0.0.1
certifi==2019.9.11
chardet==3.0.4
decorator==4.4.0
elpy==1.999
entrypoints==0.3
flake8==3.7.8
idna==2.8
ipython==7.7.0
ipython-genutils==0.2.0
jedi==0.15.1
mccabe==0.6.1
parso==0.5.1
pexpect==4.7.0
pickleshare==0.7.5
prompt-toolkit==2.0.9
ptyprocess==0.6.0
pycodestyle==2.5.0
pyflakes==2.1.1
Pygments==2.4.2
PyYAML==5.1.2
requests==2.22.0
six==1.12.0
soupsieve==1.9.5
traitlets==4.3.2
urllib3==1.25.6
virtualenv==16.7.7
wcwidth==0.1.7
/Users/zintis/.pyenv/versions[31] %

11.5.2 Now fix that using requirements.txt file

/Users/zintis/.pyenv/versions[30] %setenv PYENV_VERSION 3.7.4
/Users/zintis/.pyenv/versions[30] % pip freeze > requirements.txt
/Users/zintis/.pyenv/versions[30] % 
/Users/zintis/.pyenv/versions[30] % setenv PYENV_VERSION 3.7.5
/Users/zintis/.pyenv/versions[30] % 
/Users/zintis/.pyenv/versions[30] %pip install --upgrade pip
/Users/zintis/.pyenv/versions[30] %pip install -r requirements.txt

After that my pip packages in version 3.7.5, i.e. in ~/.pyenv/versions/3.7.5/lib/python3.7/site-packages had the same packages I had in 3.7.4

11.6 Conflict within an activated venv.

So with 3.7 alias working and setting the PYENV environment variable. This is what I had late 2019.

11.6.1 In a virtual environment the pythyon you are running is NOT the normal python:


(web-dev) /Users/zintis/bin/web-dev/zp[16] % which python
/Users/zintis/bin/web-dev/bin/python

(web-dev) /Users/zintis/bin/web-dev/zp[8] % python --version
Python 3.7.4
(web-dev) /Users/zintis/bin/web-dev/zp[9] % alias
2.7	setenv PYENV_VERSION 2.7.16
3.7	setenv PYENV_VERSION 3.7.5
deactivate	test $?_OLD_VIRTUAL_PATH != 0 && setenv PATH "$_OLD_VIRTUAL_PATH:q" && unset _OLD_VIRTUAL_PATH; rehash; test $?_OLD_VIRTUAL_PROMPT != 0 && set prompt="$_OLD_VIRTUAL_PROMPT:q" && unset _OLD_VIRTUAL_PROMPT; unsetenv VIRTUAL_ENV; test "!:*" != "nondestructive" && unalias deactivate && unalias pydoc

(web-dev) /Users/zintis/bin/web-dev/zp[13] % 3.7
(web-dev) /Users/zintis/bin/web-dev/zp[14] % python --version
Python 3.7.4
(web-dev) /Users/zintis/bin/web-dev/zp[15] % printenv PYENV_VERSION
3.7.5

11.6.2 Fixed and now in bash:

/Users/zintis/bin/python[552]: $ source venv-webdev/bin/activate
(web-dev) /Users/zintis/bin/python[553]: $ which python
/Users/zintis/.pyenv/shims/python

(web-dev) /Users/zintis/bin/python[554]: $ python --version
Python 3.7.7

(web-dev) /Users/zintis/bin/python[555]: $ alias
alias 27='export PYENV_VERSION=2.7.16'
alias 37='export PYENV_VERSION=3.7.7'

It even lets me change python versions, while in a virtual env: 

(web-dev) /Users/zintis/bin/python[556]: $ 27
(web-dev) /Users/zintis/bin/python[557]:$ python --version
Python 2.7.16

(web-dev) /Users/zintis/bin/python[558]: $ python
Python 2.7.16 (default, Aug 15 2019, 17:10:38) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

(web-dev) /Users/zintis/bin/python[559]: $

11.6.3 As of June 2019

Site packages are shown with python -m site --user-site 


/Users/zintis/bin/python[20] % which python
/Users/zintis/.pyenv/shims/python
/Users/zintis/bin/python[21] % python --version
Python 3.6.5
/Users/zintis/bin/python[22] % python -m site --user-site
/Users/zintis/.local/lib/python3.6/site-packages
/Users/zintis/bin/python[23] % 

/Users/zintis/bin[28] % source venv-xpress/bin/activate.csh
(venv-xpress) /Users/zintis/bin[29] % which python
/Users/zintis/.pyenv/shims/python
(venv-xpress) /Users/zintis/bin[30] % python -m site --user-site
/Users/zintis/.local/lib/python3.6/site-packages
(venv-xpress) /Users/zintis/bin[31] %

Reviewing:

11.6.4 pip

11.6.5 site-packages

11.6.6 specific to each pyenv

12 Misc

Pip install documention on pip.pypa.io

This doc.python-guide.org link has some good explanation of additional info. Worth the read. specifically on pipenv, a virtual environment package manager for python modules.

13 Trouble-shooting in July 2020 virtualenv

It was this troubleshooting that got me off of virtualenv and on to using the built-in python venv module, with python3 -m venv venv-ansible, but the below fix also worked by pip upgrading virtualenv.

13.1 The Problem

/Users/zintis/bin/python[63] % 
virtualenv venv-netprog-basics --python=python3

pyenv: virtualenv: command not found

The `virtualenv' command exists in these Python versions:
  3.7.5

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
/Users/zintis/bin/python[64] % 3.7
/Users/zintis/bin/python[65] % virtualenv venv-netprog-basics --python=python3
pyenv: virtualenv: command not found

The `virtualenv' command exists in these Python versions:
  3.7.5

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
/Users/zintis/bin/python[66] %

13.2 The Fix

Upgrade virtualenv with pip install --upgrade virtualenv 

/Users/zintis/bin/python[71] % pip install --upgrade virtualenv
Collecting virtualenv
  Downloading virtualenv-20.0.26-py2.py3-none-any.whl (4.9 MB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 4.9 MB 2.4 MB/s 
Requirement already satisfied, skipping upgrade: filelock<4,>=3.0.0 in /Users/zintis/.local/lib/python3.7/site-packages (from virtualenv) (3.0.12)
Requirement already satisfied, skipping upgrade: appdirs<2,>=1.4.3 in /Users/zintis/.local/lib/python3.7/site-packages (from virtualenv) (1.4.3)
Requirement already satisfied, skipping upgrade: importlib-metadata<2,>=0.12; python_version < "3.8" in /Users/zintis/.local/lib/python3.7/site-packages (from virtualenv) (1.6.0)
Collecting distlib<1,>=0.3.1
  Downloading distlib-0.3.1-py2.py3-none-any.whl (335 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 335 kB 11.6 MB/s 
Requirement already satisfied, skipping upgrade: six<2,>=1.9.0 in /Users/zintis/.local/lib/python3.7/site-packages (from virtualenv) (1.14.0)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in /Users/zintis/.local/lib/python3.7/site-packages (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv) (3.1.0)
Installing collected packages: distlib, virtualenv
  Attempting uninstall: distlib
    Found existing installation: distlib 0.3.0
    Uninstalling distlib-0.3.0:
      Successfully uninstalled distlib-0.3.0
  Attempting uninstall: virtualenv
    Found existing installation: virtualenv 20.0.18
    Uninstalling virtualenv-20.0.18:
      Successfully uninstalled virtualenv-20.0.18
Successfully installed distlib-0.3.1 virtualenv-20.0.26
/Users/zintis/bin/python[72] % virtualenv venv-netprog-basics --python=python3
created virtual environment CPython3.7.7.final.0-64 in 434ms
  creator CPython3Posix(dest=/Users/zintis/bin/python/venv-netprog-basics, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/zintis/Library/Application Support/virtualenv)
    added seed packages: pip==20.1.1, setuptools==49.1.0, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
/Users/zintis/bin/python[73] %

13.2.1 Actual install of a 2.7.16 virtual environement called controllers:

In early 2019, I successfully installed 2.7.16 for my Cisco controllers virtual environment. First, cd ~/bin virtualenv -p /Users/zintis/.pyenv/versions/2.7.16/bin/python controllers-venv

This uses the -p option which is equivalent to --python=.. and simply specifies the python to run when creating the virtual environment.

Or, (if you are ok with the defaulted python version as a base), virtualenv controllers-venv

Python 2.7.3 is the default environment on MACs. and should be left there as is So, creating virtual environments, you can specify --python=python3 in a virtual env And then when you activate that, you will be using python3

After completing your work, if you deactivate, the system will be back to using 2.7.3 However, using pyenv you can first select the python version using pyenv shell 3.6.5 after which you can use the virtualenv name commmand etc. 


/Users/zintis/bin[30] % pip install --upgrade --user virtualenv
Requirement already up-to-date: virtualenv in /Users/zintis/.local/lib/python3.6/site-packages (16.5.0)
/Users/zintis/bin[31] % virtualenv controllers
Using base prefix '/Users/zintis/.pyenv/versions/3.6.5'
New python executable in /Users/zintis/bin/controllers/bin/python3.6
Also creating executable in /Users/zintis/bin/controllers/bin/python
Installing setuptools, pip, wheel...
done.
/Users/zintis/bin[32] %

13.3 Home