With Python, the best practice of pinning all the packages in an environment at a specific version ensures that the environment can be reproduced months or even years later.
- Pinned packages in a requirements.txt file are denoted by ==. For example, requests==2.21.0. Pinned packages should never be updated except for a very good reason, such as to fix a critical bug or vulnerability.
- Conversely, unpinned packages are typically denoted by >=, which indicates that the package can be replaced by a later version. Unpinned packages are more common in development environments, where the latest version can offer bug fixes, security patches and even new functionality.
As packages age, many of them are likely to have vulnerabilities and bugs logged against them. In order to maintain the security and performance of your application, you’ll need to update these packages to a newer version that fixes the issue.
The pip package manager can be used to update one or more packages system-wide. However, if your deployment is located in a virtual environment, you should use the Pipenv package manager to update all Python packages.
NOTE: be aware that upgrading packages can break your environment by installing incompatible dependencies.
Python Package Upgrade Checklist
In general, you can use the following steps to perform a package upgrade:
1. Check that Python is installed
Before packages can be updated, ensure that a Python installation containing the necessary files needed for updating packages is in place by following the steps outlined in <Installation Requirements>
2. Get a list of all the outdated packages
To generate a list of all outdated packages:
pip list --outdated
3. Upgrade outdated packages
Depending on your operating system or virtual environment, refer to the following sections.
Update all Python Packages on Windows
The easiest way to update all packages in a Windows environment is to use pip in conjunction with Windows PowerShell:
- Open a command shell by typing ‘powershell’ in the Search Box of the Task bar
- Enter:
pip freeze | %{$_.split(‘==’)[0]} | %{pip install –upgrade $_}
This will upgrade all packages system-wide to the latest version available in the Python Package Index (PyPI).
Update all Python Packages on Linux
Linux provides a number of ways to use pip in order to upgrade Python packages, including grep and awk.
To upgrade all packages using pip with grep on Ubuntu Linux:
pip3 list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip3 install -U
To upgrade all packages using pip with awk on Ubuntu Linux:
pip3 list -o | cut -f1 -d' ' | tr " " "\n" | awk '{if(NR>=3)print}' | cut -d' ' -f1 | xargs -n1 pip3 install -U
Updating Python Packages on Windows or Linux
Pip can be used to upgrade all packages on either Windows or Linux:
- Output a list of installed packages into a requirements file (requirements.txt):
pip freeze > requirements.txt
- Edit requirements.txt, and replace all ‘==’ with ‘>=’. Use the ‘Replace All’ command in the editor.
- Upgrade all outdated packages:
pip install -r requirements.txt --upgrade
Updating all Packages in a Virtual Environment
The easiest way to update unpinned packages (i.e., packages that do not require a specific version) in a virtual environment is to run the following Python script that makes use of pip:
import pkg_resources
from subprocess import call
for dist in pkg_resources.working_set:
call("python -m pip install --upgrade " + dist.<projectname>, shell=True)
Updating all Packages in a Pipenv Environment
The simplest way to update all the unpinned packages in a specific virtual environment created with pipenv is to do the following steps:
- Activate the Pipenv shell that contains the packages to be upgraded:
pipenv shell
- Upgrade all packages:
pipenv update
Modern way to manage Python packages – ActiveState Platform
The ActiveState Platform is a cloud-based build automation and dependency management tool for Python. It provides dependency resolution for:
- Python language cores, including Python 2.7 and Python 3.5+
- Python packages and their dependencies, including:
- Transitive dependencies (ie., dependencies of dependencies)
- Linked C and Fortran libraries, so you can build data science packages
- Operating system-level dependencies for Windows, Linux, and macOS
- Shared dependencies (ie., OpenSSL)
The ActiveState Platform is the only Python package management solution that not only resolves dependencies but also provides workarounds for dependency conflicts.
Simply following the instruction prompts will resolve the conflict, eliminating dependency hell. Start by creating a new Python project, pick the latest version that applies to your project, your OS and start to add packages. Or start by simply importing your requirements.txt file and creating a Python version with all the packages you need. The Platform will automatically pick the right package versions for your environment to ensure security and reproducibility.
Just run the following command to install Python 3.9 and our package manager, the State Tool:
Windows
powershell -Command "& $([scriptblock]::Create((New-Object Net.WebClient).DownloadString('https://platform.activestate.com/dl/cli/install.ps1'))) -activate-default ActiveState-Labs/Python-3.9Beta"
Linux
sh <(curl -q https://platform.activestate.com/dl/cli/install.sh) --activate-default ActiveState-Labs/Python-3.9Beta
Now you can run state install <packagename>.
