Simple Guide for Python Packaging

Code reuse is a very common need. It saves you time for writing the same code multiple times, enables leveraging other smart people’s work to make new things happen. Even just for one project, it helps organize code in a modular way so you can maintain each part separately. When it comes to python, it means format your project so it can be easily packaged. This is a simple instruction on how to go from nothing to a package that you can proudly put it in your portfolio to be used by other people.

Package Structure

When you create a ‘.py’ file, that is a module. You can define classes, functions, variables in that module named as the filename. When you put one or more modules in a folder and add a ‘__init__.py’ file, you create a package named as the folder. So a common package folder structure could be like this:

Example package structure from official python document.

Build a Package

Now that you have put your code in a package structure, let’s take a look at how to build a package. To do that, the most important thing is the setup configure file ‘setup.py’. You use that to specify metadata. Here is an example of the file:

More details about setup function can be found here.

Specify Dependencies

To make the package ready to work after installation, you want to tell the setup process what are the other packages it relies on so they can be automatically installed. Simple put that into ‘install_requires=[…]’ within setup function. You can optionally specify a version:

install_requires=[‘A==1.0’, ‘B>=1,<2’]

These dependencies are public packages which can be installed via pip. If you are relying on ones not on pypi, use this:

dependency_links=[‘http://github.com/user/repo/tarball/master#egg=package-1.0']

Add Additional Files

Often your package may need to include other types of files. To do that:

  1. Set ‘include_package_data=True’
  2. Create a MANIFEST.in file in the folder of setup.py
  3. Add file path to MANIFEST as ‘include /path/to/test.json’, the path is relative. You can also use search pattern like ‘*.json’.

Build Package

Now you have everything in place, let’s build the package. Building a package means creating a source code distribution (e.g. tar.gz) or a binary file (e.g. wheel). It’s simple to do.

python setup.py sdist bdist_wheel

This will create both format in ‘./dist’ folder. The files will be named based on the package name and version.

Distribute a Package

Congratulations! You now have a shiny package. Now it’s the time to make it available to other people or your future self. We are gonna upload it to Pypi so you can do your favorite ‘pip install’.

First, you need to create an account on pypi site.

Then, simply run:

twine upload dist/*

Here we assume the files under ‘dist/’ will all be uploaded to pypi. You can of course specify a particular file. Twine is a simple tool to help us interacting with pypi. When you upload for the first time, it will try to register the package on pypi.

Use a Package

At this point, you have published your own package on pypi. To use that, just run ‘pip install my_package’ to get it to your machine. Then use ‘import my_package’ in your code.

If you are developing the package locally and want to use it for test. Add the path to the package folder to $PYTHONPATH and it’s good to go.

Here are two great sources to read if you want a bit more details on python packaging: