Creating a new project for MetricQ

In this section, we explain some of the conventions that we (the metricq authors) use when setting up a new project using the library. Python packaging has many (at times confusing) options; these conventions help us have a consistent developer experience between projects, reducing maintenance overhead.

In the following, we assume the project is called metricq-example, and all filesystem paths are relative to the project root directory (/path/to/metricq-example/). The Python package built from this project is called metricq-example, and its source code lives in the subdirectory metricq_example of the project root.

Dependencies, building and installation

Build system

Any new projects should use a PEP 517-compliant build system. Use setuptools for this. The directory metricq_example contains the Python source code of the project, create a new file pyproject.toml in the project root, and declare the necessary build dependencies:

[build-system]
requires = ["setuptools>=40.6.0", "wheel"]
build-backend = 'setuptools.build_meta'

This enables tools like pip to create packages from the project.

Runtime dependencies

In setup.cfg, declare the project runtime dependencies. Under section [options], point packages to the project source directory (i.e. metricq_example). Remember to set a minimum required Python version to prevent issues at runtime. Under install_requires give a list of all runtime dependencies:

# In setup.cfg:

[options]
packages =
    metricq_example
python_requires = >=3.8
install_requires =
    metricq ~= 3.0
    # Your dependencies here

Note

Use compatible release version specifiers (foo ~= x.y, PEP 440#compatible-release) to prevent breakage caused by incompatibilities introduced in future releases of dependencies.

For compatibility with older release of pip and to enable editable installs for development, include a dummy setup.py:

# setup.py:
from setuptools import setup

setup()

Note

Keep setup.cfg the single source of truth for package metadata. Only add entries to setup.py if they otherwise cannot be determined statically. For example, metricq has to determine its dependencies at build-time: it must install a PyPI-provided version of protobuf that is compatible with the host-installed version of the protobuf-compiler, protoc.

Optional dependencies

If your project has optional features that requires additional dependencies, include them in section options.extras_require of setup.cfg. For each feature my_feature, define a new extra that lists all additional dependencies:

# In setup.cfg:

[options.extras_require]
my_feature =
    foo ~= 1.0
    bar ~= 2.0
    # ... more optional dependencies here

The package can then be installed with the feature enabled like so:

$ # Local installation
$ pip install '/path/to/metricq-example[my_feature]'
$ # Installation from PyPI
$ pip install 'metricq-example[my_feature]'

Package metadata

Also in setup.cfg, include relevant package metadata:

# In setup.cfg:

[metadata]
name = metricq-example
author = TU Dresden
description = A metricq example project
long_description = file: README.rst
long_description_content_type = text/rst
url = https://example.com/metricq-example
license = BSD 3-clause "New" or "Revised License"
license_file = LICENSE
classifiers =
    License :: OSI Approved :: BSD License
    Programming Language :: Python :: 3

The entry long_description points to a README file; use either Markdown (README.md) or RST (README.rst) formatting. The content type is inferred from the file extension, but it does not hurt to set it explicitly. Choose a license appropriate to your project and enter it; metricq itself is licensed under the terms of the BSD 3-clause “New” or “Revised License”.

Command line interfaces

setuptools allows declaration of Entry points. An entry point of type console_script makes a python function the entry point of a script that is added to the $PATH of your Python environment:

# In setup.cfg:

[options.entry_points]
console_scripts =
    metricq-example = metricq_example.cli:main

The above makes the function main() in module metricq_example/cli.py the entry point for an executable named metricq-example.

For a consistent command line experience, use the click project. Add click ~= 7.0 (or an up-to-date version) to install_requires in setup.cfg. Then, decorate the script entry point with the appropriate command line arguments and options. If you are building a metricq client, include at least options to configure the MetricQ network URL and a client token:

# In metricq_example/cli.py:
import click

...

@click.command()
@click.option(
    "--server",
    metavar="URL",
    default="amqp://localhost/",
    show_default=True,
    help="MetricQ server URL.",
)
@click.option(
    "--token",
    metavar="CLIENT_TOKEN",
    default=default,
    show_default=True,
    help="A token to identify this client on the MetricQ network.",
)
def main(server: str, token: str):
    ...

Project versioning

In order to be a good network citizen, any MetricQ client should provide a version string when asked. The single source of truth of a project’s version should be its git tags. Where possible, use a semver-compatible version scheme. Use setuptools_scm as a build dependency to create a version string that will automatically be added to the package metadata and is accessible to code at runtime:

# in pythonproject.toml

[build-system]
requires = [
    ..., # other build dependencies here
    "setuptools_scm[toml]~=6.0",
]

# ...

[tool.setuptools_scm]
write_to = "metricq_example/version.py"

On installation, this creates a file metricq_example/version.py that includes variables version (a str) and version_tuple with the parsed version information. Exclude this file from being tracked by git:

# in .gitignore
metricq_example/version.py

This file must be included in the final package, so add it to the package manifest:

# in MANIFEST.in
metricq_example/version.py

The metricq library will provide client information on request, but you will need to supply a client version string. When building a client, declare an identifier __version__ in the same module as your client class to have it be picked up automatically by the metricq library:

# in metricq_example/client.py
import metricq

# import this project's version string
from .version import version as __version__


# This could also be a Sink, HistoryClient, etc.
class MySource(metricq.Source):
    # __version__ will be picked up automatically as this client's version
    ...

If you prefer less magic, explicitly provide the version string to the client’s base class constructor:

# in metricq_example/client.py
import metricq

from .version import version as client_version


class MySource(metricq.Source):
    def __init__(self, ...):
        ...
        super().__init__(client_version=client_version, ...)

When creating a new command line tool, also add a --version option:

# In metricq_example/cli.py:
import click

from .version import version

...

@click.command()
@click.version_option(version=version)
...
def main(...):
    ...

Development setup

To enable an easy development setup, define an extra dev, that transitively includes all optional dependencies needed for a local development setup:

# In setup.cfg:

[options.extras_require]
test =
    ... # Dependencies needed for running tests
lint =
    ... # Dependencies needed to run linters
dev =
    %(test)s
    %(lint)s
    ...

The string %(foo) includes all dependencies of extra foo in another extra. Create a new virtual environment for this project, and then (with this environment activated) set up a local development environment by executing

$ pip install -e '.[dev]'

in the project directory.

Tests

We use pytest to define project tests. Create an extra test that pulls pytest, and pytest-asyncio when testing async code:

# In setup.cfg:

[options.extras_require]
test =
    pytest
    pytest-asyncio
dev =
    %(test)s
    ...

Tests are usually placed outside of application code, in files at at tests/test_*.py. Place tests for module metricq_example.foo (at metricq_example/foo.py) in tests/test_foo.py. For example, to test the function in module metricq_example.hello…

# In metricq_example/hello.py

def hello(name: str) -> str:
    return f"Hello, {name}!"

…create a test like so:

# In tests/test_hello.py

import pytest

from metricq_example.hello import hello

def test_hello():
    assert hello("Tester") == "Hello, Tester!"

Note

Use absolute imports when importing from your project, see the notes here.

Linting

We recommend a basic set of linters that (hopefully) help producing better code:

# In setup.cfg:

[options.extras_require]
lint =
    black
    check-manifest
    flake8 ~= 3.8
    flake8-bugbear
    isort ~= 5.0
    pre-commit
dev =
    %(lint)s
    ...

This includes:

black:

A code formatter. No need to spend time hand-formatting your code.

check-manifest:

Keeps track of all the files included in built packages. Prevents you from accidentally forgetting files when packaging. Whooops.

check-manifest will tell you to include/exclude files in MANIFEST.in.

flake8:

Helps you enforce some useful code styles. flake8 has plugin support; flake8-bugbear adds some helpful rules. A sensible default configuration includes the following:

# In setup.cfg

[flake8]
# Tell flake8 which packages are part of your application:
application-import-names = metricq_example, tests
# This is the black default:
max-line-length = 88
extend-exclude =
    .pytest_cache,
    # Add additional directories here to exclude from checking
    ...
# Rules to check for
select =
    # Regular flake8 rules
    C, E, F, W
    # flake8-bugbear rules
    B
    # pep8-naming rules
    N
# Rules to ignore.  Add a reason why.
ignore =
    # E203: whitespace before ':' (not PEP8 compliant)
    E203
    # E501: line too long (replaced by B950)
    E501
    # W503: line break before binary operator (not PEP8 compliant)
    W503

isort:

Automatically sorts your import statements. Keeps merge conflicts in import statements to a minimum.

pre-commit:

Adds git hooks, that automatically run other linters. Configure it in .pre-commit-config.yaml:

default_language_version:
  python: python3.9

repos:
- repo: https://gitlab.com/pycqa/flake8
  rev: 3.9.2
  hooks:
  - id: flake8
- repo: https://github.com/timothycrosley/isort
  rev: 5.8.0
  hooks:
  - id: isort
    args: ["--check", "--diff"]
- repo: https://github.com/psf/black
  rev: 21.5b1
  hooks:
  - id: black
    args: ["--check", "--diff"]
- repo: https://github.com/mgedmin/check-manifest
  rev: "0.46"
  hooks:
  - id: check-manifest