Our latest contribution to GitHub Actions + hiop’s quickstart to build and deliver python packages with CI/CD

1. Foreword: hiop’s Python packages and CI/CD
2. CI/CD choice: our early bet on GitHub actions
3. What we added to setup-python
4. Hands-on: a minimal python package & CI/CD
5. Conclusion

1. Foreword: hiop’s Python packages and CI/CD

In hiop, we firmly believe that the DevOps processes should be smarter, rather than harder. Contrary to the traditional startup approach of building monolithic MVP technology we started from day zero with a top-quality toolchain for Continuous Integration and Continuous Delivery (CI/CD), in order to focus and specialize on microservices and micro-libraries.

The latter approach has become one of the core strengths of our tech team: we actively manage and develop many private packages in Python, Rust, and other languages. We use private repositories and a microservice toolchain based on cutting-edge standards. This philosophy allows us to have a critical competitive advantage in the deep tech landscape: a remarkable development speed and stability that is hard to match.

To build a package CI/CD toolchain that works well, we had to:

1. Cut away boilerplate code;
2. Enforce code linting and standard testing patterns (unit, integration and system tests);
3. Automate a reliable package versioning and avoid dependency hell;
4. Automate the entire release process based on version control events;

2. CI/CD choice: our early bet on GitHub actions

At the beginning of our dev experience design we compared different solutions in order to choose the tools that best fits with our needs, like Jenkins, CircleCI, Github Actions, Bitbucket pipelines and GitLab DevOps Platform.

After the comparison our early bet was GitHub Actions. There are many reasons behind our decision to adopt GitHub Actions but we can resume them into:

• Seamless Integration with GitHub: GitHub Actions offer a distinct advantage by seamlessly integrating with the GitHub platform. As we were already using GitHub as a git remote provider, it seemed reasonable for us to leverage its native CI/CD capabilities in order to define and manage workflows directly within our repositories, providing a centralized and cohesive development experience.
• Flexibility and Extendability: GitHub Actions natively support a diverse set of programming languages, frameworks, and platforms, making it adaptable to our tech stack. Also, by design, GitHub Actions are reusable: the GitHub Marketplace offers you a vast set of actions written by other developers, but you can also create our own custom actions tailored to our unique needs. Moreover, GitHub Workflows were made reusable in November 2021. This enabled us to reach a very important milestone for our needs: standardization.
• Built-in Security and Compliance: Security is a critical aspect for our CI/CD process. With Github Actions we can control access permissions, secrets, and environment variables, ensuring sensitive information remains protected. Therefore the connection with the main cloud providers could easily be configured with RBAC. GitHub moreover provides a comprehensive audit log and enables branch protection rules, helping us maintain compliance and enforce best practices within our development pipelines.
• Community and Ecosystem: GitHub Actions benefit from a thriving community and ecosystem. With an extensive collection of open-source workflows and actions, we can leverage the collective knowledge and expertise of the community to enhance our own CI/CD processes. The GitHub Marketplace acts as a hub for discovering, sharing, and collaborating on actions, allowing us to leverage existing solutions or contribute back to the community by publishing our own actions.

3. What we added to setup-python

With our latest PR to GitHub, we made GH Actions able to scan for additional configuration information from pyproject.toml.

Kudos for Dario Curreri, the author of this PR. Dario is one of hiop’s technical POs and works for the improvement and standardization of many of hiop’s internal package toolchains💡

This contribution enables the action to read the Python version from the pyproject.toml file both for standard python (PEP 621) and poetry.

More in detail, it deserializes the TOML files using the and then navigates it to extract the Python versions.

4. Hands-on: a minimal python package & CI/CD

Having a Python package well prepared and ready to be built is critical when adding it to a CI/CD pipeline: the name of the package, information on how to build it, the python version it supports, and its dependencies are just some of the metadata that can be specified to enable a complete and reliable CI/CD pipeline.

Currently, one of the most used languages inside hiop is Python. Thus, to give an example, it is critical that all our Python packages include a pyproject.toml

Why pyproject.toml

pyproject.toml is a configuration file for a Python application used to specify how the package should be built. This file was introduced in PEP 621 as a more consistent and comprehensive way for specifying build system requirements and package metadata.

We can resume the main reasons why we decided to adopt it with:

• Consistency and Unified Tool Configuration: Previously, Python packaging and distribution involved several files like setup.py, MANIFEST.in, requirements.txt and so on. The pyproject.toml is designed to store configuration for all tools in one place, from build-related tools to linters and type checkers. This makes it easier to manage and track configurations.
• Better Dependency Management: Using pyproject.toml allows developers to specify build dependencies separately from runtime dependencies, which was not previously possible with setup.py and requirements.txt. This makes the build process more reliable and less likely to encounter issues.
• Extendability and Standardization: pyproject.toml helps to easily replicate and standardize initial project configurations and the development of standards. Hence simplifying the project’s packaging and distribution.
• Better for Modern Python Development: The Python community and also some modern Python tools, like setuptools-scm or ruff, started to adopt built-in support for pyproject.toml, making it a more convenient choice for projects using these tools. Also, as the Python ecosystem evolves, more and more tools are starting to respect and leverage pyproject.toml, making it a more future-proof choice for Python project configuration.

hiop minimal python project setup

my-package
├── .github/workflows
│   ├── test.yml
│   └── publish.yml
├── src/my_package
│   ├── __about__.py
│   ├── __init__.py
│   └── hello_world.py
├── tests/my_package
│   └── test_base.py
├── docs/
│   ├── api_reference.md
│   └── index.md
├── .gitignore # needed for git
├── mkdocs.yml # needed to build docs
├── LICENSE
├── pyproject.toml
└── README.md

Main components:

1. /.github contains GitHub related contents, such as issue templates or other configuration files, but mainly, the workflows sub folder, where the workflows, that are configurable automated process made up of one or more jobs, are implemented
2. /src contains the source code of the package
3. /test contains tests that ensure the correctness of the code
4. /docs contains the documentation of the package. It is versioned as well enabling Documentation as Code
5. Others ( .gitignore, .flake8, LICENSE, etc.) are a set of configuration files easing developers’ life and some project-related files such as LICENSE and/or README.md.

This is an example configuration forpyproject.toml:

# the metadata needed to build the package. To version 
# our artifacts we use setuptools_scm. It extracts Python
# package versions from VCS metadata instead of declaring
# them as the version argument.

[build-system]
requires = ["setuptools>=45", "setuptools_scm[toml]>=6.2"]
build-backend = "setuptools.build_meta"

# the metadata about the project such as name, python
# versions, dependencies and so on
[project]
name = "my-package"
requires-python = ">=3.10"
dependencies = []
keywords = ["hiop"]
# we inject the version using setuptools_scm
dynamic = ["version"] 

# the path to the git root folder
[tool.setuptools_scm]
root = "."

[project.optional-dependencies]
dev = ["ruff", "black"]
test = ["pytest >= 5"]

And now this is a small package CI/CD using Github Actions and setup-python:

name: Publish
on:
  workflow_call:
  workflow_dispatch:

jobs:
  test: # this is a hiop best practice, always test before release!
    runs-on: ubuntu-latest
    timeout-minutes: 10
    uses: ./.github/workflows/test.yml

  release:
    needs: test
    runs-on: ubuntu-latest
    timeout-minutes: 10 # this is a hiop best practice!
    steps:
      - name: Checkout origin repo
        uses: actions/checkout@v3.5.3

      - name: Set up Python 3 🐍
        uses: actions/setup-python@v4
        with:
          cache: pip # this is a hiop best practice!
          cache-dependency-path: "**/pyproject.toml"

      - name: Install Python dependencies 🐍
        run: pip install build
      
      - name: Build 📦
        run: python -m build - wheel - outdir dist/ .

      - name: Publish distribution 📦 to Test PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          password: ${{ secrets.TEST_PYPI_API_TOKEN }}
          repository-url: https://test.pypi.org/legacy/
        
      - name: Publish distribution 📦 to PyPI
        if: startsWith(github.ref, 'refs/tags')
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          password: ${{ secrets.PYPI_API_TOKEN }}

5. Conclusion

What we are trying to do with our CI/CD flow is, from one side, to continuously improve and try to integrate more features and benefits to the dev experience, and from the other, to give back to the community and try to solve some of the problems that we are facing directly with the open source community of the libraries and tools we use in our everyday work.

There’s a unique sense of fulfilment that comes from contributing to open source. At hiop, we encourage our team to engage with software communities worldwide and actively contribute to the software they use daily. We do more than just believe in the value of open source, we actively take part in the endeavor. Our team has already made contributions in key elements of the data science toolchain, like Conda, Jupyter, Mkdocs, and Cookiecutter — and more to come 🚀

We’re proud to be part of such a vibrant community and to contribute to the evolution of open source development 🌍 Want to be part of this exciting journey? Find out more about hiop and our team at https://www.hiop.io