Mixed Python/Rust project with maturin

This post describes how to create Python project using Rust for time-critical parts. While the steps described below work for me, they are definitely not the only way to do things.

Disclaimer: the instructions are obsolete and probably will not work now.

Prerequisites and setup

Prerequisites

Python

Obviously, you will need Python. I am not going to cover its installation here. The Python version I am going to use is 3.11.2. The command python -c "import sys; print(sys.version)" will print the version of the Python interpreter you have installed.

Rust

To install Rust I suggest to use rustup. I have installed it from the site and my version at this moment is 1.25.2. To check rustup installation run rustup --version. To check Rust use rustc --version (I run the nightly toolchain and my version is rustc 1.70.0-nightly (2036fdd24 2023-03-27)). In my setup you will need a nightly toolchain. You can test the installation by going to some temporary directory and executing the following commands:

cargo new foo
cd foo
cargo run

This will create an example Rust project in a subdirectory foo, compile and run it. You should see Hello, world! as an output. Then you can delete the foo subdirectory.

Poetry

We are going to use poetry to manage Python dependencies. You have to install it now. To check that poetry was installed correctly, try to run poetry --version. My version (at the moment of writing this text) is 1.4.1.

Setup

The master plan is to have tkinter Python application, which allows to enter two numbers and either add them together (using Python function) or subtract the second from the first (“time critical” part, with implementation in Rust).

Why tkinter? Because it is built-in into Python, and unlike many other GUI toolkits does not require your Unix-like system to be macOs or Linux.

All in all it will be a Python application.

Tests

It will be possible to test both Python part (using pytest) and Rust part (I use nextest but good old cargo test also should work).

Code coverage

It will be possible to produce test coverage for Python part using pytest-cov and for Rust part using tarpaulin. Please notice that I do not know how to include in the coverage Rust tests exercising Python part or Python tests exercising Rust part – teach me if you know, please!

Installer

We will end up by creating an installer for the application.

Python part

We are going to create a project with imaginative name “foobar”. Go to some directory and run poetry new foobar. Now go into foobar subdirectory. If the Python version you are going to use for the project is not your default Python version, it is time to run poetry env use <path to the proper Python interpreter>. Now I run the following commands making sure that poetry created fresh new Python virtual environment for me it is activated:

poetry install
source "$(poetry env info -p)"/bin/activate

If you are using Windows, the last line should be & ((poetry env info --path) + "\Scripts\activate.ps1") instead.

First run

Create (next to __init__.py) the file foobar/app.py with the following content:

if __name__ == "__main__":
    print("Hello, world!")

Now python foobar/app.py should print Hello, world!.

Tkinter window

Replace the content of foobar/app.py with the following:

"""Example application."""
import tkinter as tk


def plus(first: int, second: int) -> int:
    """Calculate the sum of two numbers."""
    return first + second


class App(tk.Tk):
    """The application window."""

    def __init__(self) -> None:
        super().__init__()
        self.num1 = tk.Entry(self)
        self.num1.insert(0, "0")
        self.num2 = tk.Entry(self)
        self.num2.insert(0, "0")
        self.num1.pack()
        self.num2.pack()

        self.btn_plus = tk.Button(self, text="+", command=self.add_numbers)
        self.btn_minus = tk.Button(self, text="-")
        self.btn_plus.pack(fill=tk.BOTH)
        self.btn_minus.pack(fill=tk.BOTH)

        self.result = tk.Label(self, text="0")
        self.result.pack()

    def add_numbers(self) -> None:
        """Read input values and display their sum."""
        res = plus(int(self.num1.get()), int(self.num2.get()))
        self.result.configure(text=f"{res}")


if __name__ == "__main__":
    app = App()
    app.mainloop()

If you run the application (python foobar/app.py) you will see a window with the number of controls – two input fields, into which you are supposed to enter integer numbers, two buttons ("+" and “-”), and a label, which displays the result of the operation. Pressing “+” button causes execution of Python function plus, calculating the sum of entered numbers. The button “-” currently does nothing – we will implement functionality for it in Rust.

Rust part

We will use the tool, called maturin, to create Python module out of Rust sources. Start by adding it as dev dependency to the project: poetry add --group dev maturin. After the installation is finished, run maturin --version to check it has installed successfully. At the time of writing my version is maturin 0.14.16.

We will use maturin with so called PyO3 bindings.

Necessary changes

Start by modifing “build-system” setting in pyproject.toml so it looks like

[build-system]
requires = ["maturin>=0.14,<0.15"]
build-backend = "maturin"

Also add to the same file maturin-specific settings:

[tool.maturin]
features = ["pyo3/extension-module"]
cargo-extra-args = ["--features", "pyo3/extension-module"]

Now create Cargo.toml file next to pyproject.toml with the following content:

[package]
name = "foobar_rust"
version = "0.1.0"
edition = "2021"

[lib]
name = "rust_module"
crate-type = ["cdylib", "rlib"]

[dependencies]
pyo3 = { version = "0.18.1" }

Notice that here foobar_rust is the name of Python wheel which will be built and rust_module is the name of the module as it will be seen from Python. You can run python -c "import rust_module" now and see ModuleNotFoundError – the module does not exist yet.

Minimal Rust code

We will create minimal Rust code which does absolutely nothing but allows us to build our fresh new module.

Create directory src/ and a file src/lib.rs with the following content:

use pyo3::prelude::*;

#[pymodule]
fn rust_module(_py: Python, _m: &PyModule) -> PyResult<()> {
    Ok(())
}

Here rust_module is the name of our Python module, and it should be the same as defined above in Cargo.toml. As you can see, the module has absolutely nothing in it.

Verification

Run maturin develop. This command will build the wheel and install it in the current virtual environment, created by poetry.

...
📦 Built wheel for CPython 3.11 to C:\Users\XXXXX\AppData\Local\Temp\.tmpOauvFl\foobar_rust-0.1.0-cp311-none-win_amd64.whl
🛠 Installed foobar_rust-0.1.0

After you see something like that, you can try to import module in Python again and see that there is no error: python -c "import rust_module".

Adding functionality to Rust module

Python function

To have Python function in the new module, modify src/lib.rs so it looks like this:

use pyo3::prelude::*;

#[pyfunction]
fn add(left: usize, right: usize) -> PyResult<usize> {
    println!("Hello, world!");
    Ok(left + right)
}

#[pymodule]
fn rust_module(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(add, m)?)?;
    Ok(())
}

Compile and install the module with maturin develop. Now run python and try the new function:

>>> from rust_module import add
>>> add(2, 3)
Hello, world!
5
>>>

As you can see, the new function is available in the module and works as expected.

Python class

All is swell but Python usually makes heavy use of OOP. We now will define Python class in Rust. Modify src/lib.rs to look like this:

use pyo3::prelude::*;

#[pyclass]
struct Pair {
    first: i32,
    second: i32,
}

#[pymethods]
impl Pair {
    #[new]
    fn create(first: i32, second: i32) -> Self {
        Self { first, second }
    }
    fn sub(&self) -> i32 {
        self.first - self.second
    }
}

#[pymodule]
fn rust_module(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<Pair>()?;
    Ok(())
}

Build the module with maturin develop and after running python you can check that it works:

>>> from rust_module import Pair
>>> Pair(5, 3)
<builtins.Pair object at 0x80177a710>
>>> Pair(5, 3).sub()
2
>>>

Of course, the actual address of Pair object, displayed in line 3, will be different.

Connecting Python GUI to Rust module

Now we will connect “-” button in our Python GUI to the fresh new Rust module. You have to put the following into foobar/app.py:

"""Example application."""
import tkinter as tk

from rust_module import Pair


def plus(first: int, second: int) -> int:
    """Calculate the sum of two numbers."""
    return first + second


class App(tk.Tk):
    """The application window."""

    def __init__(self) -> None:
        super().__init__()
        self.num1 = tk.Entry(self)
        self.num1.insert(0, "0")
        self.num2 = tk.Entry(self)
        self.num2.insert(0, "0")
        self.num1.pack()
        self.num2.pack()

        self.btn_plus = tk.Button(self, text="+", command=self.add_numbers)
        self.btn_minus = tk.Button(self, text="-", command=self.sub_numbers)
        self.btn_plus.pack(fill=tk.BOTH)
        self.btn_minus.pack(fill=tk.BOTH)

        self.result = tk.Label(self, text="0")
        self.result.pack()

    def add_numbers(self) -> None:
        """Read input values and display their sum."""
        res = plus(int(self.num1.get()), int(self.num2.get()))
        self.result.configure(text=f"{res}")

    def sub_numbers(self) -> None:
        """Read input values and display their difference."""
        res = Pair(int(self.num1.get()), int(self.num2.get())).sub()
        self.result.configure(text=f"{res}")


if __name__ == "__main__":
    app = App()
    app.mainloop()

Comparing to the previous version, this code now imports Pair from Rust module, uses it in a new sub_numbers method and tells “-” button to call this method when pressed.

If you run the project now (python foobar/app.py) the “-” button will work as expected.

Type checking

Unfortunately, if you use some Python typechecker (highly recommended!) such as mypy, it will complain about not knowing the types for rust_module (try to run mypy . in the project root directory). The problem can be solved by creation of Python stub file. In the root of the project (next to src/ subdirectory) create rust_module.pyi file with the following content (... is literally ..., not just some omission):

class Pair:
    def __new__(cls, first: int, second: int) -> "Pair": ...
    def sub(self) -> int: ...

This should make typechecker happy.

Pylint problem

If you are using pylint it might complain that rust_module has no name Pair (try to run pylint foobar in the project root directory). The reason seems to be ages old bug. To circumvent it, add the following lines into pyproject.toml:

[tool.pylint]
extension-pkg-allow-list = ["rust_module"]

Now pylint should be happy.

Testing

Let us add tests to our application.

Python tests

We start with tests, written in Python, for Python part.

Testing tool for Python

Python tests will be run with pytest. Let us add it as a development dependency: poetry add --group dev pytest. After installing, run pytest --version to check that it was successfully installed (at the time of writing, I am getting pytest 7.2.2 as an output). Running pytest will happily inform us: collected 0 items. As we do not have any tests yet, this is very logical.

Writing Python tests

Create subdirectory tests/ in the root of the project (next to src/ subdirectory). Create empty file tests/__init__.py – if you do not do this, pytest wont find your tests. It is quite possible that poetry already created both subdirectory and empty file for you. Now write the first test in the file tests/test_plus.py:

"""Tests for 'plus' function."""
from foobar.app import plus


def test_passing() -> None:
    """This test should pass."""
    assert 5 == plus(2, 3)


def test_failing() -> None:
    """This test should fail."""
    assert 5 != plus(2, 3)

Running pytest in the root of the project will show one passing and one failing test.

Rust tests

Rust tests will live in Rust sources.

Testing tool for Rust

I use nextest to run Rust tests. To install (globally) execute cargo install cargo-nextest. To print the version, use cargo nextest --version. Mine is cargo-nextest-nextest 0.9.51. To execute the tests, use cargo nextest run in the project root. Currently, it will say Starting 0 tests, which is not a big surprise.

Writing Rust tests

Let us add the following tests to src/lib.rs:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn passing_test() {
        let p = Pair {
            first: 3,
            second: 2,
        };
        assert_eq!(p.sub(), 1);
    }
    #[test]
    fn failing_test() {
        let p = Pair {
            first: 3,
            second: 2,
        };
        assert_ne!(p.sub(), 1);
    }
}

We again define one passing and one failing test, and executing cargo nextest run at the project root will say 2 tests run: 1 passed, 1 failed, 0 skipped, as expected.

Tests coverage

As mentioned above, I do not know how to count in coverage of Python code, coming from Rust tests, or Rust code, coming from Python tests. So we will have coverage information for Python code, coming from Python tests, and for Rust code, coming from Rust tests.

Coverage for Python code

The Python coverage tool

To generate Python coverage report, we are going to use pytest plugin called pytest-cov. Add it as a development dependency: poetry add --group dev pytest-cov.

Coverage report for Python code

To generate coverage report, pytest must be run with additional parameters: pytest --cov=foobar --cov-report html. Now in the root of the project you will have a new subfolder, and you can open the file htmlcov/index.html in a browser to see a coverage report.

Coverage for Rust code

The Rust coverage tool

To generate Rust coverage report, we are going to use the tool called tarpaulin. To install it (globally) execute cargo install cargo-tarpaulin. To get the version, use cargo tarpaulin --version; mine is cargo-tarpaulin version: 0.25.1.

Coverage report for Rust code

Please notice, that no report will be generated if you have failing tests, so remove failing_test function from src/lib.rs. When this is done, run cargo tarpaulin -o html. After the run finishes, you will see some text information about the coverage, and will find tarpaulin-report.html in the root directory of your project. It can be opened with any browser to see the coverage report.

Excluding parts of the code from the coverage report

While excluding parts of the Python code from pytest-cov report is easy (Internet is full of the corresponding information) with the Rust code it is a bit more difficult.

Let us say that you want to exclude from the report rust_module function, which assembles Python module. Start by inserting #![feature(no_coverage)] as the first line of src/lib.rs (this is the place where nightly toolchain is needed). Now add #[no_coverage] before #[pymodule], rerun tarpaulin, and – voila! – rust_module function is gone from coverage report.

Creation of installer

I will describe the process of creating the installer only for Windows and macOs. If you are using Linux or some other Unix-like system you probably know your way around.

Tools

We will need a tool called pyinstaller. Add it as a development dependency: poetry add --group dev pyinstaller… and it will fail with the meaningless message: pyinstaller requires Python <3.12,>=3.7, so it will not be satisfied for Python >=3.12,<4.0. I blame poetry. Luckily, the problem is easy to circumvent: edit pyproject.toml and make sure the Python version is specified using >= and < – not ^. You should have something like this:

python = ">=3.11,<3.12"

Now pyinstaller can be added with poetry.

Windows installer

From the root of the project, run pyinstaller foobar/app.py. There will appear fresh dist\app subdirectory with a bunch of files. You can go there and run the application:

cd .\dist\app
.\app.exe

The application will start. However, a directory full of files is not an ideal solution for Windows. You want one file, and you have two ways to achieve it: simple with pyinstaller and flexible with InstallForge.

For simple way, remove app.spec file and dist\ subfolder, created by pyinstaller, and call pyinstaller again, adding -F option. You will get exactly one file (app.exe) in dist\ subfolder.

InstallForge way

Start by installing InstallForge. Now run it. I am going to describe important settings:

• Product Name must be set. Keep it as “foobar”.
• Setting Company Name to something is a good idea.
• On Files tab for me drag-and-drop did not work (contrary to what is written there). I have pressed Add Folder button above and added my dist\app\ subdirectory.
• On Installation page I recommend to put Inculde Uninstaller checkmark – if it is set, the program can be uninstalled by standard Windows means.
• On Shortcuts page press Add… button, select “Startmenu” as Destination, enter foobar as Shortcut Name and <InstallPath>\app\app.exe as Target File (sic, with double “app”, as the program app.exe resides in app\ subdirectory).
• Enter Setup File on Build page. It is a name under which the created installer will be saved.

Now press Build button on top and run the installer when it is ready. You will have working “foobar” application installed. Unfortunately, besides opening Tkinter window, it also opens a console window. If you do not want to see the console window (most probably you do not) add -w option to pyinstaller call before building the installer.

Mac installer

From the root of the project, run pyinstaller foobar/app.py. There will appear fresh dist/app subdirectory with a bunch of files. You can go there and run the application:

cd ./dist/app
./app

The application will start. However, a directory full of files is not an ideal solution for Mac. Instead we will create .dmg file, containing application bundle .app (which is just a directory full of files, albeit having special structure).

Creation of application bundle

Remove created by pyinstaller app.spec file and dist/ subdirectory, and run pyinstaller again with -w option: pyinstaller -w foobar/app.py. Now in dist/ subdirectory in addition to the subdirectory app/ you will find application bundle app.app. Remove app/ subdirectory (you do not need it) and verify that application bundle works by running open app.app. Measure app.app/ size, you will need it: du -sh app.app.

Creation of disk image

Now we will create the disk image file, containing the built application. Go to the project root folder and run hdiutil create -size <size> /tmp/tmp.dmg -fs HFS+ -srcfolder dist/, where size is a couple of megabytes more than you have measured above. This command will create disk image file /tmp/tmp.dmg containing your application.

Compression of disk image

To compress (and make read-only) created disk image, run hdiutil convert /tmp/tmp.dmg -format UDZO -o FoobarInstall.dmg. Now you have FoobarInstall.dmg disk image which can be distributed. To verify, you can open it and drag-and-drop app.app somewhere.

Final notes

The procedure above builds Rust code in debug mode. To build it in release mode, pass additional -r option to maturin.

• Python
• Rust
• Maturin
• Reminder