Tips on how to write Python extensions in Rust with PyO3

Each programming language has strengths and weaknesses. Python affords many handy programming conventions however is computationally gradual. Rust provides you machine-level velocity and robust reminiscence security however is extra advanced than Python. The excellent news is, you may mix the 2 languages, wielding Python’s ease of use to harness Rust’s velocity and energy. The PyO3 challenge permits you to leverage the very best of each worlds by writing Python extensions in Rust.

With PyO3, you write Rust code, point out the way it interfaces with Python, then compile Rust and deploy it straight right into a Python digital setting, the place you should utilize it unobtrusively together with your Python code.

This text is a fast tour of how PyO3 works. You may learn to arrange a Python challenge with a PyO3 create, tips on how to expose Rust capabilities as a Python module, and tips on how to create Python objects like courses and exceptions in Rust.

Establishing a Python challenge with PyO3

To begin making a PyO3 challenge, you must start with a Python digital setting, or venv. This isn’t only for the sake of getting your Python challenge organized, but additionally to offer a spot to put in the Rust crate you will be constructing with PyO3. (If you have not already put in the Rust toolchain, try this now.)

The precise group of the challenge directories can differ. Within the examples proven in PyO3’s documentation, the PyO3 challenge is inbuilt a listing that incorporates the Python challenge and its digital setting. One other method is to create two subdirectories: one in your Python challenge and its venv, and the opposite for the PyO3 challenge. The latter method makes it simpler to maintain issues organized, so we’ll try this:

1. Create a brand new listing to carry each your Python and Rust initiatives. We’ll name them pyexample and rustexample, respectively.
2. Within the pyexample listing, create your digital setting and activate it. We’ll ultimately add some Python code right here. It is vital that you just carry out all of your work with each the Rust and Python code in your activated venv.
3. In your activated venv, set up the maturin bundle with pip set up maturin. maturin is the software we use to construct our Rust challenge and combine it with our Python challenge.
4. Swap to the Rust challenge listing and kind maturin init. When requested what bindings to pick, select pyo3.
5. maturin will then generate a Rust challenge in that listing, full with a Cargo.toml file that describes the challenge. Be aware that the challenge might be given the identical title because the listing it is positioned in; on this case it’s going to be rustexample.

Rust capabilities in a PyO3 challenge

Once you create a PyO3 challenge’s scaffolding with maturin, it auto-creates a code stub file in src/lib.rs. This stub incorporates code for 2 capabilities—a single pattern perform, sum_as_string, and a perform named after your challenge that exposes different capabilities as a Python module.

Here is an instance sum_as_string perform:

#[pyfunction]
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
    Okay((a + b).to_string())
}

The #[pyfunction] macro, from the pyo3 crate, signifies a given perform is to be wrapped with an interface to Python. The arguments it takes in and the outcomes it returns are all translated from and to Python varieties robotically. (It is also potential to specify Python-native varieties to soak up and return; extra on this later.)

On this instance, sum_as_string takes in two arguments that have to be translatable to a Rust-native 64-bit integer. For such a case, a Python program would cross in two Python int varieties. However even then, you’d must watch out: these int varieties would should be expressable as a 64-bit integer. Should you handed 2**65 to this perform, you’d get a runtime error as a result of a quantity that massive cannot be expressed as a 64-bit integer. (We’ll speak about one other solution to get round this limitation later.)

The return worth for this perform is a local Python sort—a PyResult object that incorporates a String. The final line of the perform returns a String, which the PyO3 wrapper robotically wraps as a Python object.

It is also potential for pyfunction to describe the signature {that a} given perform will settle for—as an illustration, if you wish to settle for a number of positional or key phrase arguments.

Python and Rust varieties in PyO3 capabilities

You may wish to get famliar with how Python and Rust varieties map to one another, and make some decisions about what varieties to make use of.

Your perform can settle for Rust varieties which might be transformed robotically from Python varieties, however this implies containers like dictionaries have to be transformed completely on the perform boundary. That may be gradual when you cross a big object, resembling an inventory with 1000’s of objects. To that finish, that is greatest accomplished when you’re passing a single worth, like an integer or a float, or container objects aren’t going to have many parts.

You can too settle for Python-native varieties on the perform boundary, and use Python-native strategies to entry them inside the perform. That is quicker on the perform boundary, so it is a more sensible choice when you’re passing container objects with an indeterminate variety of parts. However accessing container objects requires utilizing Python-native strategies which might be certain by the GIL (International Interpreter Lock), so you will have to convert any values from the thing into Rust-native varieties for velocity.

Python modules in a PyO3 challenge

pyfunction capabilities by themselves aren’t straight uncovered to Python by means of a module. To do that, we have to create a Python module object by PyO3 and expose our pyfunction capabilities by it.

The lib.rs file already has a primary model created for you, which seems like this:

#[pymodule]
fn rustexample(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
    Okay(())
}

The pymodule macro signifies the perform in query might be uncovered as a module to Python, with the identical title (rustexample). We take every of the beforehand outlined capabilities and expose them by the module utilizing the .add_function technique. This will appear a little bit boilerplate, however it gives flexibility when creating the module—for instance, by permitting you to create submodules if wanted.

Compiling a PyO3 challenge

Compiling your PyO3 challenge to be used in Python is usually fairly easy:

1. If you have not accomplished so already, activate the digital setting the place you put in maturin.
2. Set your Rust challenge as your present working listing.
3. Run the command maturin dev to construct your challenge.

The outcomes ought to look one thing like this:

(.env) PS D:Devpyo3-articlerustexample> maturin dev -r
    Updating crates.io index
    [ ... snip ... ]
  Downloaded 10 crates (3.2 MB) in 2.50s (largest was `windows-sys` at 2.6 MB)
🔗 Discovered pyo3 bindings
🐍 Discovered CPython 3.11 at D:Devpyo3-articlepyexample.envScriptspython.exe
   [ ... snip ... ]
   Compiling rustexample v0.1.0 (D:Devpyo3-articlerustexample)
    Completed launch [optimized] goal(s) in 10.86s
📦 Constructed wheel for CPython 3.11 to [ ... snip ...]
.tmpUbXtlFrustexample-0.1.0-cp311-none-win_amd64.whl
🛠 Put in rustexample-0.1.0

By default, maturin builds Rust code in pre-release mode. On this instance, we handed the -r flag to maturin to construct Rust in launch mode.

The ensuing code ought to then be put in straight in your digital setting, and you must be capable to see it with pip listing:

(.env) PS D:Devpyo3-articlerustexample> pip listing
Package deal     Model
----------- -------
maturin     0.14.12
pip         23.0
rustexample 0.1.0
setuptools  67.1.0

To check out your constructed bundle, launch the Python occasion in your digital setting and check out importing the bundle:

Python 3.11.1 (tags/v3.11.1:a7a450f, Dec  6 2022, 19:58:39)
[MSC v.1934 64 bit (AMD64)] on win32
Sort "assist", "copyright", "credit" or "license" for extra info.
>>> import rustexample
>>> rustexample
<module 'rustexample' from 'D:Devpyo3-articlepyexample
.envLibsite-packagesrustexample__init__.py'>

It must import and run like every other Python bundle.

Superior PyO3

Up to now, you have seen solely the very fundamentals of what PyO3 can do. However PyO3 helps an excellent many different Python options, a lot of which you’ll possible wish to interface with Rust code.

Large integer help

Python robotically converts integers to “massive integers,” or integers of arbitrary measurement. If you wish to cross a Python integer object right into a PyO3 perform and use it as a Rust-native massive integer, you are able to do this with pyo3::num_bigint, which makes use of the prevailing num_bigint crate. Simply keep in mind that massive integers won’t help all operations.

Parallelism

As with Cython, any purely Rust code that does not contact the Python runtime might be run outdoors of the Python GIL. You possibly can wrap such a perform within the Python::allow_threads technique to droop the GIL whereas it executes. Once more, this needs to be purely Rust code with no Python objects in use.

Holding the GIL with Rust lifetimes

PyO3 gives a solution to maintain the GIL by means of Rust’s lifetimes mechanism, which provides you a solution to take both mutable or shared entry to Python objects. Totally different object varieties have completely different GIL guidelines.

You possibly can entry a generic Python object with the PyAny sort, or you should utilize extra exact varieties like PyTuple or PyList. These are a little bit quicker, since PyO3 can generate code particular to these varieties. Irrespective of which varieties you employ, you must assume you must maintain the GIL for your entire time you are working with the thing.

If you would like a reference to a Python object outdoors the GIL—as an illustration, when you’re storing a Python object reference in a Rust struct—you should utilize the Py<T> or PyObject (primarily Py<PyAny>) varieties.

For a Rust object wrapped in a (GIL-holding) Python object—sure, that is potential!—you should utilize PyCell<T>. You’d sometimes do that when you needed to entry the Rust object whereas sustaining its Rust aliasing and reference guidelines. In that case, the wrapping Python object’s conduct would not intrude with what you wish to do. Likewise, you should utilize PyRef<T> and PyRefMut<T> to get borrowing references, static and mutable, to such objects.

Courses

You possibly can outline Python courses in PyO3 modules. Should you add the #[pyclass] attribute to a Rust struct or a fieldless enum, they are often handled as the fundamental information construction for a category. So as to add occasion strategies, you’d use #[pymethods] with an impl block for the category that incorporates the capabilities to make use of as strategies. It is also potential to create class strategies, attributes, magic strategies, slots, callable courses, and plenty of different widespread behaviors.

Hold it in thoughts that Rust’s behaviors impose some limitations. You possibly can’t present lifetime parameters for courses; all of them must work as 'static. You can also’t use generic parameters on varieties getting used as Python courses.

Exceptions

Python exceptions in PyO3 might be created in Rust code with the create_exception! macro, or by importing one of some predefined commonplace exceptions with the import_exception! macro. Be aware that, as with capabilities, you must manually add PyO3-created exceptions to a module to make them out there to Python.

Conclusion

For a very long time, constructing Python extensions sometimes meant studying C with all its minimalism and lack of native safeties. Or, you can use a software like Cython with all its idiosyncrasies. However for builders who already know Rust and wish to use it hand-in-hand with Python, PyO3 gives a handy and highly effective solution to do it.