Here are some rough notes on the topic of speeding up Python with modules compiled in another language , and importing libraries written in C++.
You may find a part of a Python application runs unacceptably slow and want to use a faster C++ library that does the same thing as the slow part of your application. Or you may wish to rewrite that part yourself in a faster language. Here are three approaches:
The Pybind11 project gives you a lightweight way to expose C++ language bindings for use in Python.
The Pybind11 code is header-only, meaning you don’t need to link against anything when building an existing library. Pybind11 has good support for many of the more complex C++ types including STL types. You add a small bit of code to your C++ project using Pybind11 macros that essentially annotate the functions and types you want to expose in a Python module.
It will allow you to produce an ‘.so’ file which you can import and use with no changes to your Python project code; you can also build a Python Wheel with the library code bundled in.
Cython is its own language with optional static types that compiles to C (and gets built as part of the Cython infrastructure into an object file.) It gives Python developers a way to write faster code as similar as possible to Python itself. Depending on how types are used the performance maybe little better or a whole lot better. Cython with static type declarations is a superset of the Python language.
Cython originated from the Pyrex language, which was meant to provide a Python like language to construct bindings for C++ libraries. Cython has grown way beyond Pyrex but still supports this use-case very well. You add code in Cython to describe the C and C++ code your using; you don’t change the C and C++ code.
Cython supports building Python Wheels that include compiled Cython and external C++ libraries as well.
NimPy and Nimporter
AThe Nimporter project gives you a way to write Nim code as part of a Python project and have it seamlessly built and incorporated into the application. It requires installation of the “nimporter” Python library, and uses NimPy, the library that creates language bindings from Nim to Python. It can function so smoothly because Nim, like Cython, compiles to C.
The Nim programming language will look a little more familiar to Python developers than some other statically typed languages; it makes use of significant white space indentation for code blocks like Python. It’s more or less imperative leaning, with lightweight support for object oriented programming. There’s a lot that’s different too, but it may be less of an adjustment than Rust or C++.
While using Nimporter would require learning Nim,it would provide a more performant and typesafe way to enhance a Python library’s performance compared to Cython, and without the headache of managing a C++ project like with Pybind11 (except if using CPPimport, but that changes the scope of what you can reasonably include.) Nimporter supports – but doesn’t require – using the Nimble package manager to build full Nim projects in-place if the Nim code is complex enough to warrant a full project setup.
As with Pybind11 and Cython, Nimporter supports building Python Wheels.
Nim makes it pretty easy to use C++ libraries from Nim. While it would be possible to wrap a C++ library in Nim and use it by way of Nimporter that wouldn’t be the easiest way to use the C++ library compared to Pybind11 or Cython.
Using Existing C++ Libraries in Python
For this, you’d choose between Pybind11 and Cython.
- Lightweight library
- Somewhat simpler than Cython for using C++ types
- Can build a Python Wheel with the supplied template projects
- Doesn’t require changing your Python project except to add the import of the new module
Here’s an example of using a C++ class in Python with Pybind11.
- Allows you to combine Cython code and imports from a C/C++ library
- Doesn’t require changing the C++ library code
- While writing the bindings is slightly more complex than with Pybind11 there is extensive documentation
An eexample of using C++ libraries with Cython similar to the Pybind11 example.
Rewriting Slow Python Code
For this, first of all, pick your favorite language.
- You can build as a stand-alone C++ project and use the code elsewhere
- Easy to bring in lots of existing C++ code
- No additional changes needed to your Python project
- You can write in Python, with static types where necessary for better performance
- Integrates well with Python projects
- Can also import external C++ libraries if needed
- It’s in Nim if you like that
- Minimal changes to your project
- Little to no manually written glue code required
- All source can live in the main Python project repository and can build at one time with the rest of the project
- Most typesafe and memory safe of the three options
Setting Up Pybind11 Projects
While Pybind11 is a lightweight easy to use successor to boost::Python and is easier than using Ctypes from Python, it’s still not trivial.
After trying to wedge Pybind11 into an existing project I concluded it’s a good idea to look at and perhaps use directly one of the two project templates provided: One is a CMake project, and one is a pure Python setuptools project. Both build a C++ library, generate bindings and allow for a “pip install”. The setup isn’t really complicated but setting up things just right isn’t obvious the first time through.
Use CPPimport to make Pybind11 even easier to use. It behaves similar to Nimporter, discussed later. This will be most useful if you have a self contained chunck of C++ code you want to use from Python; it would mean adding it to your Python project. It’s by far the simplest approach if it works for you.
Pybind11 is probably most appropriate when you have a large existing C++ project code base you want to allow Python applications to use as a library. Or, it might make sense when you have a C++ developer who wants to create a pure C++ project that creates a Python module. You could add Pybind11 to a project where you need to provide language bindings and Python is just one of those languages; the Python use case isn’t central to the overall project.
While it’s outside of the two use cases here, I should note Pybind11 can be used to embed the Python interpreter into a C++ application and share functions and data. You could create a Python programmable sandbox for your mainly C++ application.
When you have a large existing Python project with some slow parts Cython makes a great deal of sense; you can start by moving those slow parts to Cython and strategically introducing static type declarations. You still have the option of bringing in external C++ libraries too.
If you like Nim and don’t have a C++ library at hand to replace the slow parts of a Python application, Nimporter could make sense. If you’re doing new development, and it needs to perform as well as possible, Nim is a good choice. It will be much easier than C++ if you don’t know either language; even if you know C++ it’s attractive as it’s more memory safe and strongly typed.