Start Wrap Episode 11: A New Rope


Despite being a slow interpreter, Python is a key component in high-performance computing (HPC). Python is easy to use. C++ is fast. They come together as a comfortable blend because Python has always been friendly to low-level code. The speaker would like to explain how a new tool, pybind11, makes this approach even more attractive to HPC code. It is the only one focusing on the niceties C++11 brings in, among the many wrapping tools have been developed in the past two decades. Beyond the syntactic sugar around the Python C API, it is interesting to see how pybind11 handles the vast difference between the two languages, and what matters to HPC.


Python has been more and more important in developing scientific / technical applications. Many scientific codes need tremendous amount of computing power, and high-performance computing (HPC) is required in the core system. The requirement complicates the system in a way uncommon in other application areas of software development. An effective solution is to divide the system into multiple layers, and use a high-level but slow language to glue the low-level and fast libraries. This is how Python shines in scientific software development. Although multiple choices of language are available for the low-level code, industries prefer C++ for its compatibility, portability, and applicability. However, it’s extremely difficult for a scientist to handle the untamed complexity in C++. The hybrid-language treatment is a must. The recent development of modern C++ (11/14/17/20) makes the hybrid-language methodology more plausible. Aided by a new wrapping tool, pybind11, which utilizes the new features including lambda function, shared pointer, move semantics, perfect forwarding, and variadic template, wrapping becomes a much more pleasant task than before. For scientific applications numpy is a required dependency, and pybind11 supports it out of the box. This makes it the most convenient tool for wrapping high-performance C++ code. This talk assumes the audience have basic understandings to modern C++ and CPython and numpy implementation.



Yung-Yu Chen

HPC software developer and Python programmer.