{"id":36,"date":"2021-07-17T01:17:46","date_gmt":"2021-07-17T01:17:46","guid":{"rendered":"https:\/\/xyz.norwestcomputing.com.au\/?p=36"},"modified":"2021-07-17T10:05:32","modified_gmt":"2021-07-17T10:05:32","slug":"python3-extension-modules-with-boost","status":"publish","type":"post","link":"https:\/\/xyz.norwestcomputing.com.au\/?p=36","title":{"rendered":"Python3 Extension Modules with Boost"},"content":{"rendered":"\n

Boost (a collection of C++ libraries) provides excellent tools for writing Python extension modules in C++. However building these modules for Python3 has some issues. This blog suggests a workaround.<\/strong><\/p><\/blockquote>\n\n\n\n

Motivation<\/h2>\n\n\n\n

Why build Python extension modules in C++? There are at least two use cases. The first one is that there is a C++ library that you would like to port to Python. The second reason is that a pure Python library is slow or too large that further extension becomes difficult and hence a C++ port of the library is desired.<\/p>\n\n\n\n

Boost Python<\/h2>\n\n\n\n

Boost (www.boost.org) is a collection of C++ libraries that have in many
instances been incorporated into standard C++. It provides a C++ framework that makes building Python extension modules easy. Here is an example from Boost<\/p>\n\n\n\n

char const* greet()\n{\n   return \"hello, world\";\n}\n\nBOOST_PYTHON_MODULE(hello_ext)\n{\n    using namespace boost::python;\n    def(\"greet\", greet);\n}\n<\/code><\/pre>\n\n\n\n
The def<\/strong> method exposes the greet<\/strong> function as “greet<\/strong>“. Calling the method from Python is simple as shown in the following code from Boost:<\/p>\n\n\n\n
import hello_ext\nprint(hello_ext.greet())<\/code><\/pre>\n\n\n\n
Exposing C++ classes is easy too:<\/p>\n\n\n\n
namespace { \/\/ Avoid cluttering the global namespace.\n\n  \/\/ A friendly class.\n  class hello\n  {\n    public:\n      hello(const std::string& country) { this->country = country; }\n      std::string greet() const { return \"Hello from \" + country; }\n    private:\n      std::string country;\n  };\n\n  \/\/ A function taking a hello object as an argument.\n  std::string invite(const hello& w) {\n    return w.greet() + \"! Please come soon!\";\n  }\n}\n\nBOOST_PYTHON_MODULE(extending)\n{\n    using namespace boost::python;\n    class_<hello>(\"hello\", init<std::string>())\n\t\/\/ Add a regular member function.\n\t.def(\"greet\", &hello::greet)\n\t\/\/ Add invite() as a member of hello!\n\t.def(\"invite\", invite)\n\t;\n\n    \/\/ Also add invite() as a regular function to the module.\n    def(\"invite\", invite);\n}<\/code><\/pre>\n\n\n\n
Here the class hello<\/strong> is exposed to Python via class_<hello><\/strong>. The individual methods in the class are exposed by taking the member function pointer. Notice also a function invite<\/strong> is exposed to Python both as member function and an external function accepting an object reference.<\/p>\n\n\n\n
Calling the C++ object method is shown below:<\/p>\n\n\n\n
from extending import *\nhi = hello('California')\nhi.greet()\ninvite(hi)<\/code><\/pre>\n\n\n\n
Creating a factory method that returns an object is more involved as we have to take care of object lifetimes. However if the factory returns an object that is independent of other objects with regards to its life time then the method can return an object. Of the method must be declared within the Module definition as well. <\/p>\n\n\n\n
Issues with Boost Python<\/h2>\n\n\n\n
To build Boost Python we need to download the latest release of Boost. Run bootstrap<\/strong>. Then build the desired libraries using .\/b2<\/strong> with the with-<library> <\/strong>option. In our case we would substitute python for <library>. This does not work if both Python 2 and Python 3 are available because by default Boost constructs the build system for Python 2.<\/p>\n\n\n\n
Users at StackOverflow<\/a> suggest a number of options. I found Maxim Dolgov’s suggestion to be the most appropriate which is to run bootstrap with with-python=python3 <\/strong>option before building the libraries. That way we could specify the specific Python virtual environment if we wanted to.<\/p>\n\n\n\n
Having done that and then running b2 with-python<\/strong> we generate the boost_python3x<\/strong> library where x refers to the minor version number. Now if we cd<\/strong> to libs\/python\/examples\/tutorial<\/strong> folder and then run \/path\/to\/b2 <\/strong>we will get build errors. To see what is going on it is best to run  \/path\/to\/b2 -a -n<\/strong>. The -a option builds everything and -n option shows the commands that would be executed. The first build error without the -n<\/strong> option will be about missing boost headers. This is easily fixed by typing the build command on the shell prompt with the -I <\/strong>option to search additional include folders. Running b2<\/strong> again will result in libraries not found error. Here we need to both include the library paths with -L and modify the library names. Boost builds libboost_python3x<\/strong> whereas the b2<\/strong> tries to link with libboost_python<\/strong>. In addition it does not link with the libpython3.7m<\/strong>. These operations have to be done manually. The following makefile might help with these issues<\/p>\n\n\n\n
CFLAGS = -I\/home\/joe\/boost_1_76_0 -I\/usr\/include\/python3.7\nLFLAGS = -L\/home\/joe\/boost_1_76_0\/stage\/lib \nall : embedding extending.so\n\n%.o : %.cpp\n\t\"g++\"   -fPIC -O0 -fno-inline -Wall $(CFLAGS)  -c $< -o $@\n\nextending.so : extending.o\n\t\"g++\"    -o $@  -Wl,-h -Wl,$@ -shared -Wl,--start-group $< $(LFLAGS) -lboost_python37 -ldl -lpthread -lutil -lpython3.7m -Wl,--end-group -fPIC \n\n\nembedding : embedding.o\n\t\"g++\" -o $@ -Wl,--start-group  $<   $(LFLAGS) -lboost_python37  -ldl -lpthread -lutil -lpython3.7m -Wl,--end-group -fPIC \n\nclean:\n\trm embedding embedding.o extending.so extending.o\n<\/code><\/pre>\n\n\n\n
The Python version number and LDFLAGS and CFLAGS  need to be modified for local conditions.<\/p>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
While Boost provides nice tools for building Python extension modules, additional work is required for Python 3. In this blog I covered the basics of building such modules. While a lot can be done with these basics, there is more to it than what I have addressed. Check out the Boost documentation for details.<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Boost (a collection of C++ libraries) provides excellent tools for writing Python extension modules in C++. However building these modules for Python3 has some issues. This blog suggests a workaround<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-36","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/xyz.norwestcomputing.com.au\/index.php?rest_route=\/wp\/v2\/posts\/36","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/xyz.norwestcomputing.com.au\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/xyz.norwestcomputing.com.au\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/xyz.norwestcomputing.com.au\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/xyz.norwestcomputing.com.au\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=36"}],"version-history":[{"count":12,"href":"https:\/\/xyz.norwestcomputing.com.au\/index.php?rest_route=\/wp\/v2\/posts\/36\/revisions"}],"predecessor-version":[{"id":49,"href":"https:\/\/xyz.norwestcomputing.com.au\/index.php?rest_route=\/wp\/v2\/posts\/36\/revisions\/49"}],"wp:attachment":[{"href":"https:\/\/xyz.norwestcomputing.com.au\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=36"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/xyz.norwestcomputing.com.au\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=36"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/xyz.norwestcomputing.com.au\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=36"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}