Blog

Boost (a collection of C++ libraries) provides excellent tools for writing Python extension modules in C++. However building these modules for Python3 has some issues. This blog suggests a workaround.

Motivation

Why build Python extension modules in C++? There are at least two use cases. The first one is that there is a C++ library that you would like to port to Python. The second reason is that a pure Python library is slow or too large that further extension becomes difficult and hence a C++ port of the library is desired.

Boost Python

Boost (www.boost.org) is a collection of C++ libraries that have in many
instances been incorporated into standard C++. It provides a C++ framework that makes building Python extension modules easy. Here is an example from Boost

char const* greet()
{
   return "hello, world";
}

BOOST_PYTHON_MODULE(hello_ext)
{
    using namespace boost::python;
    def("greet", greet);
}

The def method exposes the greet function as “greet“. Calling the method from Python is simple as shown in the following code from Boost:

import hello_ext
print(hello_ext.greet())

Exposing C++ classes is easy too:

namespace { // Avoid cluttering the global namespace.

  // A friendly class.
  class hello
  {
    public:
      hello(const std::string& country) { this->country = country; }
      std::string greet() const { return "Hello from " + country; }
    private:
      std::string country;
  };

  // A function taking a hello object as an argument.
  std::string invite(const hello& w) {
    return w.greet() + "! Please come soon!";
  }
}

BOOST_PYTHON_MODULE(extending)
{
    using namespace boost::python;
    class_<hello>("hello", init<std::string>())
	// Add a regular member function.
	.def("greet", &hello::greet)
	// Add invite() as a member of hello!
	.def("invite", invite)
	;

    // Also add invite() as a regular function to the module.
    def("invite", invite);
}

Here the class hello is exposed to Python via class_<hello>. The individual methods in the class are exposed by taking the member function pointer. Notice also a function invite is exposed to Python both as member function and an external function accepting an object reference.

Calling the C++ object method is shown below:

from extending import *
hi = hello('California')
hi.greet()
invite(hi)

Creating a factory method that returns an object is more involved as we have to take care of object lifetimes. However if the factory returns an object that is independent of other objects with regards to its life time then the method can return an object. Of the method must be declared within the Module definition as well.

Issues with Boost Python

To build Boost Python we need to download the latest release of Boost. Run bootstrap. Then build the desired libraries using ./b2 with the with-<library> option. In our case we would substitute python for <library>. This does not work if both Python 2 and Python 3 are available because by default Boost constructs the build system for Python 2.

Users at StackOverflow suggest a number of options. I found Maxim Dolgov’s suggestion to be the most appropriate which is to run bootstrap with with-python=python3 option before building the libraries. That way we could specify the specific Python virtual environment if we wanted to.

Having done that and then running b2 with-python we generate the boost_python3x library where x refers to the minor version number. Now if we cd to libs/python/examples/tutorial folder and then run /path/to/b2 we will get build errors. To see what is going on it is best to run /path/to/b2 -a -n. The -a option builds everything and -n option shows the commands that would be executed. The first build error without the -n option will be about missing boost headers. This is easily fixed by typing the build command on the shell prompt with the -I option to search additional include folders. Running b2 again will result in libraries not found error. Here we need to both include the library paths with -L and modify the library names. Boost builds libboost_python3x whereas the b2 tries to link with libboost_python. In addition it does not link with the libpython3.7m. These operations have to be done manually. The following makefile might help with these issues

CFLAGS = -I/home/joe/boost_1_76_0 -I/usr/include/python3.7
LFLAGS = -L/home/joe/boost_1_76_0/stage/lib 
all : embedding extending.so

%.o : %.cpp
	"g++"   -fPIC -O0 -fno-inline -Wall $(CFLAGS)  -c $< -o $@

extending.so : extending.o
	"g++"    -o $@  -Wl,-h -Wl,$@ -shared -Wl,--start-group $< $(LFLAGS) -lboost_python37 -ldl -lpthread -lutil -lpython3.7m -Wl,--end-group -fPIC 


embedding : embedding.o
	"g++" -o $@ -Wl,--start-group  $<   $(LFLAGS) -lboost_python37  -ldl -lpthread -lutil -lpython3.7m -Wl,--end-group -fPIC 

clean:
	rm embedding embedding.o extending.so extending.o

The Python version number and LDFLAGS and CFLAGS need to be modified for local conditions.

Conclusion

While Boost provides nice tools for building Python extension modules, additional work is required for Python 3. In this blog I covered the basics of building such modules. While a lot can be done with these basics, there is more to it than what I have addressed. Check out the Boost documentation for details.

It was soon after the tsunami in January 2005 when Marie Ange Harris (née Lernie) came to visit my mother (Antoinette Selvanadin) in our Candappa street, Pondicherry, house at around 10:00 in the morning. Presently, Antoine arrived. He was meeting Marie Ange after more than fifty years. Antoine required no prodding to recount the details of the last time they had met. It was in Lernie’s house and Marie Ange was less than ten.

(Note: The conversation was in French and Tamil. An English translation is necessarily bland if I need to remain close to the original meaning of the words uttered. Besides a conversation like that is hard to reproduce on paper. The tone, the body language and the reaction is difficult to capture)

Antoine was running a newsletter that was staunchly pro-independence and for unification of Pondicherry with the rest of India. Both Teta and Lernie were former French government functionaries who were clandestinely helping Antoine. Mrs Teta (Loluima) had no clue that her husband was involved in any way. She once pointed to an article and asked Antoine who wrote it. “I don’t know” Antoine replied, “it’s an anonymous document dropped off.” Her husband had written the article and she wasn’t told about it until unionization with India (de jure transfer) was complete. By that time, of course Antoine was well settled in Germany.

Lernie on the other hand was more cautious. He did not give Antoine any paper with his handwriting on it. So Antoine went to meet Lernie in person and took a dictation. It was during one of those meetings that Antoine met Marie Ange. “You used to sit on my lap. You can’t do that now” he told Marie Ange. Admittedly Antoine is prone to exaggeration. Perhaps Marie Ange didn’t want to spoil a nice story with facts. I am not sure. She laughed as did I.

I am not sure how much Mrs Lernie, my aunt, knew. I am pretty sure she would not have approved. She did not believe that Indians could run a country on their own.

All three of the others in the living room that day have passed away. I am writing this now when my my memory is still fresh in the hope somebody in the future may be interested.

Duckduckgo is a search engine that is about 90% as good as Google. What do I mean by that? And why despite that I still recommend using Duckduckgo.

Consider a simple search query “weed remover tool.” I am looking to buy a weed remover tool. I live in Sydney, Australia and Duckduckgo (DDG) returns results pertaining to the US. I have to specify the location in the settings to get the results I am looking for. Google on the other hand lists Bunnings and US based shops without explicit prompting. In this instance the quality of the search results are if anything better for DDG, provided the location is explicitly setup than Google. I use DDG for the most part both on my desktop and on my smart-phone. Occasionally I have to switch to Google to get the specific answer I am looking for.

Why is Google better in some cases? Google has been around a lot longer that DDG. Note that every time you click on a search result you are in fact providing a response to the the search engine. The search engine uses that information to produce better results the next time. Since the vast majority of search queries are executed in Google, it has a lot more information based on user feedback to fine tune its response to future queries.

So then why use DDG? Remember the time we had more than one word processor and at least one other spread application, Lotus-123. Now there is a virtual monopoly in office suite of applications even though there is a free version, Libre Office, that does everything required. I have to pay close AU$100 every year to use MS Office at home. I have no need for all those bells and whistles and neither does the vast majority of the population. However due to networking effects, clients want to send and receive only MS Office documents. Do you really want to handover an annual subscription to Microsoft when the free version is just as good? If the average customer does not do their bit to promote competition we will land up with a monopoly which would be anti-thesis of a free market.

In the case of search engines the price you pay is privacy. Google keeps a history of all your search queries. If you live outside the US, then the US security agencies can get your search histories without your permission or notice. But Google itself has access to your search history and can tailor advertisements based on your previous search queries. This is creepy. Given that many of our purchases are impulse purchases, you actually spend a lot of money using Google, buying stuff you don’t need, at least not immediately. Given that no organisation including Google is safe from security breaches, it is still possible for at least some of the data relating to you, to land up with nefarious elements. DDG on the other hand does not keep an individual’s search history. That provides you with a lot more privacy. Note that Google pays Apple billions of dollars to let Google be the default search engine on their smart devices. This is done to preempt the use of other search engines given that most customers do not change the default search engine.

On a future date Google might decide to hide results pertaining to companies that do not advertise on Google. In that case all customers will pay a price for the monopoly.

In short the marginal benefit you get from using Google is more than offset by the price you pay in privacy and the quality of results on a future date.

There are 1700 million websites, of which around 200 million are active. Even of the active websites the vast majority are accessed less than once an hour, more like once a day. Most small businesses host their website on some paid location. In general this is a good idea because a small business would host at most one website. A host provider such as Godaddy will also provide access to tools like WordPress, MySQL, Joomla and Drupal.

The cost is reasonable… about AU$14 a month, excluding the initial overheads of domain name, SSL certificate and so on which will be incurred no what where the website is hosted.

However there are good reasons to run the website in-house. The main business case would be that it is easier to integrate the business database with the website. For example if the business sells widgets then customers can know in ‘real time’ whether the specific widget they are looking for is in stock or when they can expect the widget to be available.

If the website is hosted externally, it is possible to have the remote web server access the in-house server for the details, except, for all that trouble it is easier to host the website in-house.