{"id":259,"date":"2011-11-04T19:29:06","date_gmt":"2011-11-04T18:29:06","guid":{"rendered":"http:\/\/blog.rewolf.pl\/blog\/?p=259"},"modified":"2011-11-05T12:24:20","modified_gmt":"2011-11-05T11:24:20","slug":"random-thoughts-about-embedding-python-into-your-application","status":"publish","type":"post","link":"http:\/\/blog.rewolf.pl\/blog\/?p=259","title":{"rendered":"Random thoughts about embedding python into your application"},"content":{"rendered":"<p style=\"text-align: justify;\">In this post I want to share some of my thoughts about embedding python into <strong>C\/C++<\/strong> applications. It will not be <span style=\"font-style: italic;\">yet another python tutorial<\/span>, but just my personal feelings about some of the mechanisms that I&#8217;ve encountered during my work on <a title=\"dirtyJOE\" href=\"http:\/\/dirty-joe.com\">dirtyJOE<\/a>. I&#8217;ll describe three completely different things:<\/p>\n<ul>\n<li>Usage of <strong>FILE<\/strong>* structure by <strong>Python<\/strong> runtime<\/li>\n<li>Small differences between different <strong>Python<\/strong> versions<\/li>\n<li>Reference counting<\/li>\n<\/ul>\n<p style=\"text-align: justify;\">Above three topics are just small part of the whole <span style=\"font-style: italic;\">python embedding<\/span> topic, but they attracted me enough to write about it. So let&#8217;s start.<!--more--><\/p>\n<h3>FILE* problem (or not)<\/h3>\n<p style=\"text-align: justify;\">The easiest way to run python script inside your application is to call one of the high level functions whose names are generally matching this mask: <span style=\"color: #ff00ff;\"><strong>PyRun_*File*<\/strong><\/span><span style=\"color: #ff00ff;\"><strong>(FILE* fp, &#8230;)<\/strong><\/span>. Those functions are very convenient, because they can read and execute whole script directly from the file. There is (at least) one got-cha, in <a title=\"Python Documentation: The Very High Level Layer\" href=\"http:\/\/www.python.org\/doc\/\/current\/c-api\/veryhigh.html\">python documentation<\/a> you can read:<\/p>\n<p style=\"text-align: justify; background-color: #eaeaea; color: black; font-style: italic;\">Note also that several of these functions take FILE* parameters. <span style=\"color: #ff0000;\"><strong>One particular issue which needs to be handled carefully is that the FILE structure for different C libraries can be different and incompatible.<\/strong><\/span> Under Windows (at least), it is possible for dynamically linked extensions to actually use different libraries, so care should be taken that FILE* parameters are only passed to these functions if it is certain that they were created by the same library that the Python runtime is using.<\/p>\n<p style=\"text-align: justify;\">From my point of view it sound a bit ridiculous, expecting that someone will use the exact version of C runtime to open a file&#8230; I&#8217;ve checked a few python versions (compiled win32 binaries from <a title=\"python.org\" href=\"http:\/\/python.org\">python.org<\/a>) and situation looks like this:<\/p>\n<ul>\n<li><strong>python25.dll<\/strong> uses <strong>msvcr71.dll<\/strong><\/li>\n<li><strong>python26.dll<\/strong> uses <strong>msvcr90.dll<\/strong><\/li>\n<li><strong>python27.dll<\/strong> uses <strong>msvcr90.dll<\/strong><\/li>\n<\/ul>\n<p style=\"text-align: justify;\">So, you need to use <span style=\"color: #ff00ff;\"><strong>fopen()<\/strong><\/span> from the specific msvcr library (Thanks god it is not using static version of C runtime). Of course you can always compile your version of python dll (that will use msvcr of your choice) and ship it with the product.<\/p>\n<p style=\"text-align: justify;\">When I was integrating python into <a title=\"dirtyJOE\" href=\"http:\/\/dirty-joe.com\"><strong>dirtyJOE<\/strong><\/a>, I decided to use <strong><span style=\"color: #ff00ff;\">Py_CompileString()<\/span><\/strong> function. It works on memory buffer with the script instead of the <strong>FILE*<\/strong> and it gives a lot of more possibilities than just running simple scripts. It also needs some additional code, especially if you want to run only specific function from the script, but it is out of the scope of this post.<\/p>\n<h3>Small differences between versions<\/h3>\n<p style=\"text-align: justify;\">I&#8217;ll describe only one peculiarity, because I&#8217;ve encountered only one that was serious enough to investigate it. <a title=\"dirtyJOE\" href=\"http:\/\/dirty-joe.com\"><strong>dirtyJOE<\/strong><\/a> can use three different python versions (<strong>2.5<\/strong>, <strong>2.6<\/strong>, <strong>2.<\/strong>7), my <strong>C++<\/strong> code was identical for every of those three versions until I&#8217;ve discovered a small bug. I was testing it against <strong>v2.7<\/strong> and everything worked fine, but when I&#8217;ve changed python to <strong>v2.6<\/strong> or <strong>v2.5<\/strong> I&#8217;ve started receiving errors from <span style=\"color: #ff00ff;\"><strong>Py_CompileString()<\/strong><\/span> function. I&#8217;ve downloaded python sources for all of three versions and I&#8217;ve started diffing (<strong>2.5<\/strong> vs <strong>2.7<\/strong>) from <span style=\"color: #ff00ff;\"><strong>Py_CompileString()<\/strong><\/span>:<\/p>\n<ul>\n<li><strong>&#8220;\\Python\\pythonrun.c&#8221;<\/strong><\/li>\n<ul>\n<li><span style=\"color: #ff00ff;\"><strong>Py_CompileString()<\/strong><\/span> &#8211; <strong><span style=\"color: #008000;\">no differences<\/span><\/strong><\/li>\n<li><strong><span style=\"color: #ff00ff;\">Py_CompileStringFlags()<\/span><\/strong> &#8211; <strong><span style=\"color: #008000;\">no differences<\/span><\/strong><\/li>\n<li><strong><span style=\"color: #ff00ff;\">PyParser_ASTFromString()<\/span><\/strong> &#8211; <strong><span style=\"color: #ffcc99;\">small differences<\/span><\/strong><\/li>\n<\/ul>\n<li><strong>&#8220;\\Parser\\parsertok.c&#8221;<\/strong><\/li>\n<ul>\n<li><strong><span style=\"color: #ff00ff;\">PyParser_ParseStringFlagsFilename(Ex)()<\/span><\/strong> &#8211; <strong><span style=\"color: #ffcc99;\">small differences<\/span><\/strong><\/li>\n<\/ul>\n<li><strong>&#8220;\\Parser\\tokenizer.c&#8221;<\/strong><\/li>\n<ul>\n<li><span style=\"color: #ff00ff;\"><strong>PyTokenizer_FromString()<\/strong><\/span> &#8211; <strong><span style=\"color: #ffcc99;\">small differences<\/span><\/strong><\/li>\n<li><strong><span style=\"color: #ff00ff;\">decode_str()<\/span><\/strong> &#8211; <span style=\"color: #ff0000;\"><strong>small difference&#8230; but&#8230;<\/strong><\/span><\/li>\n<\/ul>\n<\/ul>\n<p style=\"text-align: justify;\">&#8230;but not that small, <strong>v2.7<\/strong> includes call to mysterious function called <span style=\"color: #ff00ff;\"><strong>translate_newlines()<\/strong><\/span> which basically filters out all encountered <span style=\"color: #008000;\"><strong>&#8216;\\r&#8217;<\/strong><\/span> characters. Adding code that strips all <span style=\"color: #008000;\"><strong>&#8216;\\r&#8217;<\/strong><\/span> from the input script was rather easy task and now everything works fine (and code is still the same for every python version!).<\/p>\n<h3>Reference counting<\/h3>\n<p style=\"text-align: justify;\">All <strong>C\/C++<\/strong> code that works with <strong>Python API<\/strong> needs to care about references to <strong>PyObjects<\/strong>. <strong>PyObject*<\/strong> is a basic type for all python objects that you&#8217;ll be working on with your <strong>C\/C++<\/strong> code. Keeping track of all references sometimes can be very tedious and error prone task. <strong>Python SDK<\/strong> gave us two basic macros for reference incrementation and decrementation:<\/p>\n<ul>\n<li><strong><span style=\"color: #ff00ff;\">PyINCREF()<\/span><\/strong><\/li>\n<li><strong><span style=\"color: #ff00ff;\">PyDECREF()<\/span><\/strong><\/li>\n<\/ul>\n<p style=\"text-align: justify;\">There are few more macros described here: <a title=\"Python Documentation: Reference Counting\" href=\"http:\/\/www.python.org\/doc\/\/current\/c-api\/refcounting.html\" target=\"_blank\">http:\/\/www.python.org\/doc\/\/current\/c-api\/refcounting.html<\/a><\/p>\n<p style=\"text-align: justify;\">Another thing that is worth to mention is <span style=\"font-style: italic;\">reference stealing<\/span>:<\/p>\n<p style=\"text-align: justify; background-color: #eaeaea; color: black; font-style: italic;\">(&#8230;)when a calling function passes in a reference to an object, there are two possibilities: the function steals a reference to the object, or it does not. Stealing a reference means that when you pass a reference to a function, that function assumes that it now owns that reference, and you are not responsible for it any longer.<\/p>\n<p style=\"text-align: justify;\">Most functions does not steal references, so if the function steals the reference it is explicitly stated in the documentation (<span style=\"color: #008000; font-style: italic;\">&#8220;This function \u201csteals\u201d a reference (&#8230;)&#8221;<\/span>).<\/p>\n<p style=\"text-align: justify;\"><strong>PyObjects<\/strong> returned form a function can also behave in two ways: it can be a new reference or it can be <span style=\"font-style: italic;\">borrowed<\/span> reference. Borrowed reference means that you are not the owner and you don&#8217;t need to call <span style=\"color: #ff00ff;\"><strong>Py_DECREF<\/strong><\/span> on the object unless you have called <strong><span style=\"color: #ff00ff;\">Py_INCREF<\/span><\/strong> somewhere. If the function returns <strong>PyObject*<\/strong>, documentation describes if it is new or borrowed reference:<\/p>\n<ul>\n<li><span style=\"color: #008000; font-style: italic;\">&#8220;Return value: Borrowed reference.&#8221;<\/span><\/li>\n<li><span style=\"color: #008000; font-style: italic;\">&#8220;Return value: New reference.&#8221;<\/span><\/li>\n<\/ul>\n<p style=\"text-align: justify;\">All mentioned mechanisms (and more) are described in the original python documentation that can be found here: <a title=\"Python Documentation: Objects, Types and Reference Counts\" href=\"http:\/\/www.python.org\/doc\/\/current\/c-api\/intro.html#objects-types-and-reference-counts\" target=\"_blank\">http:\/\/www.python.org\/doc\/\/current\/c-api\/intro.html#objects-types-and-reference-counts<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this post I want to share some of my thoughts about embedding python into C\/C++ applications. It will not be yet another python tutorial, but just my personal feelings about some of the mechanisms that I&#8217;ve encountered during my work on dirtyJOE. I&#8217;ll describe three completely different things: Usage of FILE* structure by Python [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[10,8],"tags":[],"_links":{"self":[{"href":"http:\/\/blog.rewolf.pl\/blog\/index.php?rest_route=\/wp\/v2\/posts\/259"}],"collection":[{"href":"http:\/\/blog.rewolf.pl\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/blog.rewolf.pl\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/blog.rewolf.pl\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/blog.rewolf.pl\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=259"}],"version-history":[{"count":32,"href":"http:\/\/blog.rewolf.pl\/blog\/index.php?rest_route=\/wp\/v2\/posts\/259\/revisions"}],"predecessor-version":[{"id":701,"href":"http:\/\/blog.rewolf.pl\/blog\/index.php?rest_route=\/wp\/v2\/posts\/259\/revisions\/701"}],"wp:attachment":[{"href":"http:\/\/blog.rewolf.pl\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=259"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/blog.rewolf.pl\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=259"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/blog.rewolf.pl\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=259"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}