View Issue Details Jump to Notes ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0014407ParaView(No Category)public2013-11-14 09:062016-08-24 08:25
ReporterAlejandro Ribes 
Assigned ToUtkarsh Ayachit 
PriorityimmediateSeverityminorReproducibilityhave not tried
StatusclosedResolutionmoved 
PlatformOSOS Version
Product Version4.0 
Target VersionFixed in Version 
Summary0014407: Embedded Python interpreter and multi-threading
Description
Hi Utkarsh,

We are having trouble with our integration of ParaView into SALOME, this concerns Python. I asked Adrien from CEA to describe the problem in English. Please find enclosed this description. This problem is quite important for us because we have an "alpha" tag of SALOME that should be done this week ...

Cheers,

Alejandro.


----------------------------------------------------------
Hi Alejandro,

Here are the problems we face with PV 4.0.1 related to the use of the Python C API. Python's embedding/wrapping logic has significantly changed between 3.98 and 4.0.1. For us, this is mainly an issue because of the Python trace generation (Python module "smtrace") which works as follows:
- Python module "smtrace" is loaded and creates a vtkSMPythonTraceObserver object
- this C++ class registers and execute Python callbacks, located also in "smtrace" module
This back and forth between Python and C++ means the Python C API is invoked from ParaView C++ code.

On the other hand, in SALOME, we need multi-threading, and we also need mutliple embedded Python interpreters (function Py_NewInterpreter()) for YACS execution graphs mainly. There is one central entity managing the whole thing (GUI/PyInterp/PyLockWrapper class). It takes care of switching the proper interpreter and thread states, and acquiring the GIL when necessary, depending on the current OS thread, and selected sub-interpreter. This logic works fine, albeit a bit complicated.

Now, when both ParaView's trace-generation callbacks and SALOME's Python code are called simultaneously, we have a conflict and the whole thing crashes (this happens when we load a script and interacts with the GUI at the same time). The two parts are unaware of each other. I thought a simple solution would be to de-activate VTK_NO_PYTHON_THREADS CMake option when building ParaView, but this is more complicated: this flag controls whether vtkPythonCommand::Execute() acquires the GIL before invoking Python C API (PyGILState_Ensure() function). Unfortunately, it is officially said in Python doc that:
"Also note that combining this functionality [Py_NewInterpreter] with PyGILState_*() APIs is delicate, because these APIs assume a bijection between Python thread states and OS-level threads, an assumption broken by the presence of sub-interpreters. It is highly recommended that you don’t switch sub-interpreters between a pair of matching PyGILState_Ensure() and PyGILState_Release() calls."
We avoid this clash in SALOME with our PyLockWrapper logic mentioned above.

I tried several hacks in both ParaView and/or SALOME, none of them is satisfactory. It either ends with a SEGFAULT or a dead-lock.

Simple solution for now is to have the trace generation done in pure C++, not in Python (by porting the Python module "smtrace" to a pure C++ impl). This avoids calling Python C API in ParaView, leaving SALOME's logic doing its work without interference. I am starting this as we speak, and KitWare's help would be more than welcome :-)
It seems trace generation was initially written in Python to save some time, but I don't think this is a requirement. Please tell me if I am wrong! (class vtkSMPythonTraceObserver is only used by smtrace module).

Longer term, if ParaView is to embed more and more Python code callbacks (as is currently done in "smtrace"), we need to come up with a better solution. I am sure Utkarsh has some ideas about this, as I saw on the VTK mail list that he faced the issue before. On our side we will try to come with a clear idea on how this could be done to satisfy both needs, and then suggest something.

Best regards,
Adrien Bruneton.
TagsNo tags attached.
ProjectKitware
Topic Name
Typeincorrect functionality
Attached Files

 Relationships

  Notes
(0038465)
Kitware Robot (administrator)
2016-08-12 09:59

Resolving issue as `moved`.

This issue tracker is no longer used. Further discussion of this issue may take place in the current ParaView Issues page linked in the banner at the top of this page.

 Issue History
Date Modified Username Field Change
2013-11-14 09:06 Alejandro Ribes New Issue
2013-11-14 09:06 Alejandro Ribes Assigned To => Utkarsh Ayachit
2016-08-12 09:59 Kitware Robot Note Added: 0038465
2016-08-12 09:59 Kitware Robot Status backlog => closed
2016-08-12 09:59 Kitware Robot Resolution open => moved
2016-08-24 08:25 Kitware Robot View Status private => public


Copyright © 2000 - 2018 MantisBT Team