<div dir="ltr">Hi Berk,<div><br></div><div>I will try it with this patch on my dataset. I had a dataset with 11,000+ timesteps (and planning for a few more). I am not sure if it would grow linearly or quadratically with the amount of time-steps.</div>
<div><br></div><div>I am not that experienced with VTK source cde, so before I do anything silly, can you confirm the following:</div><div>- The patch is suitable for VTK 5.10 (or should I take a newer version?)</div><div>
- I apply the patch to in my VTK source (IO/vtkEnSightReader.cxx)</div><div>- Recompile VTK</div><div><br></div><div>I hope I can get back to you with results of my tests as soon as possible. Both of you already many thanks for helping me!</div>
<div><br></div><div>Best regards,</div><div>Bernhard</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 5 March 2014 18:33, Berk Geveci <span dir="ltr"><<a href="mailto:berk.geveci@kitware.com" target="_blank">berk.geveci@kitware.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I went over the EnSight Gold reader carefully and there is only one<br>
small memory issue. Instead of a clearing a collection object every<br>
timestep, it keeps adding items to it. This accumulates to something<br>
small (7.5MB over 1000 timesteps in my example). It doesn't show up as<br>
a leak because the object is properly deleted at the end. Try the<br>
attached patch. With this change, I can see that the memory footprint<br>
of your Python example stays flat. I used Instruments on the Mac for<br>
this. It shows me all memory allocations of an applications.<br>
<br>
Best,<br>
-berk<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
On Wed, Mar 5, 2014 at 9:32 AM, Berk Geveci <<a href="mailto:berk.geveci@kitware.com">berk.geveci@kitware.com</a>> wrote:<br>
> By the way, I have been looking into this as well. So far, I found no<br>
> memory issues either.<br>
><br>
> PS: The time stepping for the EnSight reader is tested. I recently<br>
> added further coverage for it also.<br>
><br>
> -berk<br>
><br>
> On Wed, Mar 5, 2014 at 4:00 AM, Bernhard Righolt <<a href="mailto:b.w.righolt@tudelft.nl">b.w.righolt@tudelft.nl</a>> wrote:<br>
>> Dear Bill,<br>
>><br>
>> With your C++ script, I am indeed facing the same issue (although the python<br>
>> code runs significantly quicker for my case).<br>
>><br>
>> I shared a few files, and a script to create more data here:<br>
>> <a href="https://www.dropbox.com/s/ndg6mwam2xdfmwq/data.tar.gz" target="_blank">https://www.dropbox.com/s/ndg6mwam2xdfmwq/data.tar.gz</a> My experience is that<br>
>> with dissimilar data, the memory leak seems to be larger, but with this data<br>
>> (which just copies the time steps), it is still present,<br>
>><br>
>> Best regards,<br>
>> Bernhard<br>
>><br>
>><br>
>> On 4 March 2014 21:27, Bill Lorensen <<a href="mailto:bill.lorensen@gmail.com">bill.lorensen@gmail.com</a>> wrote:<br>
>>><br>
>>> Attached is my C++ program and CMakeLists.txt file. But I don't think<br>
>>> this is a python issue...<br>
>>><br>
>>> On Tue, Mar 4, 2014 at 3:10 PM, Bernhard Righolt <<a href="mailto:b.w.righolt@tudelft.nl">b.w.righolt@tudelft.nl</a>><br>
>>> wrote:<br>
>>> > Dear Bill,<br>
>>> ><br>
>>> > I am a C++ speaker, so I don't mind to test your code on my current<br>
>>> > large<br>
>>> > dataset. Can you maybe share your code, such that I can test it? I've<br>
>>> > never<br>
>>> > used VTK within C++, if it solves the issues I am experiencing with<br>
>>> > Python,<br>
>>> > I am very willing to switch. I hope it is not too complicated to link<br>
>>> > the<br>
>>> > libraries correctly.<br>
>>> ><br>
>>> > Bernhard<br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > On 4 March 2014 21:05, Bill Lorensen <<a href="mailto:bill.lorensen@gmail.com">bill.lorensen@gmail.com</a>> wrote:<br>
>>> >><br>
>>> >> Bernhard,<br>
>>> >><br>
>>> >> Thanks for your patience. Since I'm not a python person, I converted<br>
>>> >> your program to C++. I ran valgrind and found no memory leaks.<br>
>>> >><br>
>>> >> Perhaps the small dataset you provided does not exercise the code that<br>
>>> >> is leaking.<br>
>>> >><br>
>>> >> Can you possible place a dataset that you know has leaksat some<br>
>>> >> accessible<br>
>>> >> site?<br>
>>> >><br>
>>> >> I will say, that I don't think any of the current ensight tests access<br>
>>> >> multiple time steps. Your data and test script will be a good starting<br>
>>> >> point for such a test.<br>
>>> >><br>
>>> >> Bill<br>
>>> >><br>
>>> >> On Tue, Mar 4, 2014 at 12:08 AM, Bernhard Righolt<br>
>>> >> <<a href="mailto:b.w.righolt@tudelft.nl">b.w.righolt@tudelft.nl</a>> wrote:<br>
>>> >> > Therr is a python script inside the archive which is the same script<br>
>>> >> > that I<br>
>>> >> > used on the large case.<br>
>>> >> ><br>
>>> >> > Bernhard<br>
>>> >> ><br>
>>> >> ><br>
>>> >> > On Mar 4, 2014 12:22 AM, "Bill Lorensen" <<a href="mailto:bill.lorensen@gmail.com">bill.lorensen@gmail.com</a>><br>
>>> >> > wrote:<br>
>>> >> >><br>
>>> >> >> This is great. Can you send a complete python script that<br>
>>> >> >> illustrates<br>
>>> >> >> the<br>
>>> >> >> problem.?<br>
>>> >> >><br>
>>> >> >> On Mar 3, 2014 2:47 PM, "Bernhard Righolt" <<a href="mailto:b.w.righolt@tudelft.nl">b.w.righolt@tudelft.nl</a>><br>
>>> >> >> wrote:<br>
>>> >> >> ><br>
>>> >> >> > Hi Bill,<br>
>>> >> >> ><br>
>>> >> >> > I added a very small sample data set. I am not sure if the memory<br>
>>> >> >> > problem I am encountering with my big data-set is measurable for<br>
>>> >> >> > this<br>
>>> >> >> > one.<br>
>>> >> >> ><br>
>>> >> >> > Anyhow, I would like to repeat that I am using Python 2.6 and VTK<br>
>>> >> >> > 5.10,<br>
>>> >> >> ><br>
>>> >> >> > Best regards,<br>
>>> >> >> > Bernhard<br>
>>> >> >> ><br>
>>> >> >> ><br>
>>> >> >> > On 3 March 2014 19:25, Bill Lorensen <<a href="mailto:bill.lorensen@gmail.com">bill.lorensen@gmail.com</a>><br>
>>> >> >> > wrote:<br>
>>> >> >> >><br>
>>> >> >> >> The ensight readers could use more testing. Can you provide a<br>
>>> >> >> >> "small"<br>
>>> >> >> >> dataset and program that cause memory leaks?<br>
>>> >> >> >><br>
>>> >> >> >> Bill<br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >> On Mon, Mar 3, 2014 at 10:38 AM, Bernhard Righolt<br>
>>> >> >> >> <<a href="mailto:b.w.righolt@tudelft.nl">b.w.righolt@tudelft.nl</a>> wrote:<br>
>>> >> >> >> > Dear all,<br>
>>> >> >> >> ><br>
>>> >> >> >> > A typical case file, looks as follows (ASCII).<br>
>>> >> >> >> ><br>
>>> >> >> >> > $ head -n 20 case.case<br>
>>> >> >> >> > FORMAT<br>
>>> >> >> >> > type: ensight gold<br>
>>> >> >> >> ><br>
>>> >> >> >> > GEOMETRY<br>
>>> >> >> >> > model: 1 sliceCentre.000.mesh<br>
>>> >> >> >> ><br>
>>> >> >> >> > VARIABLE<br>
>>> >> >> >> > vector per node: 1 U sliceCentre.*****.U<br>
>>> >> >> >> ><br>
>>> >> >> >> > TIME<br>
>>> >> >> >> > time set: 1<br>
>>> >> >> >> > number of steps: 11550<br>
>>> >> >> >> > filename start number: 1<br>
>>> >> >> >> > filename increment: 1<br>
>>> >> >> >> > time values:<br>
>>> >> >> >> > 0.002<br>
>>> >> >> >> > 0.004<br>
>>> >> >> >> > 0.006<br>
>>> >> >> >> > 0.008<br>
>>> >> >> >> > 0.01<br>
>>> >> >> >> > $<br>
>>> >> >> >> ><br>
>>> >> >> >> > The mesh (snippet only), is as follows:<br>
>>> >> >> >> ><br>
>>> >> >> >> > $ head -n 10 sliceCentre.000.mesh<br>
>>> >> >> >> > Ensight Geometry File<br>
>>> >> >> >> > =====================<br>
>>> >> >> >> > node id assign<br>
>>> >> >> >> > element id assign<br>
>>> >> >> >> > part<br>
>>> >> >> >> > 1<br>
>>> >> >> >> > sliceCentre.000.mesh<br>
>>> >> >> >> > coordinates<br>
>>> >> >> >> > 299028<br>
>>> >> >> >> > 0.00000e+00<br>
>>> >> >> >> > $ sed -n -e 897091,897098p sliceCentre.000.mesh<br>
>>> >> >> >> > 0.00000e+00<br>
>>> >> >> >> > 0.00000e+00<br>
>>> >> >> >> > 0.00000e+00<br>
>>> >> >> >> > tria3<br>
>>> >> >> >> > 595264<br>
>>> >> >> >> > 162106 162714 162799<br>
>>> >> >> >> > 162799 162226 162106<br>
>>> >> >> >> > 162714 162106 162021<br>
>>> >> >> >> ><br>
>>> >> >> >> ><br>
>>> >> >> >> > And the corresponding datafile<br>
>>> >> >> >> > $ head -n 10 sliceCentre.00001.U<br>
>>> >> >> >> > vector<br>
>>> >> >> >> > part<br>
>>> >> >> >> > 1<br>
>>> >> >> >> > coordinates<br>
>>> >> >> >> > 5.33425e-04<br>
>>> >> >> >> > 2.69515e-03<br>
>>> >> >> >> > -1.65164e-03<br>
>>> >> >> >> > 5.33425e-04<br>
>>> >> >> >> > 1.39069e-04<br>
>>> >> >> >> > 2.69515e-03<br>
>>> >> >> >> ><br>
>>> >> >> >> ><br>
>>> >> >> >> > However, I doubt if something is wrong with my datafiles<br>
>>> >> >> >> > itself,<br>
>>> >> >> >> > because I<br>
>>> >> >> >> > can easily read them in Paraview for example, the only<br>
>>> >> >> >> > difference<br>
>>> >> >> >> > is<br>
>>> >> >> >> > the<br>
>>> >> >> >> > large amount of data-files, e.g (124GB in this case)<br>
>>> >> >> >> ><br>
>>> >> >> >> > Best regards,<br>
>>> >> >> >> > Bernhard<br>
>>> >> >> >> ><br>
>>> >> >> >> ><br>
>>> >> >> >> > On 27 February 2014 22:28, Berk Geveci<br>
>>> >> >> >> > <<a href="mailto:berk.geveci@kitware.com">berk.geveci@kitware.com</a>><br>
>>> >> >> >> > wrote:<br>
>>> >> >> >> >><br>
>>> >> >> >> >> Nothing strikes me as obviously wrong in your script. Can you<br>
>>> >> >> >> >> send<br>
>>> >> >> >> >> me<br>
>>> >> >> >> >> your case file? I'll take a look.<br>
>>> >> >> >> >><br>
>>> >> >> >> >> -berk<br>
>>> >> >> >> >><br>
>>> >> >> >> >> On Thu, Feb 27, 2014 at 3:20 AM, Bernhard Righolt<br>
>>> >> >> >> >> <<a href="mailto:b.w.righolt@tudelft.nl">b.w.righolt@tudelft.nl</a>> wrote:<br>
>>> >> >> >> >> > Dear all,<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > Currently I am using a Python script to process information<br>
>>> >> >> >> >> > stored<br>
>>> >> >> >> >> > in<br>
>>> >> >> >> >> > the<br>
>>> >> >> >> >> > EnSight Gold format. My Python (2.6) scipt uses VTK (5.10.0)<br>
>>> >> >> >> >> > to<br>
>>> >> >> >> >> > process<br>
>>> >> >> >> >> > the<br>
>>> >> >> >> >> > file, where I used the vtkEnSightGoldReader for reading the<br>
>>> >> >> >> >> > data,<br>
>>> >> >> >> >> > and<br>
>>> >> >> >> >> > loop<br>
>>> >> >> >> >> > over time steps.<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > In principle this works fine for smaller datasets, however,<br>
>>> >> >> >> >> > for<br>
>>> >> >> >> >> > large<br>
>>> >> >> >> >> > datasets (i.e. 11k files of 12MB), I see the memory usage<br>
>>> >> >> >> >> > (recorded via<br>
>>> >> >> >> >> > top)<br>
>>> >> >> >> >> > increasing with time while the process is running. This<br>
>>> >> >> >> >> > filling<br>
>>> >> >> >> >> > of<br>
>>> >> >> >> >> > the<br>
>>> >> >> >> >> > memory goes slow, but in some cases problems are inevitable.<br>
>>> >> >> >> >> > Apparently,<br>
>>> >> >> >> >> > I<br>
>>> >> >> >> >> > am doing some things wrong in my script with respect to<br>
>>> >> >> >> >> > memory<br>
>>> >> >> >> >> > management.<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > I am using the following script (stripped obviously, but<br>
>>> >> >> >> >> > still<br>
>>> >> >> >> >> > showing<br>
>>> >> >> >> >> > this<br>
>>> >> >> >> >> > issue)<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > import vtk<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > reader = vtk.vtkEnSightGoldReader()<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > reader.SetCaseFileName("case.case")<br>
>>> >> >> >> >> > reader.Update()<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > # Get time values<br>
>>> >> >> >> >> > timeset=reader.GetTimeSets()<br>
>>> >> >> >> >> > time=timeset.GetItem(0)<br>
>>> >> >> >> >> > timesteps=time.GetSize()<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > #reader.ReleaseDataFlagOn()<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > for j in range(timesteps):<br>
>>> >> >> >> >> > curTime=time.GetTuple(j)[0]<br>
>>> >> >> >> >> > print curTime<br>
>>> >> >> >> >> > reader.SetTimeValue(curTime)<br>
>>> >> >> >> >> > reader.Update()<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > #reader.RemoveAllInputs()<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > I tried a few things as you can see<br>
>>> >> >> >> >> > - reader.ReleaseDataFlagOn(), without success<br>
>>> >> >> >> >> > - reader.RemoveAllInputs(), did not get it to run<br>
>>> >> >> >> >> > - running gc.collect at the end of the loop, following<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > <a href="http://vtk.1045678.n5.nabble.com/VTK-memory-management-td5715661.html" target="_blank">http://vtk.1045678.n5.nabble.com/VTK-memory-management-td5715661.html</a><br>
>>> >> >> >> >> > - Using DeepCopy as suggested here:<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > <a href="http://vtk.1045678.n5.nabble.com/Memory-leak-in-VTK-Python-module-td1234002.html" target="_blank">http://vtk.1045678.n5.nabble.com/Memory-leak-in-VTK-Python-module-td1234002.html</a><br>
>>> >> >> >> >> > However, I am already experiencing this without a call to<br>
>>> >> >> >> >> > GetOutput<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > My question is, how can I properly unload/replace the data<br>
>>> >> >> >> >> > that<br>
>>> >> >> >> >> > is<br>
>>> >> >> >> >> > stored in<br>
>>> >> >> >> >> > the memory, instead of using more memory continuously?<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > Best regards,<br>
>>> >> >> >> >> > Bernhard<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > _______________________________________________<br>
>>> >> >> >> >> > Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > Visit other Kitware open-source projects at<br>
>>> >> >> >> >> > <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > Please keep messages on-topic and check the VTK FAQ at:<br>
>>> >> >> >> >> > <a href="http://www.vtk.org/Wiki/VTK_FAQ" target="_blank">http://www.vtk.org/Wiki/VTK_FAQ</a><br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > Follow this link to subscribe/unsubscribe:<br>
>>> >> >> >> >> > <a href="http://www.vtk.org/mailman/listinfo/vtkusers" target="_blank">http://www.vtk.org/mailman/listinfo/vtkusers</a><br>
>>> >> >> >> >> ><br>
>>> >> >> >> ><br>
>>> >> >> >> ><br>
>>> >> >> >> ><br>
>>> >> >> >> > _______________________________________________<br>
>>> >> >> >> > Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
>>> >> >> >> ><br>
>>> >> >> >> > Visit other Kitware open-source projects at<br>
>>> >> >> >> > <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
>>> >> >> >> ><br>
>>> >> >> >> > Please keep messages on-topic and check the VTK FAQ at:<br>
>>> >> >> >> > <a href="http://www.vtk.org/Wiki/VTK_FAQ" target="_blank">http://www.vtk.org/Wiki/VTK_FAQ</a><br>
>>> >> >> >> ><br>
>>> >> >> >> > Follow this link to subscribe/unsubscribe:<br>
>>> >> >> >> > <a href="http://www.vtk.org/mailman/listinfo/vtkusers" target="_blank">http://www.vtk.org/mailman/listinfo/vtkusers</a><br>
>>> >> >> >> ><br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >> --<br>
>>> >> >> >> Unpaid intern in BillsBasement at noware dot com<br>
>>> >> >> ><br>
>>> >> >> ><br>
>>> >><br>
>>> >><br>
>>> >><br>
>>> >> --<br>
>>> >> Unpaid intern in BillsBasement at noware dot com<br>
>>> ><br>
>>> ><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Unpaid intern in BillsBasement at noware dot com<br>
>><br>
>><br>
</div></div></blockquote></div><br></div>