View Issue Details Jump to Notes ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0010178ParaViewFeaturepublic2010-01-22 13:242010-03-09 22:08
ReporterKen Moreland 
Assigned ToKen Moreland 
PriorityimmediateSeverityminorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target Version3.8Fixed in Version3.8 
Summary0010178: Efficient raw image reads on parallel file systems
DescriptionWhen reading raw image data from a file, the current image reader does uncoordinated reads of the data. When many processes are each trying to read a 3D subblock from a file, each process requests many small strips of data from the file. When a parallel file system gets saturated with lots of these small requests, the devices get swamped trying to satisfy all the independent requests.

In contrast, reading from a parallel file system can be very efficient when reading large blocks of data. If the reads requests from the independent process are consolidated, they can be satisfied with one or a few large reads. MPI version 2 has a set of IO commands to allow you to do just this. I propose creating a subset of the image reader that takes advantage of the MPI IO calls to make parallel reads efficient.
TagsNo tags attached.
Project
Topic Name
Type
Attached Filespatch file icon MPIImageReader.patch [^] (65,498 bytes) 2010-01-22 13:27 [Show Content]

 Relationships

  Notes
(0019273)
Ken Moreland (manager)
2010-01-22 13:34

The issue is fixed. Because this fix touches some of the base IO and parallel communication files, someone from Kitware should look at these changes and sign off before I check it in. I have assigned the bug to Utkarsh so that he knows he needs to look at the change. He should assign the bug back to me once he reviews the code.

The change is attached as a patch to this report. I have also posted the change to a branch of my github repository. The repository can be cloned from the url <git://github.com/kmorel/ParaView.git> [^] and is in the <MPIImageReader> branch. The branch is also summarized at this github.com web site.

http://github.com/kmorel/ParaView/tree/MPIImageReader [^]
(0019729)
Ken Moreland (manager)
2010-03-05 12:30

/cvsroot/VTKData/VTKData/Baseline/Parallel/ParallelIso.7proc.png,v <-- ParallelIso.7proc.png
initial revision: 1.1
/cvsroot/VTKData/VTKData/Data/headsq/quarter.nhdr,v <-- quarter.nhdr
initial revision: 1.1
/cvsroot/ParaView3/ParaView3/Applications/ParaView/ParaViewReaders.xml,v <-- Applications/ParaView/ParaViewReaders.xml
new revision: 1.6; previous revision: 1.5
/cvsroot/ParaView3/ParaView3/Servers/ServerManager/Resources/readers.xml,v <-- Servers/ServerManager/Resources/readers.xml
new revision: 1.197; previous revision: 1.196
/cvsroot/ParaView3/ParaView3/VTK/IO/vtkImageReader.cxx,v <-- VTK/IO/vtkImageReader.cxx
new revision: 1.123; previous revision: 1.122
/cvsroot/ParaView3/ParaView3/VTK/IO/vtkImageReader.h,v <-- VTK/IO/vtkImageReader.h
new revision: 1.74; previous revision: 1.73
/cvsroot/ParaView3/ParaView3/VTK/Parallel/CMakeLists.txt,v <-- VTK/Parallel/CMakeLists.txt
new revision: 1.204; previous revision: 1.203
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPI.h,v <-- VTK/Parallel/vtkMPI.h
new revision: 1.7; previous revision: 1.6
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx,v <-- VTK/Parallel/vtkMPICommunicator.cxx
new revision: 1.56; previous revision: 1.55
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.h,v <-- VTK/Parallel/vtkMPICommunicator.h
new revision: 1.43; previous revision: 1.42
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIController.cxx,v <-- VTK/Parallel/vtkMPIController.cxx
new revision: 1.29; previous revision: 1.28
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIController.h,v <-- VTK/Parallel/vtkMPIController.h
new revision: 1.26; previous revision: 1.25
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIImageReader.cxx,v <-- VTK/Parallel/vtkMPIImageReader.cxx
initial revision: 1.1
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIImageReader.h,v <-- VTK/Parallel/vtkMPIImageReader.h
initial revision: 1.1
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx,v <-- VTK/Parallel/vtkMultiProcessController.cxx
new revision: 1.39; previous revision: 1.38
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.h,v <-- VTK/Parallel/vtkMultiProcessController.h
new revision: 1.45; previous revision: 1.44
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkPNrrdReader.cxx,v <-- VTK/Parallel/vtkPNrrdReader.cxx
initial revision: 1.1
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkPNrrdReader.h,v <-- VTK/Parallel/vtkPNrrdReader.h
initial revision: 1.1
/cvsroot/ParaView3/ParaView3/VTK/Parallel/Testing/Cxx/CMakeLists.txt,v <-- VTK/Parallel/Testing/Cxx/CMakeLists.txt
new revision: 1.93; previous revision: 1.92
/cvsroot/ParaView3/ParaView3/VTK/Parallel/Testing/Cxx/ExerciseMultiProcessController.cxx,v <-- VTK/Parallel/Testing/Cxx/ExerciseMultiProcessController.cxx
new revision: 1.11; previous revision: 1.10
/cvsroot/ParaView3/ParaView3/VTK/Parallel/Testing/Cxx/ParallelIso.cxx,v <-- VTK/Parallel/Testing/Cxx/ParallelIso.cxx
new revision: 1.28; previous revision: 1.27
cvs update: warning: `vtkMPIImageReader.h' was lost
===================================================================
Checking out vtkMPIImageReader.h
RCS: /cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIImageReader.h,v
VERS: 1.1
***************
cvs update: warning: `vtkPNrrdReader.h' was lost
===================================================================
Checking out vtkPNrrdReader.h
RCS: /cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkPNrrdReader.h,v
VERS: 1.1
***************
cvs update: warning: `vtkPNrrdReader.cxx' was lost
===================================================================
Checking out vtkPNrrdReader.cxx
RCS: /cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkPNrrdReader.cxx,v
VERS: 1.1
***************
cvs update: warning: `vtkMPIImageReader.cxx' was lost
===================================================================
Checking out vtkMPIImageReader.cxx
RCS: /cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIImageReader.cxx,v
VERS: 1.1
***************
(0019792)
Alan Scott (manager)
2010-03-09 22:08

I tried this back about a week ago on the small supernova, and it worked fine. Remote servers, Trunk.

 Issue History
Date Modified Username Field Change
2010-01-22 13:24 Ken Moreland New Issue
2010-01-22 13:24 Ken Moreland Status backlog => tabled
2010-01-22 13:24 Ken Moreland Assigned To => Ken Moreland
2010-01-22 13:27 Ken Moreland File Added: MPIImageReader.patch
2010-01-22 13:34 Ken Moreland Note Added: 0019273
2010-01-22 13:34 Ken Moreland Assigned To Ken Moreland => Utkarsh Ayachit
2010-02-04 13:04 Utkarsh Ayachit Priority normal => immediate
2010-02-10 09:34 Utkarsh Ayachit Assigned To Utkarsh Ayachit => Berk Geveci
2010-03-01 09:11 Utkarsh Ayachit Assigned To Berk Geveci => Ken Moreland
2010-03-05 12:30 Ken Moreland Note Added: 0019729
2010-03-05 12:30 Ken Moreland Status tabled => @80@
2010-03-05 12:30 Ken Moreland Fixed in Version => 3.8
2010-03-05 12:30 Ken Moreland Resolution open => fixed
2010-03-09 22:08 Alan Scott Note Added: 0019792
2010-03-09 22:08 Alan Scott Status @80@ => closed
2011-06-16 13:10 Zack Galbreath Category Feature Request => Feature


Copyright © 2000 - 2018 MantisBT Team