Overview of AIPS++ Parallelization

Motivation for AIPS++ Parallelization

1. Give Astronomers at least an order-of-magnitude increase in computational power.
   
   • This processing power can be used to process data sets larger than currently possible.
   • The increased capability can also be used on moderate-sized data sets to explore the use of a variety of algorithms, (some of which may be computationally expensive).
2. Use a familiar interface
   • Since we are building the parallel system within the AIPS++ system, we should use the same interface.
   • Astronomers who use the parallel system for some portion of their processing should notice almost no difference in the interface (even though some of their processes are being carried out on a batch system).
3. Get as much code parallelized in a short of a time as possible
   • This priority motivates us to implement a parallel infrastructure into the existing code.
   • Also, we should make it easy for a programmer to use the parallel infrastructure.

The State of AIPS++ Parallelization

1. An Algorithm-Applicator scheme has been chosen to deal with embarrassingly parallel (e.g. spectral line) problems. The Applicator is the controller, which sets up the problem, transports dat to the Algorithm processes using the Message-Passing-Interface (MPI). MPI is a portable system that allows data and instructions to be sent to remote processors (either on the same machine or difference machines).
2. A graphical interface to the parallel system and batch processing has been implemented. A screen dump of the interface is available. The user only has to tell the application the number of processors to use. Submission to the batch queue is carried out with only one additional step.
3. Parallel system now shows good scaling up to 16 processors. More details on timing are available.
4. Lower degree of speed up beyond 16 processors may be due to I/O inefficiencies. We are working with the Pablo Research Group at the University of Illinois Computer Science Department (see Partners section). They are instrumenting the AIPS++ I/O code to accumulate statistics about I/O usage.

Short Term Goals

1. Users can begin to the use the parallel system on NCSA computers systems after the first of the year. The parallel system will include parallelization of various spectral line processing: gridding, imaging, deconvolution (various flavors of CLEAN and Maximum-Entropy-Method).
2. The parallel infrastructure should work on any parallel system with MPI.

Long Term Goals

1. Integrate batch communication into the Glish IPC.
2. Investigate use of parallel I/O from the MPI-I/O library of MPI-2.
3. Port AIPS++ to NT. Investigate how speedup improvements designed for systems such as the SGI Origin 2000 at NCSA translate to the NCSA NT Supercluster.
4. Continue to use the parallel Algorithm-Applicator classes to implement more parallel algorithms.
5. Possibly use AIPS++ for on-line processing. The JCMT has decided in principle to use AIPS++ to process the backend of the new 1 GHz Correlator. The correlator will have a data rate of about 10 MB/sec (up to 3.8 GB could be collected in a 6 minute observation). The JCMT will have a cluster of 8 to 16 workstations to carry out the on-line processing.