- 07 Aug 2020
Working with INDI and dealing with its limitations having
to do with streaming of large FITS images was the impetus to make this
happen. Another factor was the latency which occurred when acquiring
images and performing operations on them in one process. Having other
processes performing operations while the camera was acquiring another
frame would go a long way toward speeding up our observing.
Threading would have gone a long way towards this end, but I felt that
having each task parceled out into a different process would give
flexibility in how we would put each system together. Some designs
would want to save to disk, perform background subtractions or run
fourier transforms. By making each of these tasks a separate process,
they could be modular, and easily added and taken away without any
recompilation. By keeping each process's responsibilities small, it
aids in debugging. This also helps if a process crashes, in that only
that process is restarted, not the whole camera. Another side benefit
is the ability to run different processes on separate machines, but in
order to keep latency to a minimum, we don't generally do this.
The design may seem familiar, as the inspiration for it is also based
on the UNIX style of "chaining" commands together. That is, the output
of one command is the input of another. in this way, one can build
more complex software by simply putting together smaller pieces and
benefitting from not having to rebuild each part from scratch.
The way it works is based on several simple properties of FITS files.
FITS files are always a multiple of 2880 blocks. This is part of the
specification, and each part of the file (Header and data) is always a
multiple of this size. Sending a FITS file over a network is a case of
reading the first block, interrogating it to calculate the total
number of blocks, and receiving them properly. Each process can send a
byte-for-byte FITS file out, and since the format is standardized,
each process knows when each file ends. The beginning is the first
block which contains the "SIMPLE" keyword.
To aid in parallelization, the FITS files are not sent in between the
processes using TCP, but by multicast. When a process wishes to send a
FITS image, it sends it to the "channel", not a recipient. Other
processes may join or leave this channel as they wish. Processes may
be started or stopped at any time. When a process joins the channel,
it simply waits for the start of the next FITS file, then begins
processing it. A process assumes that the FITS files arrive one after
the other, continuously (this is the "stream").
This is the mechanism by which the "chaining" works, however, it is
different from UNIX command chaining in one detail: it has a one to
many relationship, given to it by way of the multicast design. A
process can generate a stream of FITS files and many processes can
join the channel containing the stream and process the data in
different ways, concurrently. This is how an FFT can be done at the
same time ROI statistics are being calculated. If each process is
relegated to a separate core, the speed gains are very real.
Each process is buffered as well, to ensure no data is lost. the code
uses circular buffers to keep things moving along.
As of now, most cameras used by LBTI generate this "FITS stream",
although each uses a different channel. The set of processes that can
be attached to these streams are:
indi_save: Saves each FITS file to a directory with an ascending file name.
indi_roi: Maintains a list of ROI's calculated statistics based on each one.
indi_display: Crops a region out of each received file, may subtract a
background, etc. sends out the result on a different channel. This may
be broken into two processes at some point.
indi_fftw: Calculates an FFT, send the resultant phase and magnitude
images on their own channels.
indi_xpa: Sends the FITS image to an XPA connection to ds9 for display
(not well tested yet).
Each of these processes are also well-behaved INDI drivers, publishing
and subscribing to various INDI properties as they are run.
I hope that gives a fairly high level description of how all this
machinery works, let me know if any of you have questions.