23rd Annual Astronomical Data Analysis Software and Systems (ADASS) Conference

September 29 - October 3, 2013, Waikoloa Beach Marriott, Hawai'i

See the main page: http://adass2013.cfht.hawaii.edu/ for program information.
See http://www.astrosociety.org/publications/conference-series/ for previous publications.

Doug's Notes

I attended all sessions except the last session of the final day (due to a conflicting ARGOS/VATT meeting). I also attended the Advanced Code Sharing BOF discussion. My overall impression of this conference is that the conference is good, but there is significant potential not being realized. Some examples of issues I see are that papers are too short, posters and presentations are not generally available (unless you specifically ask...and then maybe not), and organization of the program was very late. I think the value of the conference is that it gathers folks for discussion. If one works hard to maximize these discussions and seek out people to talk with, some good reuse and information sharing can occur. I will support ADASS in the future, but I think we will want to think about our future submissions for suitability to the conference program.

I'll note that there were LOTS of "BIG DATA" talks....folks with Petabytes of telemetry and archiving. These systems are true fire-hoses compared to our telemetry/archiving needs. Also of note, the Virtual Observatory guys seem to be in a bit of a slump. Folks I have previously associated with championing the cause are now grumbling. The VO talk I heard was pretty defensive...as if there's internal grumbling about too many standards, and not enough products. So, VO seems to have prevalent support (including the LBT archive), but also appears to be slipping a bit in popularity within the community. I think the idea is great, but the implementation probably should be further along by now.

There were definitely some good concepts on display at the conference. My own curiosities were piqued by the following papers/posters:

P13	Gemini's Recipe System - pipeline infrastructure
P17	The Alma Pipeline - Additional tasks based on CASA (common Astronomy SW Apps)
P18	Sky Subtraction for Near-IR - SkyCorr - I have a mini-poster for this one
P46	UREKA - ssb.stsci.edu/ureka - Python and IRAF software for Astronomy
P48	NADIR - a paper about our new archiving architecture (coming soon) from Christina and Riccardo
P58	Project PANOPTES - a unique collaboration of DSLR based exoplanet hunting HW/SW. Author is Olivier Guyon
P60	Thesaurus - an astronomy thesaurus
P69	Astrophysics Source Code Library - a volunteer based source library.....subsequently found less desirable after the talk - we have a mini poster
P77	Molecfit - related to P18 - a nice thought about efficiency increase by model vs telluric star calibration - I have a mini poster for this
P81	Crates & Transform - cxc.harvard.edu/ciao/ahelp/crates.html
P90	FITS standards complaints - http://goo.dg/c7sxbm
P95	Astropy - github.com/embray/adass23-astropy, bit.ly/adass23-astropy, This is a library we should pay more attention to.

I had breakout sessions with one of the Keck software team to discuss ideas for increasing our web footprint and capturing/graphing operational metrics. I previously used this developer's template at Keck for many LGS-AO archive graphing functions, and I was looking at the possibillity of reuse at LBTO. However, he now favors the more simple and powerful approach of using python web-based development directly. So, we had a good exchange and I'll explore this space a bit.

Another breakout session I had was with CFHT. I spoke to them about queue observing and their status-server monitoring program. I obtained a copy of a briefing on their new signal-to-noise (SNR) based queue mode, and was able to gain agreement for us evaluating a copy of their status server code (still to be sent). We will investigate both concepts further in 2014.

I captured 7 pages of notes on the various talks; I will not reproduce them here. Instead, I recommend anyone who might be interested in the talks come take a look at my ADASS program which has the abstracts for all talks. Any particular talk of interest will likely have associated notes I captured which I will make available. I also captured some of the mini-posters that authors gave for the poster list above. I took pictures of all of these posters (not all good enough quality to read....but most are at least partially readable). These I will make available for those who want them.

Finally, I think there is some room for us to explore topics more. I signed up for the best of 20 years of ADASS (to come after publishing). There is also a nice "astronomy and computing" journal I retrieved a copy of. There's a nice article on software reuse from the ASCL folks in this issue.

The list of items I would like to expand upon and research / develop more:

astroplotlib.stsci.edu	astroplotlib is a nice multi-language plotting library. I'd like to look at this a bit closer
Python/Astropy	we (I) need to checkout Astropy much more for both general use, and for web-based metrics
CFHT's queue strategy	need to research their previous SPIE papers on the queue subject generally as well as other (VLT) papers
IPython notebooks	interesting technology that combines execution and documentation
ASCL	would like to know what's in their repository now
Gemini, ALMA, OPERA	pipeline approaches - OPERA may be of use for PEPSI?
Image Explorer, Source Explorer, Aladin-lite	Nice image viewing tools that are web-friendly

Kellee's Notes

I attended all of the talks and all but one of the focus demos. I did not do any of the demos in the poster room - most were related to processing science data. I attended one of the two BoF discussions - the one on Advancing Code Sharing.
Overall, I thought the conference was very worthwhile - especially just seeing what everyone else is using.

I think we will be able to take advantage of some of the stuff that folks have developed. They have interfaces already developed to FITS files and VO for pretty much everything. There are tools for transferring data. And endless tools for processing the data, in every language.

Some of the topics that came up that I will be researching more:

• lots of use of clouds
• PyVo based on astropy is a tool for Python access to VO archives
• Python notebooks
• Image Explorer / Source Explorer image analysis tools use interesting techniques we may want to copy
• Web 2.0
• HTML5, java script
• workflow implementations

Michele's Notes

I attended all sessions with the exception of Thursday morning as my plane was rescheduled and left the island too early for me to participate further. I also attended the "FITS BOF" discussion, and the tutorial on "Modern Web Development". While I did not go on the organized tour of the particular observatories on Mauna Kea, I was able to go on a private tour of UKIRT and JCMT; I have some nice pictures of the telescope domes on Mauna Kea and more detailed pics of UKIRT and JCMT. Most importantly, during the visit I had a long discussion with one of the developers regarding the software design (actually redesign) of the instrument software. The software design is general enough that it is in use at both UKIRT and JCMT. The software was designed with an end-to-end (acquisition of data to quick-look tools for real-time assessment on-the-fly) view.

I will note that I have been to seventeen of the twenty-three ADASS conferences, and this one was a bit weaker than previous conferences if only because many of the players in astronomical software from STScI (PyFITS, PyRAF, NumPY) could not attend due to NASA oversight. There was also a bit less and late organization of the tutorials and BOFs than seen in previous years; there certainly was far lower attendance than I have ever seen previously for the tutorials. I will note I have the published books from sixteen of the ADASS conferences I have attended. Also, papers for previous conferences are available on-line via the NASA Astrophysics Data System (ADS). As Doug notes, a real value of this or any conference is being able to discuss issues with folks and keep abreast of what is going on in the world of astronomy. This is something I have personally missed by not attending. I would not classify astronomical software as cutting-edge, but folks do try to be innovative. While the Virtual Observatory (and its various other guises VOA, IVOA, NVO, etc.) concept has been active for some number of years, the VOA (Virtual Astronomical Observatory) which is the US portion of the project has lost its funding and is winding down.

I had a discussion with a colleague who is now at STScI, but who formerly worked at LOFAR. The discussion was regarding HDF5 and its possible use as a replacement for FITS. My interest in the discussion was to express my dissatisfaction with the manner that the LBT Telemetry project had to specify the metadata (units, descriptions) for the data model "packetTable" in HDF5. LOFAR adopted HDF5 and has defined data models that it needs to store and examine its tremendous amount of data. I noted that I had written to the HDF5 help desk, and I got no response. Anastasia will put me in contact with someone from HDF5 who can address my question. As far as a replacement for FITS, poster P90 describes in a rambling manner why FITS cannot accommodate astronomical data into the future. The issues are well-recognized and understood. While there are certainly technical challenges in finding or creating a new format, there are also institutional challenges in terms of devoting resources (time, money, the will) to such a project. Here is where I think leveraging something which has been done by a larger community (HDF5 or ???) would be worthwhile. I will note the group I used to be in at STScI and which created PyFITS, PyRAF, and (a version) of NumPy is not happy with moving forward with HDF5 and is creating their own format. I am trying to get info on this issue. Creating their own format is costly, but STScI has done this before in order to accommodate their data. The original FITS format was too limiting, so STScI created GEIS until what we now know as multi-extension FITS came into existence.

Since our group has already seen the 2014 Project Plan, I was keeping an eye out for information specifically pertaining to our needs. In terms of Web development, it seems to me there were two basic camps represented: JavaScript and Python. I cannot say which might suit our needs better at this time, but see the "Tutorial" below for positive comments made regarding JavaScript and its family of tools.

The posters I found interesting are a subset of what Doug has already noted so I will not repeat; I will note only additions to Doug's comments.

	scipy	This was not explicitly a poster at the conference, but was mentioned on some of the Python posters. It is a general Python science library which contains some useful packages such as matplotlib and NumPy.
	demo booth	Theli: astronomical data reduction package (LBC Red and Blue are pre-configured in this system)

Please find below my notes on the Tutorial and BOF I attended.

Tutorial

Tutorial - Modern Web Development (I may have miswrote or misspelled some items)
Speaker: Amit Kapadia (Adler Planetarium)

Projects:
Space Warps, Zooniverse, Galaxy Zoo, Snapshot Serengetti

The concept of many projects on many servers is not scalable.  Now
this group proposes one server and one central API.

Technologies they use:
- GitHub: version control
- Node JS: Java script on server
- CoffeeScript: Nicer way to write Java Script.  Clean, easy to
  read, compiles into Java Script
- Twitter: promotes communication with colleagues
- AstroJS: develop and collect Java Script libraries

Make single page applications (HTML, CSS, Java Script)
They ship all static code at the beginning and no further
code needs to be sent (no latency).

Use Model-View Frameworks, improve perceived performance.
The frameworks are: SpineJS, BackboneJS, AngularJS
Java Script: has fast engines, V8, SpiderMonkey
  runs on every device, heavily supported by large
  community

Amit also gave a talk titled "The Open Web and JavaScript for
Astronomy".  The thrust here was to say the Web is now a viable
and robust platform suitable for science applications.  
JavaScript attributes:
- stable, evolving, open, and ubiquitous (which is not true of Python)
- FileAPI, typed arrays and array buffer, WebGL (tap into the graphics
  card), WebRTC (easy peer-to-peer)
- Libraries: fitsjs, webfits, ruse.js, Three.js

Mozilla/Ignite - transform the browser into the desktop.  This
is an open innovation challenge sponsored by Mozilla and NSF.

FITS BOF

I have added some background information to this discussion to give
the discussion context for folks who are not familiar with the
issue.

Flexible Image Transport System Birds of a Feather (FITS BOF)
-------------------------------------------------------------
A) FITS is to be the archival storage format for the Vatican.
Riccardo Smareglia involved in this project!

B) There is recognition the current FITS digital file format standard
used by the majority of Astronomy projects needs to change/expand/beReplaced.
The size of the data collection is the main driver behind the need
for an improved format.  This has been an on-going conversation for the
last 4 or 5 years.  FITS was standardized in 1981 and has changed (new
data models) slowly over time.  FITS, thus far, is entirely backwards
compatible.  The headers containing the metadata are entirely ASCII. Data
models are: image, binary scalar tables, binary tables with arrays in cells,
ascii tables, data cubes, and database tables.

Astronomy needs a format which can accommodate more complex data
models, and a different data model for in-memory work versus the data
model for disk storage.

Proposal:
1 - Small changes to FITS 3.0 and retain backwards compatibility
    - Increase length of keyword names (hierarchical)
    - Extend characters for keyword names
    - Increase length of string value of a keyword
2 Expand and rework FITS and break backwards compatibility
3 Get a new format entirely

Item 1 will address some annoying issues, but it does not solve
the more fundamental problems regarding data size and access.  However,
for smaller telescopes and projects, FITS 3.0 may be fine for a long
time.

==> I conclude that Item 1 will be done while Item 3 is pursued
    by other folks.

==> Future possibilities for data: HDF5 or ???  HDF5 is being used by
    LOFAR (large radio telescope).  Also, NASA earth-centric (EOS) satellite
    project has been using HDF5.  STScI (Hubble Space Telescope) is
    pursing their own format.