Instrument MongoDB Interface

Please note, we are not planning to use MongoDB anymore, but have decided to use the INDI architecture model.
See Software/INDIInterface.

In July, 2016, we started looking at implementing a web-based status page, similar to: http://statserv.cfht.hawaii.edu
A lot of the required instrument data for something like this is in the TCS data dictionary (for use by FACSUM and the alarm handler). But, do we want future instruments to use the TCS data dictionary? Or should we have another, instrument data dictionary for this information - it's not required to be fast since it is not used for operations - it's not how the instruments communicate with TCS, and they do not communicate with each other.

We discussed three options:
  1. Use the TCS DD. This would require using spares and a maybe a config file/map with some code changes to avoid releasing TCS builds when instruments change data.
    Everyone agreed that using the TCS DD is not the right approach. We used it to implement instrument data for FACSUM and alarm handler, but it was not a good idea to tie instrument data to TCS builds.
  2. Reuse the CFHT Status Server. This would require us to make consistent (where possible) the access mechanisms so that TCS DD fetch and Instrument DD fetch are not so dissimilar so as to irritate users.
    The CFHT status server has a lot of nice features (command line interface, lifetime settings on data, monitor, autosave), but it uses old technology.
  3. Roll our own instrument DD. (same caveat on access mechanisms as above).
    Since it is a web status page that we need and we already have a lot of the data available, we could implement a web based front end with a nonSql backend, such as mongoDB. Where the mongoDB document describes the state of the whole telescope. The states in that document could be collected in various manners: for specific instruments or devices it could be derived from direct access to DD values, others might be a combination or a derivation of several DD entries, while others might be a combination of DD entries, even log files entries or even by probing specific machines, files, etc \x85 not too different in concept to a Nagios or Zabbix.
    This approach could be implemented immediately and wouldn\x92t place much or any burden on the instruments side or development wise.
    In addition to this the mongoDB, would effective be the Status server and be queried by anybody that needs its information (I wouldn\x92t use it for critical production tasks(?), but for everything else it should be fine)

Prototype

As a test case for using MongoDB as the Instrument Data Dictionary.
Created a MongoDB server on the LBC CMU and on rm580f-1 in Tucson, and a standalone C-program to update it. The structure looks like:
> db.stats()
{
        "db" : "STATUS_DATA",
        "collections" : 5,
        "objects" : 11,
...

> show collections
lbcb
lbcr
system.indexes

"name" : "lbcr", "housekeeping" : { "dewarpres" : 0.005, "dewartemp" : 250, "updatetime" : 1468945034 }, "units" : [ "pounds", "deg C" ] }

I used a unix timestamp in the document so that we can query on that from the other side.


On rm580f-1.tucson.lbto.org , created a Python client to read the MongoDB based on the timestamp.
Run this from the command line with a collection name ("lbcr" or "lbcb"):

python mongoclient.py lbcr


#!/lbt/lbto_runtime/bin/python
from pymongo import MongoClient
import time
   
def readDB(coll):
    client = MongoClient('lbccontrol.mountain.lbto.org', 27017)
    db = client['STATUS_DATA']
    print 'collection names:'
    print(db.collection_names(include_system_collections=False))
    print 'get '+coll+' collection'
    collection = db[coll]
   
    timestamp = int(time.time())
    timestamp = timestamp - 10  ## get updates since 10 secs ago
    print 'find updatetime gt ' + str(timestamp)
  
    ##for record in collection.find():              ## finds all records 
    ##for record in collection.find({"name" : "lbcr"}):  ## finds all records
    for record in collection.find({"housekeeping.updatetime" : {"$gt":timestamp}}):
        ##print(record)
        print record.get('housekeeping').get('dewartemp')
        print record.get('housekeeping').get('dewarpres')


## This file can be usable as a script as well as an importable module, 
## because the code that parses the command line only runs if the module 
## is executed as the main file.
if __name__ == "__main__":
    import sys
    readDB(sys.argv[1])


Working with the MongoDB from the command line:
...../mongodb-linux-i686-3.2.8/bin/mongo
> show dbs
STATUS_DATA  0.078GB
local        0.078GB

> use STATUS_DATA
switched to db STATUS_DATA

> show collections
lbcb
lbcr
system.indexes

> db.lbcb.insert({"shutterstate":"closed"})
WriteResult({ "nInserted" : 1 })

> db.lbcb.find()
{ "_id" : ObjectId("5789088eb5e63754ee97dd1a"), "shutterstate" : "closed" }
{ "_id" : ObjectId("578908f6b5e63754ee97dd1b"), "dewartemp" : 145.3 }
{ "_id" : ObjectId("57890906b5e63754ee97dd1c"), "fwhm" : 1.34 }
> db.lbcb.find().pretty()
{ "_id" : ObjectId("5789088eb5e63754ee97dd1a"), "shutterstate" : "closed" }
{ "_id" : ObjectId("578908f6b5e63754ee97dd1b"), "dewartemp" : 145.3 }
{ "_id" : ObjectId("57890906b5e63754ee97dd1c"), "fwhm" : 1.34 }

> db.lbcr.drop()
true

Specification

We need to create a specification for instruments to write to our MongoDB.

collection name string values: "lbcr", "lbcb", "luci1", "luci2", "mods1", "mods2" ...
string "housekeeping" "dewartemp"

If the collection name is the instrument name, it is easy to query and to drop the collection when the instrument goes down.

We could have separate sets of data "housekeeping" for temps/pressures and "status" for observing status.

Must include a housekeeping.timestamp in Unix time, seconds only precision.

Instrument teams will have to let us know what information should be in the status stream:
  • what temperatures and pressures should be monitored
  • what status information should constitute "ready"

This topic: Software > Software > InstrumentSoftwarePages > InstrumentMongoDBInterface
Topic revision: 12 Dec 2016, KelleeSummers
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback