How to start the LUCI simulator

This describes how to start and stop the LUCI control software in simulation mode. One first starts the Telescope Control System (TCS) software in simulation mode and then runs the LUCI control software in simulation mode which talks to the simulated TCS. One can then run observing scripts which configure the instrument and send commands (presets and offsets) to the simulated TCS. Any legal LUCI script can be executed. The resulting data files are recorded in luci-sim2.tucson.lbto.org:/home/readout2/DATA/ and luci-sim.tucson.lbto.org:/home/readout1/DATA/.

1. One first must start the TCS software in simulation mode. That process is described here: TCS and LUCI Simulation.
   
   
2. Connect to the luci-sim2.tucson.lbto.org computer using noMachine NX. Log in as user "lucifer". It is usually best to tell it to start a "new" session. When the NX window opens, resize it if necessary - the larger the better.
   
   
3. If the GEIRS software is already running on luci-sim2, you can skip the GEIRS startup. It usually is not running. To start it you need to first ssh from another computer (usually one of the room 525 computers) into luci-sim2. You can't run GEIRS GUIs on luci-sim2 right now because of a font problem. Note that if you are only going to use one of the LUCI instruments (a monocular script), then you only need to start GEIRS for that instrument. If you aren't sure what the GEIRS status is, you can run the "geirs_cleanup" script to stop all GEIRS programs. See below.
   
   From some machine on the Tucson network (not luci-sim2, usually a room 525 computer), ssh into luci-sim2 as user readout1.
   
   ssh -X readout1@luci-sim2.tucson.lbto.org
   
   
4. That will open a terminal window. In that window type:
   cd GEIRS/scripts
   
   
5. Youmay run a program called 'geirs_cleanup' at this point. (This is NOT normally necessary unless GEIRS did not shut down properly the last time it was used.)
   ./geirs_cleanup
   
   
6. Start the GEIRS program in the terminal window for readout1 by typing :
   ./start_luci1_new
   
   After several seconds, that should open up a GUI that looks like this
   
   
   
   Click on the "all" (or "OK") button at the bottom of the GUI. ("All" does a bit more initialization.) The GUI will disappear and be replaced several seconds later by the 2 main GEIRS GUIs. These are the "control" GUI (small one with a yellow X drawn over it) and the "Display" GUI which shows you the last exposure taken.
   
   
7. If you need to use the simulation of LUCI-2, also, then you need to repeat steps, 3, 4, and 6 above as user readout2 and running script "start_luci2_new". You will then have a total of 4 GEIRS GUIs open.
   
   
8. In the smaller GUI of each pair, click the "read" button. This should cause it to display some fake data (just noise) in the larger, image GUI. This shows that the two programs are running.
   
   
9. In the luci-sim2 desktop which you opened in step 2 above, double click the ICON at the left edge of the screen which is labeled "LMC(simulation)". (Alternatively, open a terminal, cd to lcsp/ and execute "./bin/startSimulation.sh". There is a slight advantage to doing it that way.)
   
   After a few seconds it will open the Lucifer Managment Console (LMC). If all the programs are marked with green flags, they are already running and you can skip to step 10. If they are all red (which is normal), then click on the "start" button at the top of the LMC GUI. It will show the Lucifer animation while it starts all the services. It takes a minute or two to start all the programs. When it is done, the LMC will re-appear and all the process flags should be green. (If about half of process flags are red, click on the "stop" button and, after waiting for all processes to stop and possibly having to click "refresh" a few times, click the "exit button. Then try again.) There can be pauses of a few seconds at each stage here. It's not as fast as the real LUCI computer. Give it some time to come up.
   
   
10. When all the software is running, Click the "system" tab. Only the last entry - observer GUI - should be flagged red. Right click on the "TelescopeService" and select "open GUI" in the pop-up menu. The 2 columns of status information at the top start out saying "UNKNOWN". When it establishes communication with the telescope service, the "unknown" will disappear to be replaced with real status (plus a 1 Hz animation of dots). Wait until this happens.
    
    
11. Click on the "skip" wait for collimation radio button on the RIGHT side only. (This gets around a TCS simulation bug in the GCS subsystem.) (Do not do this at the telescope!) You can close the telescope GUI at this point.
    
    
12. Next, under the "system" tab, right click on the "scheduler GUI". In the pop-up menu which that brings up, select "start scheduler GUI". The scheduler GUI will probably show a busy "connecting" display for the readout service on luci1 or luci2. This animation will remain in this state until a readout command is actually sent. It is harmless.
    
    
13. In the scheduler GUI you must initialize the left and right LUCI instruments (or at least the one you plan to use.) Click on the "LUCI1" button on the left side of the GUI. It will open up an Instrument Manager GUI for LUCI1.)
    
    
14. Click on the word "initialize" in the button at the top of that GUI to pull down the initialization menu and select 'all-units'. The green "submit" button at the bottom of the GUI will turn yellow while it is busy initializing. You can close the GUI after the "submit" button turns green again.
    
    
15. When "submit" turn green again, repeat this process with the "LUCI2" button to initialize LUCI2.

If you haven't already, you should open some of the simulated TCS GUIs and make sure TCS is configured the way tou need it to be. See TCS and LUCI Simulation for instructions on how to configure the TCS.

The luci software is now running in simulation mode. You can copy (via sftp or scp) your scripts to luci-sim2. You (user lucifer) can create a new directory to hold them or use one of the existing directories if that is appropriate. Use the "Load" button on the GUI to navigate to your script and load it into the LUCI control program. Then click "GO" to start execution of the script. While the script is executing, watch the IIF GUI and the MCS GUI (in the TCS) to monitor what the TCS is doing.

Object Below Horizon - Change the RA

It is common that scripts expected to be run at night will be tested during the day with the simulator. You will likely find that the object is below the horizon at that time. This will make the TCS preset fail. To work around this, the script adjustTime.py is provided in the /home/lucifer/ directory. This script finds all the RA's in the script and adds a signed constant to them. It leaves the original script unchanged, but outputs the modified form to stdout. So if you have "scriptNight.xml" whose target object is below the horizon when you want to run it on the simulator, you can generate scriptDay.xml from it, by, for instance, subtracting 5.0 hours from all the RA's in the script. That would be done with this command:

./adjustTime.py scriptNight.xml -5.0 > scriptDay.xml

The delta RA parameter ("-5.0" in this example) can be anything between -24.0 and 24.0 hours. The resulting RA's will always be >= 0 and < 24.0.

Another concern with test scripts is integration times. For real observations you might need 300 second integrations, but for simulation purposes that might be deemed to make the whole test prohibitively long. So you might want to change that. (I guess we should have a python script for that too.)

Lost connection to TCS

If the TCS simulation is restarted or if just the IIF subsystem is restarted, the LUCI simulation will lose it's ICE connection to the TCS. It will not be able to communicate with the TCS any more. You will see a couple dozen errors in the log about not being able to read data dictionary variables. To make it re-establish the connection, you should open the telescope service GUI (as described in step 10 above). Then click on the "reconnect to IIF" button in the lower left corner.

If that failes to re-establish a connection (very rare), then stop both the telescope service and the Data Dictionary Connector service. Then start the Data Dictionary Connector service first and then start the Telescope Service.

How to stop the LUCI and/or GEIRS simulator.

It is not absolutely necessary to stop the observer GUI when you are done. However, the luci-sim2 machine is a rather modest virtual computer and leaving the software running indefinitely tends to use up file space with worthless logging and database information which will have to be deleted eventually anyway. So, it is recommended to stop the simulation program when you are done using it.

1. It is not absolutely necessary to stop the instrument if it is in the process of running a script, but I recommend it. Inserting a pause in the script is one way to stop it. Or generate an error on the TCS side by, for instance, turning off the instrument rotators or stopping the IIF so that an offset will fail. It will get an error the next time it talks to the IIF. You can also click on the "Other" button in the scheduler GUI and select "ABORT". That should stop the script pretty quickly. Wait for it to stop.
   
   
2. If the MCSGUI in the TCS simulation shows that the "Tracking State" is "TRACKING", click the AZ and EL "HOLD" buttons. This will make the simulated telescope stop tracking and guiding.
3. In the LUCI LMC select the "all" tab (so you can see the largest number of processes) and then click on the top-most "Stop" button in the GUI. After several seconds it should return with nearly all the annunciators (dots) in the LMC now colored red. Click the "refresh table" button at the bottom of the GUI. All the process flags should be red at this point. If a few are still green, wait a few seconds and "refresh" again or click "stop" again. It can take minute to completely stop reg RMID processes in some cases.
   If the GUI hangs at this point, you can force it to stop by typing a CTRL-C in the window from which it was started (if it was started from a window instead of the desktop icon in step 9 above).
   
   
4. I recommend stopping GEIRS, though that isn't necessary. To stop GEIRS, pull down the "File" menu in the small GEIRS control GUI (the one with the yellow X drawn across it). That pair of GUIs (control and display) will disappear. Do the same for the other pair.
   
   
5. After the LUCI code and the GEIRS code is stopped, that last thing is to close the opennx (or x2go) session. Click on "system" in the menu bar. In the pop-up GUI which that brings up, select "logout".

-- %USERSIG{TomSargent - 2014-08-19}%

Comments