Archive Software

• LBT Archive Troubleshooting Guide (INAF, Fall-2017)

Archive Startup Procedure (Adapted from an email to IT from Cristina June-2015)

There are two different procedures. One is for the mountain and the other for Tucson. The mountain archive only acquires data and flushes it. The Tucson host also is the web interface.

The archive software runs on Virtual Machines within the VmWare environment. The hosts are addressed like any other host (they have DNS entries and IP addresses).

See also the google doc: LBTO-Archive-support-information

Mountain Archive

Stop the NADIR acquisition procedures with the following command:

ssh controls@mt-archive.mountain.lbto.org    (contact IT for passwd)
[controls@mt-archive ~]$ cd script/lbt
[controls@mt-archive lbt]$ ./stop_all.sh

Shutdown the host. When the host is restarted, all software should start automatically.

To verify archive software has restarted, do:

ssh controls@mt-archive.mountain.lbto.org
[controls@mt-archive ~]$ cd script/lbt
[controls@mt-archive lbt]$ ./check_state.sh

What should be seen after running the check_state.sh command:

-----------------------------------------
Device: lbt/preProcessor/new_data
State: ON
Status: Event thread running
Ignored Attribute: 4
Regular Attribute: 49
Warning Attribute: 0
Error Attribute: 0

-----------------------------------------
Device: lbt/fitsImporter/in_dir
State: ON
Status: Event thread running
Regular Attribute: 72
Warning Attribute: 0
Error Attribute: 0

-----------------------------------------
Device: lbt/metadataExporter/mt-archive
State: ON
Status: Listening on 192.168.38.32:4000

-----------------------------------------
Device: lbt/dataExporter/mt-archive
State: ON
Status: Listening on 192.168.38.32:5000

-----------------------------------------
Device: lbt/lbtrDistributor/irt
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 0
Error Attribute: 0

-----------------------------------------
Device: lbt/lbtrDistributor/lbc
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 26
Error Attribute: 0

-----------------------------------------
Device: lbt/lbtrDistributor/luci
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 24
Error Attribute: 0

-----------------------------------------
Device: lbt/lbtrDistributor/mods
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 22
Error Attribute: 0

-----------------------------------------
Device: lbt/lbtrDistributor/pis
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 0
Error Attribute: 0

-----------------------------------------
Device: lbt/lbtrDistributor/warning
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 0
Error Attribute: 0

When the State is ON for all devices, then everything is ok.

If not, please run the ./stop_all command and then do the ./start_all command again:

[controls@mt-archive lbt]$ ./stop_all.sh
[controls@mt-archive lbt]$ ./start_all.sh

If after the previously described check the answer is not ok, please verify that mysql is running:

[controls@mt-archive lbt]$ su -
Password:  (you may need IT help for this!)
[root@mt-archive ~]# service mysql status
MySQL running (1637)                                       [  OK  ]

If for any reason it is not running, do:

[root@mt-archive ~]# service mysql start

If other issues occur, please contact Cristina Knapic <knapic@oats.inaf.it> and Riccardo Smareglia <smareglia@oats.inaf.it>

Tucson Archive

The Tucson archive machines require essentially the same procedure.
Please note the back-end machine tuc-archive1.tucson.lbto.org (150.135.245.14) has four folders instead of only two.

Note that the Tucson machine NAdir software must be restarted after a mountain archive machine restart.

ssh controls@tuc-archive1.tucson.lbto.org  (IT has the passwd)
[controls@tuc-archive1 ~]$ cd script/
[controls@tuc-archive1 script]$ ls
lbt  mpia  osurc  util

Check service states first, and then stop or start all services affects lbt,mpia, and osurc folders:

[controls@tuc-archive1 script]$ cd lbt
[controls@tuc-archive1 lbt]$ ./check_state.sh
-----------------------------------------
Device: lbt/metadataExporter/tuc-archive
State: ON
Status: Listening on 150.135.245.14:4000

-----------------------------------------
Device: lbt/dataExporter/tuc-archive
State: ON
Status: Listening on 150.135.245.14:5000

-----------------------------------------
Device: lbt/metadataImporter/irt
State: ON
Status: Connected to 192.168.38.32:4000
-----------------------------------------
Device: lbt/metadataImporter/lbc
State: ON
Status: Connected to 192.168.38.32:4000
-----------------------------------------
Device: lbt/metadataImporter/luci
State: ON
Status: Connected to 192.168.38.32:4000
-----------------------------------------
Device: lbt/metadataImporter/mods
State: ON
Status: Connected to 192.168.38.32:4000
-----------------------------------------
Device: lbt/metadataImporter/pis
State: ON
Status: Connected to 192.168.38.32:4000
-----------------------------------------
Device: lbt/metadataImporter/warning
State: ON
Status: Connected to 192.168.38.32:4000

-----------------------------------------
Device: lbt/dataImporter/irt
State: ON
Status: Database loop active
Regular Attribute: 0
Failed Attribute: 0
-----------------------------------------
Device: lbt/dataImporter/lbc
State: ON
Status: Database loop active
Regular Attribute: 943
Failed Attribute: 0
-----------------------------------------
Device: lbt/dataImporter/luci
State: ON
Status: Database loop active
Regular Attribute: 247
Failed Attribute: 0
-----------------------------------------
Device: lbt/dataImporter/mods
State: ON
Status: Database loop active
Regular Attribute: 1432
Failed Attribute: 0
-----------------------------------------
Device: lbt/dataImporter/pis
State: ON
Status: Database loop active
Regular Attribute: 0
Failed Attribute: 0
-----------------------------------------
Device: lbt/dataImporter/warning
State: ON
Status: Database loop active
Regular Attribute: 2
Failed Attribute: 0

-----------------------------------------
Device: lbt/lbtrDistributor/irt
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 0
Failed Attribute: 0
-----------------------------------------
Device: lbt/lbtrDistributor/lbc
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 946
Failed Attribute: 0
-----------------------------------------
Device: lbt/lbtrDistributor/luci
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 251
Failed Attribute: 0
-----------------------------------------
Device: lbt/lbtrDistributor/mods
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 1441
Failed Attribute: 0
-----------------------------------------
Device: lbt/lbtrDistributor/pis
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 0
Failed Attribute: 0
-----------------------------------------
Device: lbt/lbtrDistributor/warning
State: ON
Status: Distributor thread waiting new files
Regular Attribute: 2
Failed Attribute: 0

Stop Services:

       [controls@tuc-archive1 lbt]$ ./stop_all.sh

After machine reboot:

[controls@tuc-archive1 lbt]$ ./start_all.sh 

See procedure for mountain if necessary to restart mysql (only if necessary..it should be up already).

---

On the front-end machine in Tucson tuc-archiveweb.tucson.lbto.org (150.135.245.25) you could stop by the usual shutdown: there are only mysql and tomcat services that need to be checked after that.

[root@tuc-archiveweb ~]# service mysql status
 SUCCESS! MySQL running (24749)

Replication should automatically restart at machine restart (otherwise, please do a service mysql restart ).

About tomcat:

[root@tuc-archiveweb ~]# ps aux | grep tomcat
root     18437  0.0  0.0 103252   820 pts/0    S+   00:18   0:00 grep tomcat
root     21117  0.1 13.2 2833308 254268 ?      Sl   Mar20 190:14
/usr/java/jdk1.6.0_16/bin/java
-Djava.util.logging.config.file=/usr/local/apache-tomcat-6.0.20/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.endorsed.dirs=/usr/local/apache-tomcat-6.0.20/endorsed -classpath
:/usr/local/apache-tomcat-6.0.20/bin/bootstrap.jar
-Dcatalina.base=/usr/local/apache-tomcat-6.0.20
-Dcatalina.home=/usr/local/apache-tomcat-6.0.20
-Djava.io.tmpdir=/usr/local/apache-tomcat-6.0.20/temp
org.apache.catalina.startup.Bootstrap start

If no tomcat is running (or Java is experiencing some faults like memory leaks... sometimes it happens) please first kill the non-responding tomcat process and then:

[root@tuc-archiveweb ~]# cd /usr/local/apache-tomcat/bin/
[root@tuc-archiveweb bin]# ./startup.sh

or to restart:

[root@tuc-archiveweb bin]# ./shutdown.sh
[root@tuc-archiveweb bin]# ./startup.sh  

Computers / Mounts

mt-archive mounts:

Filesystem                               1K-blocks        Used  Available Use% Mounted on
/dev/mapper/vg_nadir-lv_root               8063408     1537804    6116004  21% /
tmpfs                                       961212           0     961212   0% /dev/shm
/dev/sda1                                   495844       93365     376879  20% /boot
/dev/mapper/vg_nadir-lv_home               6047492     3870300    1869992  68% /home
/dev/mapper/vg_nadir-lv_tmp                3023760       91212    2778948   4% /tmp
/dev/mapper/vg_nadir-lv_usr               16126920     3059888   12247832  20% /usr
/dev/mapper/vg_nadir-lv_var               12095032     1284640   10195992  12% /var
/dev/sdb1                                516054864    21458472  468387456   5% /mnt/newdata
192.168.39.20:/volume6/lbto_repository 18730543488 12085919872 6644623616  65% /mnt/repository
192.168.39.20:/volume6/lbto_archive    18730543488 12085919872 6644623616  65% /mnt/archive/store 

tuc-archive1 mounts:

Filesystem             1K-blocks        Used  Available Use% Mounted on
/dev/mapper/vg_nadir-lv_root
                         8063408     1446364    6207444  19% /
tmpfs                     961008           0     961008   0% /dev/shm
/dev/sda1                 495844      121643     348601  26% /boot
/dev/mapper/vg_nadir-lv_home
                         6047492      844296    4895996  15% /home
/dev/mapper/vg_nadir-lv_tmp
                         3023760      129516    2740644   5% /tmp
/dev/mapper/vg_nadir-lv_usr
                        16126920     3018868   12288852  20% /usr
/dev/mapper/vg_nadir-lv_var
                        12095032     2706320    8774312  24% /var
10.130.134.20:/volume2/lbto_archive
                     19433417472 16035561472 3397737216  83% /mnt/archive
10.130.134.20:/volume2/lbto_repository
                     19433417472 16035561472 3397737216  83% /mnt/repository
10.130.134.20:/volume4/nfstest
                     11614151040  9414860032 2199172224  82% /mnt/nfstest