Log of LBC Twilight Sky Flats

To flat-field LBC images, twilight sky flats are needed; the enclosure is too compact and the surface too uneven to obtain useful dome flats. During each twilight period, usually only ~2 filters per side can be covered, so it is sometimes necessary to use archived flats. Flats taken within several months of the run should be fine (need a reference or check on this), so long as nothing was done to the filters or instrument that might alter the illumination pattern, e.g. a filter was cleaned. Cleaning of a lens that is far from the focal plane, like the corrector lens L1, should not affect the flats. Dates when the filters or lenses were cleaned are listed here.

Twilight sky flats can be downloaded from the LBT archive by anyone, since they are flagged as "Calibrations". Click the box next to "Add Calibration Files" and enter "SkyFlat" as the Object name to limit the search within any given range of dates.

Lists of LBC twilight sky flats, by month:

The tables below, the first for LBC-Blue and the second for LBC-Red, list the number of twilight flats with median count levels within a good (acceptable) range for each filter and each month for the last 12 months (if you need flat info that is older than this period, then please let us know). The median counts and median absolute deviation were measured within a 2048 x 2048 box on chip 2 [51:2098,1877:3924], roughly centered on the optical axis. The ~500 ADU bias level has not been subtracted. Flats with median counts between 10,000 and 30,000 ADU are considered "good", while those with median counts between 7500 and 40,000 ADU are deemed "acceptable". These ranges are somewhat arbitrary, however the "good" range is based on SNR considerations at the low end and, at the high end, it runs up to about half-well. The "acceptable" range loosens these restrictions in case only 1 or 2 "good" flats are found.

20230606: Images from this date are NOT sky flats - please take note. We will be removing these from the list.

202405 7(7) 21(23)     14(17) 25(29)
202404 1(2)   2(3) 0(0)    
202403 5(5)   3(3) 0(3)    
202402 3(4)   0(0)      
202312 4(7)   3(5) 11(13)    
202311     9(10) 8(8)    
202310 15(19)   8(11)   9(10) 14(17)
202309 14(15) 6(12) 15(15) 5(5) 18(28) 13(20)
202307 11(13)   12(13)      
202306 21(26) 13(19) 17(21) 13(15) 35(38) 20(31)

202405       17(19) 15(17) 6(9) 15(23) 7(7)    
202404   3(8)                
202403   5(8)                
202402   0(1)           0(0)    
202312   11(17)         2(6)      
202311   8(8)   9(10)            
202310 10(11)     6(7) 7(11) 8(8) 15(17)      
202309 9(10)     10(19) 20(27) 8(13) 18(30) 3(6)    
202307   12(15)         9(12)      
202306 8(10) 46(56) 11(12) 8(9) 34(36) 10(16) 3(4) 10(16)    

Table Notes:
  • 20230606: These are not sky flats - please take note. We will be removing these from the list.
  • 20200228: Flats were taken in cloudy conditions.
  • 20201122: Flats were taken in cloudy conditions.
  • 20201123: Flats were taken in cloudy conditions.
  • 20210319: Ignore the beginning of night flats; dome lights were on.

A note about how the tables are created and what you can do

The above tables are now automatically created by a daily task running on the machine "dms.tucson.lbto.org" (currently runs once a day). It scans the LBTO image repository and if it finds new LBC flat images, it will update the tables to reflect that. In addition, in order to keep the wiki page tidy, entries older than 12 months are removed. So normally there isn't anything that anyone needs to do. When you need information on LBC flats, you just come to this page and that's it.

How do I retrieve older flat information?

But maybe occasionally you need flat image statistics older than a year (which is cleaned from the page). In that case, can you still get them? The answer is (a partial) yes, you can still view the old table entries (with some details).

If you scroll down to the end of the wiki page, you should see that it keeps the update history as separate versions. For example, you may see the link at bottom that says: | History: r259 < r258 < r257 < r256 |. These are different revisions of this wiki page. r259 in this case is the current (top) revision, and what comes after it are previous revisions. You can click on any of the previous revisions and then you will be able to see what the wiki page looks like previously (with older table entries and possibly other different texts). Note the table entries (e.g., 8(10)) show the number of good (acceptable) flat images for that particular time/filter. Each entry is also a link, when you click it, it leads you to a separate file (we call that "flat stats file") that shows you the details.

In the past, when the table entries (those prior to 2023/02) were cleaned, their associated flat stats files were also removed. Now by default, when the daily task cleans the tables, it will NOT remove any flat stats files. So if you view table revisions before 2023/02, then the links in the tables would not work (you will see a page with missing files) since all the associated flat stats files are gone. If you click table entry links on or after 2023/02, then all of them should still work. As we are currently transitioning from the wiki system to another hosting site for LBC calibration image information, we will not be rebuilding all of the missing flat stats files from the past before 2023/02.

Note however you need a valid LBTO wiki login to view the revisions. An anonymous user cannot do this.

How can I create new table entries?

If for whatever reason you need to create (or delete) specific table entries (that the periodic task won't do), then you need to create it manually. You can do so by running the table creation program yourself. The following information details what you can do.

For the impatient

This is quick information on using the table creation program (lbcflatwiki):

  • You need to run it on "dms.tucson.lbto.org". Contact IT to obtain login information if you don't already have it.
  • Use the program like this: $ lbcflatwiki year/month[ year/month]* where lbcflatwiki is the program name, year is a four digit year number and month is a one or two digit month number. And you can repeat the list for additional time. For example, if you want to add table entries for both LBCs for May 2020 and June 2021, you can run the program like this: $ lbcflatwiki 2020/05 2021/6 (as can be seen, the "0" in the month spec is optional).
  • Note if you create/update table entries that are older than 12 months, then they will last on the wiki page for at most one day and will be cleaned by the background task when it runs next time. Read the following section on how can you prevent what you created from being deleted. If you want the wiki table's history to be permanently increased beyond 12 months (or never be purged), then you need to contact the software group for a change.
  • Also note the program works by default scanning images in the LBTO's Tucson image repository, which "officially" only retains images within a 6-month period (but may actually retain images older than a year or two). If you are looking for very old LBC images, you need to use the archive for that purpose. Read the next section on how to do that.
  • Note it is important that you refrain from manually editing the tables. Doing so may mess up the tables' format and result in the failure of the program. You can edit any wiki text outside of the tables. Please use the lbcflatwiki program to edit the tables (unless you have good knowledge of the program's internals on how it identifies and parses the table structures).

Detailed info on the table creation program

This section provides detailed information on how to use the table creation program (lbcflatwiki) as it supports many options. Note you should only read the following information if you want detailed control on the wiki table management. Otherwise the previous section provides you all that you need to know or care about.

You can use the lbcflatwiki program to edit the "official" LBC flats info wiki page (which is this page), or you can use it to make your own tables. The requirements to run it are:
  • You have a recent Python installation (Python 3.10 and above recommended) with Python library modules astropy and numpy.
  • You have access to the LBTO image repository mount (if you want to search for recent flat images), or
  • You have access to the LBTO archive disk mount (if you want to search for most older flat images).
  • In addition, if you want to be able to edit the "official" LBC flat wiki page, then you need to have access to the LBTO's FOSWiki disk partition. The machine "dms.tucson.lbto.org" provides you all these requirements. But any machine that has these conditions will be okay. If you just want to make your tables, then it is not required to have access to the FOSWiki disk partition. In the past, the program used to use the FOSWiki's standard HTTP interface to edit the "official" wiki page. However that proves to be brittle and easily broken. That's why we use direct wiki files access to edit the wiki page now.
  • The program is already installed on "dms.tucson.lbto.org" and if you want to use it there, then there is nothing to install. If you want to use it on your own machine (or any other LBTO machines), then you need to make sure the above listed conditions are met. And then you can download a copy of it from LBTO's GitHub repository: https://github.com/LBTO/swg-tools/blob/master/lbcflatwiki/lbcflatwiki

The LBC flats tables are structured by grouping each month as a single table line (with different columns representing different filters). Thus this is the resolution we have (you cannot go finer by creating daily or weekly table entries for example). Also each time an entire table line must be created/updated/deleted. You cannot operate on columns. Doing so would not have any benefits as all LBC images need to be scanned to identify whether they are flats and what filters they use. So once you request any specific filter's information, flat statistics of all other filters on the same table line will also be known.

The general format to use lbcflatwiki looks like this:

$ lbcflatwiki [options] [TU[,TU]*:cam=modes[,cam=modes]*]*

Here [options] are several optional command line options you can use. We will discuss them later. The main command line argument you provide is a list of TU separated by commas, and a list of cam=modes also separated by commas. And this entire format can be repeated by different values separated by space. TU is called a time unit specification, cam is either b (denoting LBC blue) or r (denoting LBC red), modes is one or more operating modes. The meaning of this argument is to specify what to do (by specifying the modes) on which table (by specifying the cam part), and on what table lines (by specifying the TU). There are many ways to specify these. Let's start by looking at a simple example:

$ lbcflatwiki 2021/06:b=c 2020/03,2020/09:r=u

Here 2021/06 and 2020/03,2020/09 are the TU part. b and r are cam, and c and u are called modes. The meaning of the command is to create the LBC blue table entry for the month of June 2021, and update the table entries March of 2020 and September of 2020 in the LBC red table. Notice you cannot include space around the comma in the command. If you do, the string will be broken by the Linux shell and no longer has the original intended meaning (in fact it will become a syntax error to lbcflatwiki).

We will first describe the different modes you can specify:
  • c: create the table lines specified only if there are no such lines in existence already. If the lines specified already exist, then nothing is done. Note a non-exist line is a line not currently in the table. It is different from an empty line, which is a line that exists in the table but has no flat information (i.e., displays nothing other than blank spaces).
  • u: update the specified table lines only if such lines already exist. If the specified lines do not yet exist in the table, then nothing is done. If the specified table lines already have some flat info, and the program now finds more flat info, then the line will be updated to reflect that. Otherwise the table line stays the same. Note an update will never cause a line to disappear from the table, it is however possible for an update to change a line that has contents to an empty line (e.g., flat images are somehow removed from the image repository since last update, maybe to correct some problems).
  • p: this mode "pins" the specified table lines so that they cannot be deleted (see the 'd' and 'e' modes). To visually distinguish a pinned line, an extra "*" symbol is added to the line's date heading. For example, a normal table line might start with "202005", while a pinned line will start with "202005*". Note p has no effects on an already pinned line, or a line that does not exist.
  • x: this is the opposite of the p mode. It "unpins" the specified line (and so afterwards it will remove the extra "*" symbol in the line's date heading). x has no effects on an unpinned line, or a line that does not exist.
  • d: delete the specified table lines if they are unpinned. If a line is pinned, or does not exist, then nothing is done. Note: this mode is used by the background task to clean the table lines. The background task will currently remove any table lines older than 12 months and unpinned. So if you want a specific table line from being purged by the background task, just pin it.
  • e: delete the specified lines only if the line is empty and unpinned. Recall an empty line is a table line with blank information in all of its columns. Nothing is done if the line is not empty, or pinned, or does not exist.

All these modes can be sequenced together, such as cue. When doing so, the final semantics is equivalent to executing each mode in order. The previous sentence may seem overly pedantic. This is because the program does not necessarily execute the given modes in order blindly. It uses a simplification algorithm to produce a semantically equivalent list to avoid unnecessary work from crazy input (e.g., b=uuuuucucucuuuuuuuuxpxpxpxpxxdppeeecuecu is semantically equivalent to b=c regardless of what the targeting line is).

You do not have to sequence modes together. You can also specify them separately. For example: 2021/05:b=c,b=u,b=e. This is equivalent to 2021/05:b=cue, which is much shorter. So you normally want to use sequence to save some typing. Note if you use the separate input format, you can also mix cameras, e.g., 2021/05:b=c,r=c,b=u,b=e,r=e.

By putting different modes together, you can achieve different meanings. Here are a few commonly used examples:
  • cu creates lines that do not exist, or if they already exist in the table, update them. This ensures that the specified table lines will be present in the table after the operation.
  • ce creates lines do not exist, and then checks if they are empty, and removes them if so. Effectively this operation says only put the specified lines in the table if they are not empty. And if they are empty (i.e., no flat information can be found), then don't bother putting them in the table.
  • cue achieves the combined effects of the previous two, namely create/update the specified lines and only retain them if they are not empty.
  • xd does unpin followed by an unconditional delete. Combined together, these will absolutely delete any table lines you have specified.
  • cup either creates or updates lines and then pins them regardless of the update result.
  • cuxep looks complicated. What it means is to create or update certain lines, and keep them there if non-empty, and then pin them so they cannot be deleted normally. The x mode in between is to make sure that if the line is empty, it can be deleted (because it could be pinned).

Also there are various shortcuts built in to save you some typing:
  • The entire modes can be omitted, and if so, it is taken to be the cu sequence. E.g., 2021/05:b,r is equivalent to 2021/05:b=cu,r=cu.
  • The camera part can be omitted, and if so, it is interpreted as the mode spec applying to both cameras. For example, 2021/05:cue is equivalent to 2021/05:b=cue,r=cue.
  • Both the camera and the mode parts can be omitted entirely. In that case, the default is taken to be modes cu applying to both cameras. You can include or omit the ":" separator after the TU spec. For example, both 2021/05 and 2021/05: are valid inputs, and they are equivalent to 2021/05:b=cu,r=cu.
  • As already discussed before, you can either use sequenced mode, use separate camera modes, or even mix them together. For example, all of these are valid and equivalent inputs: 2021/05:b=c,r=u,b=u,b=e,r=e and 2021/05:b=c,b=u,r=ue,b=e and 2021/05:r=ue,b=cue. However the order of modes is important and does make a difference (notice in all previous examples, the orders of modes specified on each camera are the same).

Next we discuss the specification of the time unit (TU). A TU can be any of the followings:
  • A standard time unit (STU), which is simply of the format year/month, where year is (always) a four-digit year and month is a one or two digit month. For example, 2021/05, 2021/5, and 2021/11 are all valid. Note if you omit the year part and only specify the month part, it is taken to be the specified month of the current year, and it is still a valid STU. For example: 9 or /9 are both valid STU, meaning 2024/09 if the current year is 2024 (and as can been seen, you can omit the / symbol, or leave it in). If you omit the month part (e.g., 2009, or 2009/), then it is no longer a valid STU as it denotes the all the months of the specified year. It becomes a time range (TR, see below). But it is nonetheless a valid TU.
  • Three special STU are given:
    • now means the current year and month (depending on when the program is invoked).
    • min represents the oldest time entry in the specified table.
    • max represents the most recent time entry in the specified table.
    • For example, if the specified table has lines: 201510, 201511, 201512, 201603, 201709, and 201802. Then min is 201510 and max is 201802. If however the specified table is empty (i.e., does not have any lines), then both min and max have no meaning, and any expression built from min or max will be ignored. On the other hand, now is always well defined.
  • STU can also have offsets to produce another STU. Offsets come in three flavors and can be arbitrarily chained with either plus or minus operators:
    • You can use the form 2y5m to mean an offset of two years and five months.
    • Use year alone, e.g., 3y represents an offset of three years.
    • Use month offsets alone, e.g., 20m represents a 20 month offset.
    • When using offsets, you need to surround the entire expression with curly braces. For example, {2018/2-2y5m} represents a new STU that is based two years and five months backward from 2018/2 (which is 2015/9).
    • You can chain offsets together with plus or minus. For example, {2018/2-1y3m+40m-10y} represents a STU first going backwards one year and three months from 2018/2, and then going forward by 40 months, and finally going backward by ten years (which is 2010/03).
    • Since now, min, and max are also STU, they can include offsets as well. For example, use {now-1m} to represent last month (which is not fixed and changes from when you run the program); {min-6m} means going back six months further than the oldest table entries; and so on...
  • A time range (TR) specification specifies a list of time. It can be formulated by multiple ways:
    • By joining two STU using a hyphen symbol "-". For example, 2018/3-2019/5 represents all months from 2018/3 to 2019/5, with both ends included. Note if the beginning STU is later than the second STU, then it denotes an empty range. For example, now-{now-1m} is always empty. Also note when using this form, you should NOT include any space around "-", otherwise it becomes broken strings which result in syntax errors.
    • You can use ranges or lists (or both) in the year and month part of a canonical STU. And when doing so, remember to put the corresponding part inside a pair of square brakets [].
      • [2015-2019]/3 represents all Marchs in years 2015, 2016, 2017, 2018, and 2019.
      • [2015-2017,2011]/3 represents all Marchs in 2015, 2016, 2017, and 2011.
      • [2011,2020,2018]/3 represents all Marchs in 2011, 2018, 2020.
      • 2020/[3-6] represents March to June (both included) in 2020.
      • 2020/[3-5,10-12,8,1] represents the highlighted months in 2020.
      • [2017-2019]/[3-5] is equivalent to 2017/03, 2017/04, 2017/05, 2018/03, 2018/04, 2018/05, 2019/03, 2019/04, and 2019/05. It is effectively a Cartesian product of the two sets.
      • You can mix them all together, for example, [2015,2017-2019]/[1-3,9,12], etc.
    • Two special TR are given:
      • * represents all existing table lines (a time range). For example, you can use this spec to delete all empty and unpinned lines from both tables: $ lbcflatwiki *:e. Note if the specified table is empty (i.e., does not contain any lines), then * is an empty time range (i.e., specs built from it will not result in any actual operations).
      • monthly has time dependent meanings. If you run the program on the first day of each month, then monthly is equivalent to {now-1m}-now (i.e., this and last months). Otherwise (running the program on days other than the first day of each month), then monthly simply equals now. The utility of the special time range is to make sure you don't miss any flat images no matter when you run the program. Since the program is not continuously run, when you run it on the first day of a month and ask it update the current month, it may be possible to miss potential flat images taken on the last day of the previous month, results in incomplete table information. When you run the program on any other day in a month, then typically images from previous months will not change (though not guaranteed as people sometimes add old images to archive as well, or update existing images). Then it will be a waste of time to check the previous months if you just want to make sure the table gets all the information.
    • Note a TR unit cannot include any offsets. Only an STU can have offsets.
  • All valid TU can be chained together in a comma separated list. For example, 2017/5,2015,{now-3m}-now,2016/1-2017/2,[2018,2019]/3 is a complex but valid TU list and can be used as the basis for an action spec. For example "2017/5,2015,{now-3m}-now,2016/1-2017/2,[2018,2019]/3:b=ce" is a valid input spec.
  • The TU part in the entire TU[,TU]*:cam=modes[,cam=modes]* spec can be omitted, and if so, denotes the current year and month. For example, $ lbcflatwiki b=u,r=d and $ lbcflatwiki :b=u,r=d are both valid and equivalent (note one with the ":" separator and one without), meaning update the line from current year and month in the LBC blue table, and delete the line from current year and month from the LBC red table (if unpinned).
  • Finally if nothing is given (as in simply running $ lbcflatwiki), then it is equivalent to applying the modes cu to both camera tables on the lines from current year and month.

By now hopefully you have understood how the table lines and associated actions can be specified for the program lbcflatwiki. As a final example, we will look at how the daily task running on "dms.tucson.lbto.org" is setup. It is specified as:

$ lbcflatwiki monthly min-{now-12m}:d

By now you should understand the intention of this spec is to keep the table updated by scanning possible LBC images every day and purges any unpinned table entries that are more than 12 months old. It takes advantage of several shortcuts we have discussed. In the full form, it is equivalent to: $ lbcflatwiki now:b=cu,r=cu {now-1m}:b=cu,r=cu min-{now-12m}:b=d,r=d if running on the first day of each month, otherwise it is equivalent to: $ lbcflatwiki now:b=cu,r=cu min-{now-12m}:b=d,r=d

Now we briefly mention some of the useful optional arguments:
  • If you have other machines that have access to the LBTO's FOSWiki disk partition, then use the option --wdir to specify the location of the wiki page source code, and use the option --fdir to specify the location of the wiki page file attachment directory.
  • You can use the --repo option to specify the base directory of the LBTO image repository mount. Suppose the full path of an image in the repository is /lbt/data/repository/20240213/lbcr.20240213.015600.fits, then the base is /lbt/data/repository.
  • You can also use the --repo option to point to the base of the LBTO archive mount. If you do so, you must also include the option --archive to indicate that what pointed to by --repo is in fact the archive. Suppose the full path of an image in the archive is /lbt/data/archive/store/2024/02/13/lbc/0/lbcr.20240213.015600.fits.gz, then the base directory of the archive is /lbt/data/archive/store.
  • If you run the program interactively, you can use the --verbose option to enable some progress and information report on the terminal screen. Sometimes depending on the number of images you are scanning, and the status of the image server, it can take a very long time to finish (e.g., hours). In that case, you can use double verbose option -vv to show additional information including the progress of image search and scanning (so you know it is not stuck).
  • Another option you can use is --rmfile, which if given will delete any LBC flat info dat files associated with the table entries purged from the wiki page. This option is NOT used by the default daily task so all the previously created flat image statistics files are still valid even when their associated table entries are removed from the wiki page. As discussed before, when you go back to flat wiki page's old history, you will still be able to view these files.
  • Another useful option to use is the --gentable option. With this option, you are now no longer editing the "official" LBC flat wiki page. Instead you supply the name and a path of a new file you'd like to create. For example, $ lbcflatwiki --verbose --gentable /home/telescope/lbc_flats.txt will create the file lbc_flats.txt inside the directory /home/telescope (assuming you have the correct file permission), and then will create a similar table as on the "official" wiki page and populate it with the current year/month's LBC flat information. If flat images are found, it will also create the corresponding stats files in the same directory. How you use this information is up to you. For example, you can create your own special wiki page and paste the contents inside lbc_flats.txt as part of your page, and perhaps you can also upload all the stats files to your own wiki page as well.
  • You can also use the --dryrun option to see what the program will do without actually committing any of the changes (it is a good way to do practice and exercise of the program when you are not familiar with it enough). When you are satisfied, you can then remove the --dryrun option to commit the changes.
  • You can also use the --info option to dump the current command line configuration. This option is different from --dryrun in that it does not run the program. It just shows you what the program thinks what's given to it. It will be helpful for example, to check if it reads your TU specs as you have intended.

Ancient information (still retained):

Before the lbcflatwiki program, the tables were created manually. The steps outlined below are for archival purposes, they are (hopefully) not needed now.

The following procedure was used to create the table manually above:
  1. In ~lbto/kuhn/LBCTwilightFlats lbcsky [B|R] [filter] -y YYYYMM
    1. this will create a series of *.dat files, one for each filter and camera combination.
  2. copy files over to my laptop (/Users/olga/LBC/SkyFlats/LBCTwilightFlats)
  3. run grepsed.scr to grep all of the *dat files in that directory and output to a file (like "newsum.txt")
  4. run createtab.py on newsum.txt to create the table that can be copied and pasted into the Twiki above.
  5. attach all non-null *.dat files to this Twiki page.

-- %USERSIG{OlgaKuhn - 2018-05-31}%


Topic revision: r273 - 11 May 2024, AdminUser
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback