---

10.1 Sort FITS files by night ¶

FITS images usually contain (several) keywords for preserving important dates. In particular, for lower-level data, this is usually the observation date and time (for example, stored in the DATE-OBS keyword value). When analyzing observed datasets, many calibration steps (like the dark, bias or flat-field), are commonly calculated on a per-observing-night basis.

However, the FITS standard’s date format (YYYY-MM-DDThh:mm:ss.ddd) is based on the western (Gregorian) calendar. Dates that are stored in this format are complicated for automatic processing: a night starts in the final hours of one calendar day, and extends to the early hours of the next calendar day. As a result, to identify datasets from one night, we commonly need to search for two dates. However calendar peculiarities can make this identification very difficult. For example, when an observation is done on the night separating two months (like the night starting on March 31st and going into April 1st), or two years (like the night starting on December 31st 2018 and going into January 1st, 2019). To account for such situations, it is necessary to keep track of how many days are in a month, and leap years, etc.

Gnuastro’s astscript-sort-by-night script is created to help in such important scenarios. It uses Fits to convert the FITS date format into the Unix epoch time (number of seconds since 00:00:00 of January 1st, 1970), using the --datetosec option. The Unix epoch time is a single number (integer, if not given in sub-second precision), enabling easy comparison and sorting of dates after January 1st, 1970.

You can use this script as a basis for making a much more highly customized sorting script. Here are some examples

• If you need to copy the files, but only need a single extension (not the whole file), you can add a step just before the making of the symbolic links, or copies, and change it to only copy a certain extension of the FITS file using the Fits program’s --copy option, see HDU information and manipulation.
• If you need to classify the files with finer detail (for example, the purpose of the dataset), you can add a step just before the making of the symbolic links, or copies, to specify a file-name prefix based on other certain keyword values in the files. For example, when the FITS files have a keyword to specify if the dataset is a science, bias, or flat-field image. You can read it and to add a sci-, bias-, or flat- to the created file (after the --prefix) automatically.

  For example, let’s assume the observing mode is stored in the hypothetical MODE keyword, which can have three values of BIAS-IMAGE, SCIENCE-IMAGE and FLAT-EXP. With the step below, you can generate a mode-prefix, and add it to the generated link/copy names (just correct the filename and extension of the first line to the script’s variables):

  modepref=$(astfits infile.fits -h1 \
                     | sed -e"s/'/ /g" \
                     | awk '$1=="MODE"{ \
                         if($3=="BIAS-IMAGE") print "bias-"; \
                         else if($3=="SCIENCE-IMAGE") print "sci-"; \
                         else if($3==FLAT-EXP) print "flat-"; \
                         else print $3, "NOT recognized"; exit 1}')
  

  Here is a description of it. We first use astfits to print all the keywords in extension 1 of infile.fits. In the FITS standard, string values (that we are assuming here) are placed in single quotes (') which are annoying in this context/use-case. Therefore, we pipe the output of astfits into sed to remove all such quotes (substituting them with a blank space). The result is then piped to AWK for giving us the final mode-prefix: with $1=="MODE", we ask AWK to only consider the line where the first column is MODE. There is an equal sign between the key name and value, so the value is the third column ($3 in AWK). We thus use a simple if-else structure to look into this value and print our custom prefix based on it. The output of AWK is then stored in the modepref shell variable which you can add to the link/copy name.

  With the solution above, the increment of the file counter for each night will be independent of the mode. If you want the counter to be mode-dependent, you can add a different counter for each mode and use that counter instead of the generic counter for each night (based on the value of modepref). But we will leave the implementation of this step to you as an exercise.

• Invoking astscript-sort-by-night