) is a fusion database, integrating
the information about extra-galactic sources from many large sky
surveys into a single catalog. It covers the full spectrum, from
Gamma rays to radio frequencies and is updated when new data
arrives. A TAP query to ‘ned’ is submitted to
‘https://ned.ipac.caltech.edu/tap/sync’.
• ‘objdir --> NEDTAP.objdir’: default TAP-based dataset in NED.
• ‘extinction’: A command-line interface to the NED Extinction
Calculator
(https://ned.ipac.caltech.edu/extinction_calculator). It only
takes a central coordinate and returns a VOTable of the
calculated extinction in many commonly used filters at that
point. As a result, options like ‘--width’ or ‘--radius’ are
not supported. However, Gnuastro does not yet support the
VOTable format. Therefore, if you specify an ‘--output’ file,
it should have an ‘.xml’ suffix and the downloaded file will
not be checked.
Until VOTable support is added to Gnuastro, you can use GREP,
AWK and SED to convert the VOTable data into a FITS table with
a command like below (assuming the queried VOTable is called
‘ned-extinction.xml’):
grep '^| ' ned-extinction.xml \
| sed -e's| |
| ||' \
-e's| |
||' \
-e's||@|g' \
| awk 'BEGIN{FS="@"; \
print "# Column 1: FILTER [name,str15] Filter name"; \
print "# Column 2: CENTRAL [um,f32] Central Wavelength"; \
print "# Column 3: EXTINCTION [mag,f32] Galactic Ext."; \
print "# Column 4: ADS_REF [ref,str50] ADS reference"} \
{printf "%-15s %g %g %s\n", $1, $2, $3, $4}' \
| asttable -oned-extinction.fits
Once the table is in FITS, you can easily get the extinction
for a certain filter (for example, the ‘SDSS r’ filter) like
the command below:
asttable ned-extinction.fits --equal=FILTER,"SDSS r" \
-cEXTINCTION
‘vizier’
Vizier () is arguably the largest
catalog database in astronomy: containing more than 20500 catalogs
as of mid January 2021. Almost all published catalogs in major
projects, and even the tables in many papers are archived and
accessible here. For example, VizieR also has a full copy of the
Gaia database mentioned below, with some additional standardized
columns (like RA and Dec in J2000).
The current implementation of ‘--limitinfo’ only looks into the
description of the datasets, but since VizieR is so large, there is
still a lot of room for improvement. Until then, if ‘--limitinfo’
is not sufficient, you can use VizieR's own web-based search for
your desired dataset:
Because VizieR curates such a diverse set of data from tens of
thousands of projects and aims for interoperability between them,
the column names in VizieR may not be identical to the column names
in the surveys' own databases (Gaia in the example above). A query
to ‘vizier’ is submitted to
‘http://tapvizier.u-strasbg.fr/TAPVizieR/tap/sync’.
Here is the list of short names for popular datasets within VizieR
(sorted alphabetically by their short name). Please feel free to
suggest other major catalogs (covering a wide area or commonly used
in your field).. For details on each dataset with necessary
citations, and links to web pages, look into their details with
their ViziR names in .
• ‘2mass --> II/246/out’ (2MASS All-Sky Catalog)
• ‘akarifis --> II/298/fis’ (AKARI/FIS All-Sky Survey)
• ‘allwise --> II/328/allwise’ (AllWISE Data Release)
• ‘apass9 --> II/336/apass9’ (AAVSO Photometric All Sky Survey,
DR9)
• ‘catwise --> II/365/catwise’ (CatWISE 2020 catalog)
• ‘des1 --> II/357/des_dr1’ (Dark Energy Survey data release 1)
• ‘gaiadr3 --> I/355/gaiadr3’ (GAIA Data Release 3)
• ‘gaiaedr3 --> I/350/gaiadr3’ (GAIA Early Data Release 3)
• ‘gaiadr2 --> I/345/gaia2’ (GAIA Data Release 2)
• ‘galex5 --> II/312/ais’ (All-sky Survey of GALEX DR5)
• ‘nomad --> I/297/out’ (Naval Observatory Merged Astrometric
Dataset)
• ‘panstarrs1 --> II/349/ps1’ (Pan-STARRS Data Release 1).
• ‘ppmxl --> I/317/sample’ (Positions and proper motions on the
ICRS)
• ‘sdss12 --> V/147/sdss12’ (SDSS Photometric Catalogue, Release
12)
• ‘usnob1 --> I/284/out’ (Whole-Sky USNO-B1.0 Catalog)
• ‘ucac5 --> I/340/ucac5’ (5th U.S. Naval Obs. CCD Astrograph
Catalog)
• ‘unwise --> II/363/unwise’ (Band-merged unWISE Catalog)
• ‘wise --> II/311/wise’ (WISE All-Sky data Release)
5.4.2 Invoking Query
--------------------
Query provides a high-level interface to downloading subsets of data
from databases. The executable name is ‘astquery’ with the following
general template
$ astquery DATABASE-NAME [OPTION...] ...
One line examples:
## Information about all datasets in ESA's GAIA database:
$ astquery gaia --information
## Only show catalogs in VizieR that have 'MUSE' in their
## description. The '-i' is short for '--information'.
$ astquery vizier -i --limitinfo=MUSE
## List of columns in 'J/A+A/608/A2/udf10' (one of the above).
$ astquery vizier --dataset=J/A+A/608/A2/udf10 -i
## ID, RA and Dec of all Gaia sources within an image.
$ astquery gaia --dataset=dr3 --overlapwith=image.fits \
-csource_id,ra,dec
## RA, Dec and Spectroscopic redshifts of objects in SDSS DR12
## spectroscopic redshift that overlap with 'image.fits'.
$ astquery vizier --dataset=sdss12 --overlapwith=image.fits \
-cRA_ICRS,DE_ICRS,zsp --range=zsp,1e-10,inf
## All columns of all entries in the Gaia DR3 catalog (hosted at
## VizieR) within 1 arc-minute of the given coordinate.
$ astquery vizier --dataset=gaiadr3 --output=my-gaia.fits \
--center=113.8729761,31.9027152 --radius=1/60 \
## Similar to above, but only ID, RA and Dec columns for objects with
## magnitude range 10 to 15. In VizieR, this column is called 'Gmag'.
## Also, using sexagesimal coordinates instead of degrees for center.
$ astquery vizier --dataset=gaiadr3 --output=my-gaia.fits \
--center=07h35m29.51,31d54m9.77 --radius=1/60 \
--range=Gmag,10:15 -cDR3Name,RAJ2000,DEJ2000
Query takes a single argument which is the name of the database. For
the full list of available databases and accessing them, see *note
Available databases::. There are two methods to query the databases,
each is more fully discussed in its option's description below.
• *Low-level:* With ‘--query’ you can directly give a raw query
statement that is recognized by the database. This is very low
level and will require a good knowledge of the database's query
language, but of course, it is much more powerful. If this option
is given, the raw string is directly passed to the server and all
other constraints/options (for Query's high-level interface) are
ignored.
• *High-level:* With the high-level options (like ‘--column’,
‘--center’, ‘--radius’, ‘--range’ and other constraining options
below), the low-level query will be constructed automatically for
the particular database. This method is only limited to the
generic capabilities that Query provides for all servers. So
‘--query’ is more powerful, however, in this mode, you do not need
any knowledge of the database's query language. You can see the
internally generated query on the terminal (if ‘--quiet’ is not
used) or in the 0-th extension of the output (if it is a FITS
file). This full command contains the internally generated query.
The name of the downloaded output file can be set with ‘--output’.
The requested output format can have any of the *note Recognized table
formats:: (currently ‘.txt’ or ‘.fits’). Like all Gnuastro programs, if
the output is a FITS file, the zero-th/first HDU of the output will
contain all the command-line options given to Query as well as the full
command used to access the server. When ‘--output’ is not set, the
output name will be in the format of ‘NAME-STRING.fits’, where ‘NAME’ is
the name of the database and ‘STRING’ is a randomly selected 6-character
set of numbers and alphabetic characters. With this feature, a second
run of ‘astquery’ that is not called with ‘--output’ will not over-write
an already downloaded one. Generally, when calling Query more than
once, it is recommended to set an output name for each call based on
your project's context.
The outputs of Query will have a common output format, irrespective
of the used database. To achieve this, Query will ask the databases to
provide a FITS table output (for larger tables, FITS can consume much
less download volume). After downloading is complete, the raw
downloaded file will be read into memory once by Query, and written into
the file given to ‘--output’. The raw downloaded file will be deleted
by default, but can be preserved with the ‘--keeprawdownload’ option.
This strategy avoids unnecessary surprises depending on database. For
example, some databases can download a compressed FITS table, even
though we ask for FITS. But with the strategy above, the final output
will be an uncompressed FITS file. The metadata that is added by Query
(including the full download command) is also very useful for future
usage of the downloaded data. Unfortunately many databases do not write
the input queries into their generated tables.
‘--dry-run’
Only print the final download command to contact the server, do not
actually run it. This option is good when you want to check the
finally constructed query or download options given to the download
program. You may also want to use the constructed command as a
base to do further customizations on it and run it yourself.
‘-k’
‘--keeprawdownload’
Do not delete the raw downloaded file from the database. The name
of the raw download will have a ‘OUTPUT-raw-download.fits’ format.
Where ‘OUTPUT’ is either the base-name of the final output file
(without a suffix).
‘-i’
‘--information’
Print the information of all datasets (tables) within a database or
all columns within a database. When ‘--dataset’ is specified, the
latter mode (all column information) is downloaded and printed and
when it is not defined, all dataset information (within the
database) is printed.
Some databases (like VizieR) contain tens of thousands of datasets,
so you can limit the downloaded and printed information for
available databases with the ‘--limitinfo’ option (described
below). Dataset descriptions are often large and contain a lot of
text (unlike column descriptions). Therefore when printing the
information of all datasets within a database, the information
(e.g., database name) will be printed on separate lines before the
description. However, when printing column information, the output
has the same format as a similar option in Table (see *note
Invoking asttable::).
Important note to consider: the printed order of the datasets or
columns is just for displaying in the printed output. You cannot
ask for datasets or columns based on the printed order, you need to
use dataset or column names.
‘-L STR’
‘--limitinfo=STR’
Limit the information that is downloaded and displayed (with
‘--information’) to those that have the string given to this option
in their description. Note that _this is case-sensitive_. This
option is only relevant when ‘--information’ is also called.
Databases may have thousands (or tens of thousands) of datasets.
Therefore just the metadata (information) to show with
‘--information’ can be tens of megabytes (for example, the full
VizieR metadata file is about 23Mb as of January 2021). Once
downloaded, it can also be hard to parse manually. With
‘--limitinfo’, only the metadata of datasets that contain this
string _in their description_ will be downloaded and displayed,
greatly improving the speed of finding your desired dataset.
‘-Q "STR"’
‘--query="STR"’
Directly specify the query to be passed onto the database. The
queries will generally contain space and other meta-characters, so
we recommend placing the query within quotations.
‘-s STR’
‘--dataset=STR’
The dataset to query within the database (not compatible with
‘--query’). This option is mandatory when ‘--query’ or
‘--information’ are not provided. You can see the list of
available datasets within a database using ‘--information’
(possibly supplemented by ‘--limitinfo’). The output of
‘--information’ will contain the recognized name of the datasets
within that database. You can pass the recognized name directly to
this option. For more on finding and using your desired database,
see *note Available databases::.
‘-c STR’
‘--column=STR[,STR[,...]]’
The column name(s) to retrieve from the dataset in the given order
(not compatible with ‘--query’). If not given, all the dataset's
columns for the selected rows will be queried (which can be
large!). This option can take multiple values in one instance (for
example, ‘--column=ra,dec,mag’), or in multiple instances (for
example, ‘-cra -cdec -cmag’), or mixed (for example, ‘-cra,dec
-cmag’).
In case, you do not know the full list of the dataset's column
names a-priori, and you do not want to download all the columns
(which can greatly decrease your download speed), you can use the
‘--information’ option combined with the ‘--dataset’ option, see
*note Available databases::.
‘-H INT’
‘--head=INT’
Only ask for the first ‘INT’ rows of the finally selected columns,
not all the rows. This can be good when your search can result a
large dataset, but before downloading the full volume, you want to
see the top rows and get a feeling of what the whole dataset looks
like.
‘-v FITS’
‘--overlapwith=FITS’
File name of FITS file containing an image (in the HDU given by
‘--hdu’) to use for identifying the region to query in the give
database and dataset. Based on the image's WCS and pixel size, the
sky coverage of the image is estimated and values to the
‘--center’, ‘--width’ will be calculated internally. Hence this
option cannot be used with ‘--center’, ‘--width’ or ‘--radius’.
Also, since it internally generates the query, it cannot be used
with ‘--query’.
Note that if the image has WCS distortions and the reference point
for the WCS is not within the image, the WCS will not be
well-defined. Therefore the resulting catalog may not overlap, or
correspond to a larger/small area in the sky.
‘-C FLT,FLT’
‘--center=FLT,FLT’
The spatial center position (mostly RA and Dec) to use for the
automatically generated query (not compatible with ‘--query’). The
comma-separated values can either be in degrees (a single number),
or sexagesimal (‘_h_m_’ for RA, ‘_d_m_’ for Dec, or ‘_:_:_’ for
both).
The given values will be compared to two columns in the database to
find/return rows within a certain region around this center
position will be requested and downloaded. Pre-defined RA and Dec
column names are defined in Query for every database, however you
can use ‘--ccol’ to select other columns to use instead. The
region can either be a circle and the point (configured with
‘--radius’) or a box/rectangle around the point (configured with
‘--width’).
‘--ccol=STR,STR’
The name of the coordinate-columns in the dataset to compare with
the values given to ‘--center’. Query will use its internal
defaults for each dataset (for example, ‘RAJ2000’ and ‘DEJ2000’ for
VizieR data). But each dataset is treated separately and it is not
guaranteed that these columns exist in all datasets. Also, more
than one coordinate system/epoch may be present in a dataset and
you can use this option to construct your spatial constraint based
on the others coordinate systems/epochs.
‘-r FLT’
‘--radius=FLT’
The radius about the requested center to use for the automatically
generated query (not compatible with ‘--query’). The radius is in
units of degrees, but you can use simple division with this option
directly on the command-line. For example, if you want a radius of
20 arc-minutes or 20 arc-seconds, you can use ‘--radius=20/60’ or
‘--radius=20/3600’ respectively (which is much more human-friendly
than ‘0.3333’ or ‘0.005556’).
‘-w FLT[,FLT]’
‘--width=FLT[,FLT]’
The square (or rectangle) side length (width) about the requested
center to use for the automatically generated query (not compatible
with ‘--query’). If only one value is given to ‘--width’ the
region will be a square, but if two values are given, the widths of
the query box along each dimension will be different. The value(s)
is (are) in the same units as the coordinate column (see ‘--ccol’,
usually RA and Dec which are degrees). You can use simple division
for each value directly on the command-line if you want relatively
small (and more human-friendly) sizes. For example, if you want
your box to be 1 arc-minutes along the RA and 2 arc-minutes along
Dec, you can use ‘--width=1/60,2/60’.
‘-g STR,FLT,FLT’
‘--range=STR,FLT,FLT’
The column name and numerical range (inclusive) of acceptable
values in that column (not compatible with ‘--query’). This option
can be called multiple times for applying range limits on many
columns in one call (thus greatly reducing the download size). For
example, when used on the ESA gaia database, you can use
‘--range=phot_g_mean_mag,10:15’ to only get rows that have a value
between 10 and 15 (inclusive on both sides) in the
‘phot_g_mean_mag’ column.
If you want all rows larger, or smaller, than a certain number, you
can use ‘inf’, or ‘-inf’ as the first or second values
respectively. For example, if you want objects with SDSS
spectroscopic redshifts larger than 2 (from the VizieR ‘sdss12’
database), you can use ‘--range=zsp,2,inf’
If you want the interval to not be inclusive on both sides, you can
run ‘astquery’ once and get the command that it executes. Then you
can edit it to be non-inclusive on your desired side.
‘-b STR[,STR]’
‘--noblank=STR[,STR]’
Only ask for rows that do not have a blank value in the ‘STR’
column. This option can be called many times, and each call can
have multiple column names (separated by a comma or <,>). For
example, if you want the retrieved rows to not have a blank value
in columns ‘A’, ‘B’, ‘C’ and ‘D’, you can use ‘--noblank=A
-bB,C,D’.
‘--sort=STR[,STR]’
Ask for the server to sort the downloaded data based on the given
columns. For example, let's assume your desired catalog has column
‘Z’ for redshift and column ‘MAG_R’ for magnitude in the R band.
When you call ‘--sort=Z,MAG_R’, it will primarily sort the columns
based on the redshift, but if two objects have the same redshift,
they will be sorted by magnitude. You can add as many columns as
you like for higher-level sorting.
6 Data manipulation
*******************
Images are one of the major formats of data that is used in astronomy.
The functions in this chapter explain the GNU Astronomy Utilities which
are provided for their manipulation. For example, cropping out a part
of a larger image or convolving the image with a given kernel or
applying a transformation to it.
6.1 Crop
========
Astronomical images are often very large, filled with thousands of
galaxies. It often happens that you only want a section of the image,
or you have a catalog of sources and you want to visually analyze them
in small postage stamps. Crop is made to do all these things. When
more than one crop is required, Crop will divide the crops between
multiple threads to significantly reduce the run time.
Astronomical surveys are usually extremely large. So large in fact,
that the whole survey will not fit into a reasonably sized file.
Because of this, surveys usually cut the final image into separate tiles
and store each tile in a file. For example, the COSMOS survey's Hubble
space telescope, ACS F814W image consists of 81 separate FITS images,
with each one having a volume of 1.7 Gigabytes.
Even though the tile sizes are chosen to be large enough that too
many galaxies/targets do not fall on the edges of the tiles, inevitably
some do. So when you simply crop the image of such targets from one
tile, you will miss a large area of the surrounding sky (which is
essential in estimating the noise). Therefore in its WCS mode, Crop
will stitch parts of the tiles that are relevant for a target (with the
given width) from all the input images that cover that region into the
output. Of course, the tiles have to be present in the list of input
files.
Besides cropping postage stamps around certain coordinates, Crop can
also crop arbitrary polygons from an image (or a set of tiles by
stitching the relevant parts of different tiles within the polygon), see
‘--polygon’ in *note Invoking astcrop::. Alternatively, it can crop out
rectangular regions through the ‘--section’ option from one image, see
*note Crop section syntax::.
6.1.1 Crop modes
----------------
In order to be comprehensive, intuitive, and easy to use, there are two
ways to define the crop:
1. From its center and side length. For example, if you already know
the coordinates of an object and want to inspect it in an image or
to generate postage stamps of a catalog containing many such
coordinates.
2. The vertices of the crop region, this can be useful for larger
crops over many targets, for example, to crop out a uniformly deep,
or contiguous, region of a large survey.
Irrespective of how the crop region is defined, the coordinates to
define the crop can be in Image (pixel) or World Coordinate System (WCS)
standards. All coordinates are read as floating point numbers (not
integers, except for the ‘--section’ option, see below). By setting the
_mode_ in Crop, you define the standard that the given coordinates must
be interpreted. Here, the different ways to specify the crop region are
discussed within each standard. For the full list options, please see
*note Invoking astcrop::.
When the crop is defined by its center, the respective (integer)
central pixel position will be found internally according to the FITS
standard. To have this pixel positioned in the center of the cropped
region, the final cropped region will have an add number of pixels (even
if you give an even number to ‘--width’ in image mode).
Furthermore, when the crop is defined as by its center, Crop allows
you to only keep crops what do not have any blank pixels in the vicinity
of their center (your primary target). This can be very convenient when
your input catalog/coordinates originated from another survey/filter
which is not fully covered by your input image, to learn more about this
feature, please see the description of the ‘--checkcenter’ option in
*note Invoking astcrop::.
Image coordinates
In image mode (‘--mode=img’), Crop interprets the pixel coordinates
and widths in units of the input data-elements (for example, pixels
in an image, not world coordinates). In image mode, only one image
may be input. The output crop(s) can be defined in multiple ways
as listed below.
Center of multiple crops (in a catalog)
The center of (possibly multiple) crops are read from a text
file. In this mode, the columns identified with the
‘--coordcol’ option are interpreted as the center of a crop
with a width of ‘--width’ pixels along each dimension. The
columns can contain any floating point value. The value to
‘--output’ option is seen as a directory which will host (the
possibly multiple) separate crop files, see *note Crop
output:: for more. For a tutorial using this feature, please
see *note Reddest clumps cutouts and parallelization::.
Center of a single crop (on the command-line)
The center of the crop is given on the command-line with the
‘--center’ option. The crop width is specified by the
‘--width’ option along each dimension. The given coordinates
and width can be any floating point number.
Vertices of a single crop
In Image mode there are two options to define the vertices of
a region to crop: ‘--section’ and ‘--polygon’. The former is
lower-level (does not accept floating point vertices, and only
a rectangular region can be defined), it is also only
available in Image mode. Please see *note Crop section
syntax:: for a full description of this method.
The latter option (‘--polygon’) is a higher-level method to
define any polygon (with any number of vertices) with floating
point values. Please see the description of this option in
*note Invoking astcrop:: for its syntax.
WCS coordinates
In WCS mode (‘--mode=wcs’), the coordinates and width are
interpreted using the World Coordinate System (WCS, that must
accompany the dataset), not pixel coordinates. You can optionally
use ‘--widthinpix’ for the width to be interpreted in pixels (even
though the coordinates are in WCS). In WCS mode, Crop accepts
multiple datasets as input. When the cropped region (defined by
its center or vertices) overlaps with multiple of the input
images/tiles, the overlapping regions will be taken from the
respective input (they will be stitched when necessary for each
output crop).
In this mode, the input images do not necessarily have to be the
same size, they just need to have the same orientation and pixel
resolution. Currently only orientation along the celestial
coordinates is accepted, if your input has a different orientation
or resolution you can use Warp's ‘--gridfile’ option to align the
image before cropping it (see *note Warp::).
Each individual input image/tile can even be smaller than the final
crop. In any case, any part of any of the input images which
overlaps with the desired region will be used in the crop. Note
that if there is an overlap in the input images/tiles, the pixels
from the last input image read are going to be used for the
overlap. Crop will not change pixel values, so it assumes your
overlapping tiles were cutout from the same original image. There
are multiple ways to define your cropped region as listed below.
Center of multiple crops (in a catalog)
Similar to catalog inputs in Image mode (above), except that
the values along each dimension are assumed to have the same
units as the dataset's WCS information. For example, the
central RA and Dec value for each crop will be read from the
first and second calls to the ‘--coordcol’ option. The width
of the cropped box (in units of the WCS, or degrees in RA and
Dec mode) must be specified with the ‘--width’ option. You
can optionally use ‘--widthinpix’ for the value of ‘--width’
to be interpreted in pixels.
Center of a single crop (on the command-line)
You can specify the center of only one crop box with the
‘--center’ option. If it exists in the input images, it will
be cropped similar to the catalog mode, see above also for
‘--width’.
Vertices of a single crop
The ‘--polygon’ option is a high-level method to define any
convex polygon (with any number of vertices). Please see the
description of this option in *note Invoking astcrop:: for its
syntax.
*CAUTION:* In WCS mode, the image has to be aligned with the
celestial coordinates, such that the first FITS axis is parallel
(opposite direction) to the Right Ascension (RA) and the second
FITS axis is parallel to the declination. If these conditions are
not met for an image, Crop will warn you and abort. You can use
Warp to align the input image to standard celestial coordinates,
see *note Warp::.
As a summary, if you do not specify a catalog, you have to define the
cropped region manually on the command-line. In any case the mode is
mandatory for Crop to be able to interpret the values given as
coordinates or widths.
6.1.2 Crop section syntax
-------------------------
When in image mode, one of the methods to crop only one rectangular
section from the input image is to use the ‘--section’ option. Crop has
a powerful syntax to read the box parameters from a string of
characters. If you leave certain parts of the string to be empty, Crop
can fill them for you based on the input image sizes.
To define a box, you need the coordinates of two points: the first
(‘X1’, ‘Y1’) and the last pixel (‘X2’, ‘Y2’) pixel positions in the
image, or four integer numbers in total. The four coordinates can be
specified with one string in this format: '‘X1:X2,Y1:Y2’'. This string
is given to the ‘--section’ option. Therefore, the pixels along the
first axis that are $\geq$‘X1’ and $\leq$‘X2’ will be included in the
cropped image. The same goes for the second axis. Note that each
different term will be read as an integer, not a float.
The reason it only accepts integers is that ‘--section’ is a
low-level option (which is also very fast!). For a higher-level way to
specify region (any polygon, not just a box), please see the ‘--polygon’
option in *note Crop options::. Also note that in the FITS standard,
pixel indexes along each axis start from unity(1) not zero(0).
You can omit any of the values and they will be filled automatically.
The left hand side of the colon (‘:’) will be filled with ‘1’, and the
right side with the image size. So, ‘2:,:’ will include the full range
of pixels along the second axis and only those with a first axis index
larger than ‘2’ in the first axis. If the colon is omitted for a
dimension, then the full range is automatically used. So the same
string is also equal to ‘2:,’ or ‘2:’ or even ‘2’. If you want such a
case for the second axis, you should set it to: ‘,2’.
If you specify a negative value, it will be seen as before the
indexes of the image which are outside the image along the bottom or
left sides when viewed in SAO DS9. In case you want to count from the
top or right sides of the image, you can use an asterisk (‘*’). When
confronted with a ‘*’, Crop will replace it with the maximum length of
the image in that dimension. So ‘*-10:*+10,*-20:*+20’ will mean that
the crop box will be 20\times40 pixels in size and only include the top
corner of the input image with 3/4 of the image being covered by blank
pixels, see *note Blank pixels::.
If you feel more comfortable with space characters between the
values, you can use as many space characters as you wish, just be
careful to put your value in double quotes, for example,
‘--section="5:200, 123:854"’. If you forget the quotes, anything after
the first space will not be seen by ‘--section’ and you will most
probably get an error because the rest of your string will be read as a
filename (which most probably does not exist). See *note Command-line::
for a description of how the command-line works.
6.1.3 Blank pixels
------------------
The cropped box can potentially include pixels that are beyond the image
range. For example, when a target in the input catalog was very near
the edge of the input image. The parts of the cropped image that were
not in the input image will be filled with the following two values
depending on the data type of the image. In both cases, SAO DS9 will
not color code those pixels.
• If the data type of the image is a floating point type (float or
double), IEEE NaN (Not a number) will be used.
• For integer types, pixels out of the image will be filled with the
value of the ‘BLANK’ keyword in the cropped image header. The
value assigned to it is the lowest value possible for that type, so
you will probably never need it any way. Only for the unsigned
character type (‘BITPIX=8’ in the FITS header), the maximum value
is used because it is unsigned, the smallest value is zero which is
often meaningful.
You can ask for such blank regions to not be included in the output
crop image using the ‘--noblank’ option. In such cases, there is no
guarantee that the image size of your outputs are what you asked for.
In some survey images, unfortunately they do not use the ‘BLANK’ FITS
keyword. Instead they just give all pixels outside of the survey area a
value of zero. So by default, when dealing with float or double image
types, any values that are 0.0 are also regarded as blank regions. This
can be turned off with the ‘--zeroisnotblank’ option.
6.1.4 Invoking Crop
-------------------
Crop will crop a region from an image. If in WCS mode, it will also
stitch parts from separate images in the input files. The executable
name is ‘astcrop’ with the following general template
$ astcrop [OPTION...] [ASCIIcatalog] ASTRdata ...
One line examples:
## Crop all objects in cat.txt from image.fits:
$ astcrop --catalog=cat.txt image.fits
## Crop all options in catalog (with RA,DEC) from all the files
## ending in `_drz.fits' in `/mnt/data/COSMOS/':
$ astcrop --mode=wcs --catalog=cat.txt /mnt/data/COSMOS/*_drz.fits
## Crop the outer 10 border pixels of the input image and give
## the output HDU a name ('EXTNAME' keyword in FITS) of 'mysection'.
$ astcrop --section=10:*-10,10:*-10 --hdu=2 image.fits \
--metaname=mysection
## Crop region around RA and Dec of (189.16704, 62.218203):
$ astcrop --mode=wcs --center=189.16704,62.218203 goodsnorth.fits
## Same crop above, but coordinates given in sexagesimal (you can
## also use ':' between the sexagesimal components).
$ astcrop --mode=wcs --center=12h36m40.08,62d13m5.53 goodsnorth.fits
## Crop region around pixel coordinate (568.342, 2091.719):
$ astcrop --mode=img --center=568.342,2091.719 --width=201 image.fits
## Crop all HDUs within a FITS file at a certain coordinate, while
## preserving the names of the HDUs in the output.
$ for hdu in $(astfits input.fits --listimagehdus); do \
astcrop input.fits --hdu=$hdu --append --output=crop.fits \
--metaname=$hdu --mode=wcs --center=189.16704,62.218203 \
--width=10/3600
done
Crop has one mandatory argument which is the input image name(s), shown
above with ‘ASTRdata ...’. You can use shell expansions, for example,
‘*’ for this if you have lots of images in WCS mode. If the crop box
centers are in a catalog, you can use the ‘--catalog’ option. In other
cases, you have to provide the single cropped output parameters must be
given with command-line options. See *note Crop output:: for how the
output file name(s) can be specified. For the full list of general
options to all Gnuastro programs (including Crop), please see *note
Common options::.
Floating point numbers can be used to specify the crop region (except
the ‘--section’ option, see *note Crop section syntax::). In such
cases, the floating point values will be used to find the desired
integer pixel indices based on the FITS standard. Hence, Crop
ultimately does not do any sub-pixel cropping (in other words, it does
not change pixel values). If you need such crops, you can use *note
Warp:: to first warp the image to the a new pixel grid, then crop from
that. For example, let's assume you want a crop from pixels 12.982 to
80.982 along the first dimension. You should first translate the image
by $-0.482$ (note that the edge of a pixel is at integer multiples of
$0.5$). So you should run Warp with ‘--translate=-0.482,0’ and then
crop the warped image with ‘--section=13:81’.
There are two ways to define the cropped region: with its center or
its vertices. See *note Crop modes:: for a full description. In the
former case, Crop can check if the central region of the cropped image
is indeed filled with data or is blank (see *note Blank pixels::), and
not produce any output when the center is blank, see the description
under ‘--checkcenter’ for more.
When in catalog mode, Crop will run in parallel unless you set
‘--numthreads=1’, see *note Multi-threaded operations::. Note that when
multiple outputs are created with threads, the outputs will not be
created in the same order. This is because the threads are asynchronous
and thus not started in order. This has no effect on each output, see
*note Reddest clumps cutouts and parallelization:: for a tutorial on
effectively using this feature.
6.1.4.1 Crop options
....................
The options can be classified into the following contexts: Input, Output
and operating mode options. Options that are common to all Gnuastro
program are listed in *note Common options:: and will not be repeated
here.
When you are specifying the crop vertices yourself (through
‘--section’, or ‘--polygon’) on relatively small regions (depending on
the resolution of your images) the outputs from image and WCS mode can
be approximately equivalent. However, as the crop sizes get large, the
curved nature of the WCS coordinates have to be considered. For
example, when using ‘--section’, the right ascension of the bottom left
and top left corners will not be equal. If you only want regions within
a given right ascension, use ‘--polygon’ in WCS mode.
Input image parameters:
‘--hstartwcs=INT’
Specify the first keyword card (line number) to start finding the
input image world coordinate system information. This is useful
when certain header keywords of the input may cause bad conflicts
with your crop (see an example described below). To get line
numbers of the header keywords, you can pipe the fully printed
header into ‘cat -n’ like below:
$ astfits image.fits -h1 | cat -n
For example, distortions have only been present in WCSLIB from
version 5.15 (released in mid 2016). Therefore some pipelines
still apply their own specific set of WCS keywords for distortions
and put them into the image header along with those that WCSLIB
does recognize. So now that WCSLIB recognizes most of the standard
distortion parameters, they will get confused with the old ones and
give wrong results. For example, in the CANDELS-GOODS South images
that were created before WCSLIB 5.15(1).
The two ‘--hstartwcs’ and ‘--hendwcs’ are thus provided so when
using older datasets, you can specify what region in the FITS
headers you want to use to read the WCS keywords. Note that this
is only relevant for reading the WCS information, basic data
information like the image size are read separately. These two
options will only be considered when the value to ‘--hendwcs’ is
larger than that of ‘--hstartwcs’. So if they are equal or
‘--hstartwcs’ is larger than ‘--hendwcs’, then all the input
keywords will be parsed to get the WCS information of the image.
‘--hendwcs=INT’
Specify the last keyword card to read for specifying the image
world coordinate system on the input images. See ‘--hstartwcs’
Crop box parameters:
‘-c FLT[,FLT[,...]]’
‘--center=FLT[,FLT[,...]]’
The central position of the crop in the input image. The positions
along each dimension must be separated by a comma (<,>) and
fractions are also acceptable. The comma-separated values can
either be in degrees (a single number), or sexagesimal (‘_h_m_’ for
RA, ‘_d_m_’ for Dec, or ‘_:_:_’ for both).
The number of values given to this option must be the same as the
dimensions of the input dataset. The width of the crop should be
set with ‘--width’. The units of the coordinates are read based on
the value to the ‘--mode’ option, see below.
‘-O STR’
‘--mode=STR’
Mode to interpret the crop's coordinates (for example with
‘--center’, ‘--catalog’ or ‘--polygon’). The value must either be
‘img’ (to assume image/pixel coordinates) or ‘wcs’ (to assume WCS,
usually RA/Dec, coordinates), see *note Crop modes:: for a full
description.
‘-w FLT[,FLT[,...]]’
‘--width=FLT[,FLT[,...]]’
Width of the cropped region about coordinate given to ‘--center’.
If in WCS mode, value(s) given to this option will be read in the
same units as the dataset's WCS information along this dimension
(unless ‘--widthinpix’ is given). This option may take either a
single value (to be used for all dimensions: ‘--width=10’ in
image-mode will crop a $10\times10$ pixel image) or multiple values
(a specific value for each dimension: ‘--width=10,20’ in image-mode
will crop a $10\times20$ pixel image).
The ‘--width’ option also accepts fractions. For example, if you
want the width of your crop to be 3 by 5 arcseconds along RA and
Dec respectively and you are in wcs-mode, you can use:
‘--width=3/3600,5/3600’.
The final output will have an odd number of pixels to allow easy
identification of the pixel which keeps your requested coordinate
(from ‘--center’ or ‘--catalog’). If you want an even sided crop,
you can run Crop afterwards with ‘--section=":*-1,:*-1"’ or
‘--section=2:,2:’ (depending on which side you do not need), see
*note Crop section syntax::.
The basic reason for making an odd-sided crop is that your given
central coordinate will ultimately fall within a discrete pixel in
the image (defined by the FITS standard). When the crop has an odd
number of pixels in each dimension, that pixel can be very well
defined as the "central" pixel of the crop, making it unambiguously
easy to identify. However, for an even-sided crop, it will be very
hard to identify the central pixel (it can be on any of the four
pixels adjacent to the central point of the image!).
‘-X’
‘--widthinpix’
In WCS mode, interpret the value to ‘--width’ as number of pixels,
not the WCS units like degrees. This is useful when you want a
fixed crop size in pixels, even though your center coordinates are
in WCS (for example, RA and Dec).
‘-l STR’
‘-l FLT:FLT,...’
‘--polygon=STR’
‘--polygon=FLT,FLT:FLT,FLT:...’
Polygon vertice coordinates (when value is in ‘FLT,FLT:FLT,FLT:...’
format) or the filename of a SAO DS9 region file (when the value
has no ‘,’ or ‘:’ characters). Each vertice can either be in
degrees (a single floating point number) or sexagesimal (in formats
of '‘_h_m_’' for RA and '‘_d_m_’' for Dec, or simply '‘_:_:_’' for
either of them).
The vertices are used to define the polygon: in the same order
given to this option. When the vertices are not necessarily
ordered in the proper order (for example, one vertice in a square
comes after its diagonal opposite), you can add the ‘--polygonsort’
option which will attempt to sort the vertices before cropping.
Note that for concave polygons, sorting is not recommended because
there is no unique solution, for more, see the description under
‘--polygonsort’.
This option can be used both in the image and WCS modes, see *note
Crop modes::. If a SAO DS9 region file is used, the coordinate
mode of Crop will be determined by the contents of the file and any
value given to ‘--mode’ is ignored. The cropped image will be the
size of the rectangular region that completely encompasses the
polygon. By default all the pixels that are outside of the polygon
will be set as blank values (see *note Blank pixels::). However,
if ‘--polygonout’ is called all pixels internal to the vertices
will be set to blank. In WCS-mode, you may provide many FITS
images/tiles: Crop will stitch them to produce this cropped region,
then apply the polygon.
The syntax for the polygon vertices is similar to, and simpler
than, that for ‘--section’. In short, the dimensions of each
coordinate are separated by a comma (<,>) and each vertex is
separated by a colon (<:>). You can define as many vertices as you
like. If you would like to use space characters between the
dimensions and vertices to make them more human-readable, then you
have to put the value to this option in double quotation marks.
For example, let's assume you want to work on the deepest part of
the WFC3/IR images of Hubble Space Telescope eXtreme Deep Field
(HST-XDF). According to the web page
(https://archive.stsci.edu/prepds/xdf/)(2) the deepest part is
contained within the coordinates:
[ (53.187414,-27.779152), (53.159507,-27.759633),
(53.134517,-27.787144), (53.161906,-27.807208) ]
They have provided mask images with only these pixels in the
WFC3/IR images, but what if you also need to work on the same
region in the full resolution ACS images? Also what if you want to
use the CANDELS data for the shallow region? Running Crop with
‘--polygon’ will easily pull out this region of the image for you,
irrespective of the resolution. If you have set the operating mode
to WCS mode in your nearest configuration file (see *note
Configuration files::), there is no need to call ‘--mode=wcs’ on
the command-line.
$ astcrop --mode=wcs desired-filter-image(s).fits \
--polygon="53.187414,-27.779152 : 53.159507,-27.759633 : \
53.134517,-27.787144 : 53.161906,-27.807208"
More generally, you have an image and want to define the polygon
yourself (it is not already published like the example above). As
the number of vertices increases, checking the vertex coordinates
on a FITS viewer (for example, SAO DS9) and typing them in, one by
one, can be very tedious and prone to typo errors. In such cases,
you can make a polygon "region" in DS9 and using your mouse, easily
define (and visually see) it. Given that SAO DS9 has a graphic
user interface (GUI), if you do not have the polygon vertices
before-hand, it is much more easier build your polygon there and
pass it onto Crop through the region file.
You can take the following steps to make an SAO DS9 region file
containing your polygon. Open your desired FITS image with SAO DS9
and activate its "region" mode with Edit→Region. Then define the
region as a polygon with Region→Shape→Polygon. Click on the
approximate center of the region you want and a small square will
appear. By clicking on the vertices of the square you can shrink
or expand it, clicking and dragging anywhere on the edges will
enable you to define a new vertex. After the region has been
nicely defined, save it as a file with Region→"Save Regions". You
can then select the name and address of the output file, keep the
format as ‘REG (*.reg)’ and press the "OK" button. In the next
window, keep format as "ds9" and "Coordinate System" as "fk5" for
RA and Dec (or "Image" for pixel coordinates). A plain text file
is now created (let's call it ‘ds9.reg’) which you can pass onto
Crop with ‘--polygon=ds9.reg’.
For the expected format of the region file, see the description of
‘gal_ds9_reg_read_polygon’ in *note SAO DS9 library::. However,
since SAO DS9 makes this file for you, you do not usually need to
worry about its internal format unless something un-expected
happens and you find a bug.
‘--polygonout’
Keep all the regions outside the polygon and mask the inner ones
with blank pixels (see *note Blank pixels::). This is practically
the inverse of the default mode of treating polygons. Note that
this option only works when you have only provided one input image.
If multiple images are given (in WCS mode), then the full area
covered by all the images has to be shown and the polygon excluded.
This can lead to a very large area if large surveys like COSMOS are
used. So Crop will abort and notify you. In such cases, it is
best to crop out the larger region you want, then mask the smaller
region with this option.
‘--polygonsort’
Sort the given set of vertices to the ‘--polygon’ option. For a
concave polygon it will sort the vertices correctly, however for a
convex polygon it there is no unique sorting, so be careful because
the crop may not be what you expected.
Polygons come in two classes: convex and concave (or generally,
non-convex!), see below for a demonstration. Convex polygons are
those where all inner angles are less than 180 degrees. By
contrast, a concave polygon is one where an inner angle may be more
than 180 degrees.
Concave Polygon Convex Polygon
D --------C D------------- C
\ | E / |
\E | \ |
/ | \ |
A--------B A ----------B
‘-s STR’
‘--section=STR’
Section of the input image which you want to be cropped. See *note
Crop section syntax:: for a complete explanation on the syntax
required for this input.
‘-C FITS/TXT’
‘--catalog=FITS/TXT’
File name of catalog for making multiple crops from the input
images/cubes. The catalog can be in any of Gnuastro's recognized
*note Recognized table formats::. The columns containing the
coordinates for the crop centers can be specified with the
‘--coordcol’ option (using column names or numbers, see *note
Selecting table columns::). The catalog can also contain the name
of each crop, you can specify the column containing the name with
the ‘--namecol’.
‘--cathdu=STR/INT’
The HDU (extension) containing the catalog (if the file given to
‘--catalog’ is a FITS file). This can either be the HDU name (if
it has one) or number (counting from 0). By default (if this
option is not given), the second HDU will be used (equivalent to
‘--cathdu=1’. For more on how to specify the HDU, see the
explanation of the ‘--hdu’ option in *note Input output options::.
‘-x STR/INT’
‘--coordcol=STR/INT’
The column in a catalog to read as a coordinate. The value can be
either the column number (starting from 1), or a match/search in
the table meta-data, see *note Selecting table columns::. This
option must be called multiple times, depending on the number of
dimensions in the input dataset. If it is called more than
necessary, the extra columns (later calls to this option on the
command-line or configuration files) will be ignored, see *note
Configuration file precedence::.
‘-n STR/INT’
‘--namecol=STR/INT’
Column selection of crop file name. The value can be either the
column number (starting from 1), or a match/search in the table
meta-data, see *note Selecting table columns::. This option can be
used both in Image and WCS modes, and not a mandatory. When a
column is given to this option, the final crop base file name will
be taken from the contents of this column. The directory will be
determined by the ‘--output’ option (current directory if not
given) and the value to ‘--suffix’ will be appended. When this
column is not given, the row number will be used instead.
---------- Footnotes ----------
(1)
(2)
6.1.4.2 Crop output
...................
The string given to ‘--output’ option will be interpreted depending on
how many crops were requested, see *note Crop modes:::
• When a catalog is given, the value of the ‘--output’ (see *note
Common options::) will be read as the directory to store the output
cropped images. Hence if it does not already exist, Crop will
abort with an "No such file or directory" error.
The crop file names will consist of two parts: a variable part (the
row number of each target starting from 1) along with a fixed
string which you can set with the ‘--suffix’ option. Optionally,
you may also use the ‘--namecol’ option to define a column in the
input catalog to use as the file name instead of numbers.
• When only one crop is desired, the value to ‘--output’ will be read
as a file name. If no output is specified or if it is a directory,
the output file name will follow the automatic output names of
Gnuastro, see *note Automatic output::: The string given to
‘--suffix’ will be replaced with the ‘.fits’ suffix of the input.
When the desired crop is not within the input image(s) crop will not
produce any image for that crop. If the ‘--quiet’ option is not given,
crop will report which one of the inputs was created and which wasn't
(due to an overlap or the center being empty, see ‘--checkcenter’. This
information can be formally written in an optional log-file also, see
below. At the end, Crop will return successfully to the shell if at
least one output file was created. If no output was created at all,
then crop will return to the shell with a failure.
By default, as suggested by the FITS standard and implemented in all
Gnuastro programs, the first/primary extension of the output files will
only contain metadata. The cropped images/cubes will be written into
the 2nd HDU of their respective FITS file (which is actually counted as
‘1’ because HDU counting starts from ‘0’). However, if you want the
cropped data to be written into the primary (0-th) HDU, run Crop with
the ‘--primaryimghdu’ option.
If the output file already exists by default Crop will re-write it
(so that all existing HDUs in it will be deleted). If you want the
cropped HDU to be appended to existing HDUs, use ‘--append’ described
below.
The 0-th HDU of each output cropped image will contain the names of
the input image(s) it was cut from. If a name is longer than the 70
character space that the FITS standard allows for header keyword values,
the name will be cut into several keywords from the nearest slash ().
The keywords have the following format: ‘ICFn_m’ (for Crop File). Where
‘n’ is the number of the image used in this crop and ‘m’ is the part of
the name (it can be broken into multiple keywords). Following the name
is another keyword named ‘ICFnPIX’ which shows the pixel range from that
input image in the same syntax as *note Crop section syntax::. So this
string can be directly given to the ‘--section’ option later.
Once done, a log file can be created in the current directory with
the ‘--log’ option. This file will have three columns and the same
number of rows as the number of cropped images. There are also comments
on the top of the log file explaining basic information about the run
and descriptions for the columns. A short description of the columns is
also given below:
1. The cropped image file name for that row.
2. The number of input images that were used to create that image.
3. A ‘0’ if the central few pixels (value to the ‘--checkcenter’
option) are blank and ‘1’ if they are not. When the crop was not
defined by its center (see *note Crop modes::), or ‘--checkcenter’
was given a value of 0 (see *note Invoking astcrop::), the center
will not be checked and this column will be given a value of ‘-1’.
If the output crop(s) have a single element (pixel in an image) and
‘--oneelemstdout’ has been called, no output file will be produced!
Instead, the single element's value is printed on the standard output.
See the description of ‘--oneelemstdout’ below for more:
‘-p STR’
‘--suffix=STR’
The suffix (or post-fix) of the output files for when you want all
the cropped images to have a special ending. One case where this
might be helpful is when besides the science images, you want the
weight images (or exposure maps, which are also distributed with
survey images) of the cropped regions too. So in one run, you can
set the input images to the science images and ‘--suffix=_s.fits’.
In the next run you can set the weight images as input and
‘--suffix=_w.fits’.
‘-a STR’
‘--metaname=STR’
Name of cropped HDU (value to the ‘EXTNAME’ keyword of FITS). If
not given, a default ‘CROP’ will be placed there (so the ‘EXTNAME’
keyword will always be present in the output). If crop produces
many outputs from a catalog, they will be given the same string as
‘EXTNAME’ (the file names containing the cropped HDU will be
different).
‘-A’
‘--append’
If the output file already exists, append the cropped image HDU to
the end of any existing HDUs. By default (when this option isn't
given), if an output file already exists, any existing HDU in it
will be deleted. If the output file doesn't exist, this option is
redundant.
‘--primaryimghdu’
Write the output into the primary (0-th) HDU/extension of the
output. By default, like all Gnuastro's default outputs, no data
is written in the primary extension because the FITS standard
suggests keeping that extension free of data and only for metadata.
‘-t’
‘--oneelemstdout’
When a crop only has a single element (a single pixel), print it to
the standard output instead of making a file. By default (without
this option), a single-pixel crop will be saved to a file, just
like a crop of any other size.
When a single crop is requested (either through ‘--center’, or a
catalog of one row is given), the single value alone is printed
with nothing else. This makes it easy to immediately write the
value into a shell variable for example:
value=$(astcrop img.fits --mode=wcs --center=1.234,5.678 \
--width=1 --widthinpix --oneelemstdout \
--quiet)
If a catalog of coordinates is given (that would produce multiple
crops; or multiple values in this scenario), the solution for a
single value will not work! Recall that Crop will do the crops in
parallel, therefore each time you run it, the order of the rows
will be different and not correspond to the order of the inputs.
To allow identification of each value (which row of the input
catalog it corresponds to), Crop will first print the name of the
would-be created file name, and print the value after it (separated
by an empty SPACE character). In other words, the file in the
first column will not actually be created, but the value of the
pixel it would have contained (if this option was not called) is
printed after it.
‘-c FLT/INT’
‘--checkcenter=FLT/INT’
Square box width of region in the center of the image to check for
blank values. If any of the pixels in this central region of a
crop (defined by its center) are blank, then it will not be stored
in an output file. If the value to this option is zero, no
checking is done. This check is only applied when the cropped
region(s) are defined by their center (not by the vertices, see
*note Crop modes::).
The units of the value are interpreted based on the ‘--mode’ value
(in WCS or pixel units). The ultimate checked region size (in
pixels) will be an odd integer around the center (converted from
WCS, or when an even number of pixels are given to this option).
In WCS mode, the value can be given as fractions, for example, if
the WCS units are in degrees, ‘0.1/3600’ will correspond to a check
size of 0.1 arcseconds.
Because survey regions do not often have a clean square or
rectangle shape, some of the pixels on the sides of the survey FITS
image do not commonly have any data and are blank (see *note Blank
pixels::). So when the catalog was not generated from the input
image, it often happens that the image does not have data over some
of the points.
When the given center of a crop falls in such regions or outside
the dataset, and this option has a non-zero value, no crop will be
created. Therefore with this option, you can specify a width of a
small box (3 pixels is often good enough) around the central pixel
of the cropped image. You can check which crops were created and
which were not from the command-line (if ‘--quiet’ was not called,
see *note Operating mode options::), or in Crop's log file (see
*note Crop output::).
‘-b’
‘--noblank’
Pixels outside of the input image that are in the crop box will not
be used. By default they are filled with blank values (depending
on type), see *note Blank pixels::. This option only applies only
in Image mode, see *note Crop modes::.
‘-z’
‘--zeroisnotblank’
In float or double images, it is common to give the value of zero
to blank pixels. If the input image type is one of these two
types, such pixels will also be considered as blank. You can
disable this behavior with this option, see *note Blank pixels::.
6.1.4.3 Crop known issues
.........................
When running Crop, you may encounter strange errors and bugs. In these
cases, please report a bug and we will try to fix it as soon as
possible, see *note Report a bug::. However, some things are beyond our
control, or may take too long to fix directly. In this section we list
such known issues that may occur in known cases and suggest the hack (or
work-around) to fix the problem:
Crash with ‘Killed’ when cropping catalog from ‘.fits.gz’
This happens because CFISTIO (that reads and writes FITS files)
will internally decompress the file in a temporary place (possibly
in the RAM), then start reading from it. On the other hand, by
default when given a catalog (with many crops) and not specifying
‘--numthreads’, Crop will use the maximum number of threads
available on your system to do each crop faster. On an normal (not
compressed) file, parallel access will not cause a problem,
however, when attempting parallel access with the maximum number of
threads on a compressed file, CFITSIO crashes with ‘Killed’.
Therefore the following solutions can be used to fix this crash:
• Decrease the number of threads (at the minimum, set
‘--numthreads=1’). Since this solution does not attempt to
change any of your previous Crop command components or does
not change your local file structure, it is the preferred way.
• Decompress the file (with the command below) and feed the
‘.fits’ file into Crop without changing the number of threads.
$ gunzip -k image.fits.gz
6.2 Arithmetic
==============
It is commonly necessary to do operations on some or all of the elements
of a dataset independently (pixels in an image). For example, in the
reduction of raw data it is necessary to subtract the Sky value (*note
Sky value::) from each image image. Later (once the images as warped
into a single grid using Warp for example, see *note Warp::), the images
are co-added (the output pixel grid is the average of the pixels of the
individual input images). Arithmetic is Gnuastro's program for such
operations on your datasets directly from the command-line. It
currently uses the reverse polish or post-fix notation, see *note
Reverse polish notation:: and will work on the native data types of the
input images/data to reduce CPU and RAM resources, see *note Numeric
data types::. For more information on how to run Arithmetic, please see
*note Invoking astarithmetic::.
6.2.1 Reverse polish notation
-----------------------------
The most common notation for arithmetic operations is the infix notation
(https://en.wikipedia.org/wiki/Infix_notation) where the operator goes
between the two operands, for example, $4+5$. The infix notation is the
preferred way in most programming languages which come with scripting
features for large programs. This is because the infix notation
requires a way to define precedence when more than one operator is
involved.
For example, consider the statement ‘5 + 6 / 2’. Should 6 first be
divided by 2, then added by 5? Or should 5 first be added with 6, then
divided by 2? Therefore we need parenthesis to show precedence:
‘5+(6/2)’ or ‘(5+6)/2’. Furthermore, if you need to leave a value for
later processing, you will need to define a variable for it; for
example, ‘a=(5+6)/2’.
Gnuastro provides libraries where you can also use infix notation in
C or C++ programs. However, Gnuastro's programs are primarily designed
to be run on the command-line and the level of complexity that infix
notation requires can be annoying/confusing to write on the command-line
(where they can get confused with the shell's parenthesis or variable
definitions). Therefore Gnuastro's Arithmetic and Table (when doing
column arithmetic) programs use the post-fix notation, also known as
reverse polish notation
(https://en.wikipedia.org/wiki/Reverse_Polish_notation). For example,
instead of writing ‘5+6’, we write ‘5 6 +’.
The Wikipedia article on the reverse polish notation provides some
excellent explanation on this notation but here we will give a short
summary here for self-sufficiency. In short, in the reverse polish
notation, the operator is placed after the operands. As we will see
below this removes the need to define parenthesis and lets you use
previous values without needing to define a variable. In the future(1)
we do plan to also optionally allow infix notation when arithmetic
operations on datasets are desired, but due to time constraints on the
developers we cannot do it immediately.
To easily understand how the reverse polish notation works, you can
think of each operand (‘5’ and ‘6’ in the example above) as a node in a
"last-in-first-out" stack. One such stack in daily life is a stack of
dishes in the kitchen: you put a clean dish, on the top of a stack of
dishes when it is ready for later usage. Later, when you need a dish,
you pick the top one (hence the "last" dish placed "in" the stack is the
"first" dish that comes "out" when necessary).
Each operator will need a certain number of operands (in the example
above, the ‘+’ operator needs two operands: ‘5’ and ‘6’). In the
kitchen metaphor, an operator can be an oven. Every time an operator is
confronted, the operator takes (or "pops") the number of operands it
needs from the top of the stack (so they do not exist in the stack any
more), does its operation, and places (or "pushes") the result back on
top of the stack. So if you want the average of 5 and 6, you would
write: ‘5 6 + 2 /’. The operations that are done are:
1. ‘5’ is an operand, so Arithmetic pushes it to the top of the stack
(which is initially empty). In the kitchen metaphor, you can
visualize this as taking a new dish from the cabinet, putting the
number 5 inside of the dish, and putting the dish on top of the
(empty) cooking table in front of you. You now have a stack of one
dish on the table in front of you.
2. ‘6’ is also an operand, so it is pushed to the top of the stack.
Like before, you can visualize this as taking a new dish from the
cabinet, putting the number 6 in it and placing it on top of the
previous dish. You now have a stack of two dishes on the table in
front of you.
3. ‘+’ is a _binary_ operator, so it will pop the top two elements of
the stack out of it, and perform addition on them (the order is
$5+6$ in the example above). The result is ‘11’ which is pushed to
the top of the stack.
To visualize this, you can think of the ‘+’ operator as an oven
with a place for two dishes. You pick up the top-most dish (that
has the number 6 in it) and put it in the oven. The top dish is
now the one that has the number 5. You also pick it up and put it
in the oven, and close the oven door. When the oven has finished
its cooking, it produces a single output (in one dish, with the
number 11 inside of it). You take that output dish and put it back
on the table. You now have a stack of one dish on the table in
front of you.
4. ‘2’ is an operand so push it onto the top of the stack. In the
kitchen metaphor, you again go to the cabinet, pick up a dish and
put the number 2 inside of it and put the dish over the previous
dish (that has the number 11). You now have a stack of two dishes
on the table in front of you.
5. ‘/’ (division) is a binary operator, so pull out the top two
elements of the stack (top-most is ‘2’, then ‘11’) and divide the
second one by the first. In the kitchen metaphor, the ‘/’ operator
can be visualized as a microwave that takes two dishes. But unlike
the oven (‘+’ operator) before, the order of inputs matters (they
are on top of each other: with the top dish holder being the
numerator and the bottom one being the denominator). Again, you
look at your stack of dishes on the table.
You pick up the top one (with value 2 inside of it) and put it in
the microwave's bottom (denominator) dish holder. Then you go back
to your stack of dishes on the table and pick up the top dish (with
value 11 inside of it) and put that in the top (nominator) dish
holder. The microwave will do its work and when it is finished,
returns a new dish with the single value 5.5 inside of it. You
pick up the dish from the microwave and place it back on the table.
6. There are no more operands or operators, so simply return the
remaining operand in the output. In the kitchen metaphor, you see
that your recipe has no more steps, so you just pick up the
remaining dish and take it to the dining room to enjoy a good
dinner.
In the Arithmetic program, the operands can be FITS images of any
dimensionality, or numbers (see *note Invoking astarithmetic::). In
Table's column arithmetic, they can be any column in the table (a series
of numbers in an array) or a single number (see *note Column
arithmetic::).
With this notation, very complicated procedures can be created
without the need for parenthesis or worrying about precedence. Even
functions which take an arbitrary number of arguments can be defined in
this notation. This is a very powerful notation and is used in
languages like Postscript (2) which produces PDF files when compiled.
---------- Footnotes ----------
(1)
(2) See the EPS and PDF part of *note Recognized file formats:: for a
little more on the Postscript language.
6.2.2 Integer benefits and pitfalls
-----------------------------------
Integers are the simplest numerical data types (*note Numeric data
types::). Because of this, their storage space is much less, and their
processing is much faster than floating point types. You can confirm
this on your computer with the series of commands below. You will make
four 5000 by 5000 pixel images filled with random values. Two of them
will be saved as signed 8-bit integers, and two with 64-bit floating
point types. The last command prints the size of the created images.
$ astarithmetic 5000 5000 2 makenew 5 mknoise-sigma int8 -oint-1.fits
$ astarithmetic 5000 5000 2 makenew 5 mknoise-sigma int8 -oint-2.fits
$ astarithmetic 5000 5000 2 makenew 5 mknoise-sigma float64 -oflt-1.fits
$ astarithmetic 5000 5000 2 makenew 5 mknoise-sigma float64 -oflt-2.fits
$ ls -lh int-*.fits flt-*.fits
The 8-bit integer images are only 24MB, while the 64-bit floating
point images are 191 MB! Besides helping in storage (on your disk, or in
RAM, while the program is running), the small size of these files also
helps in faster reading of the inputs. Furthermore, CPUs can process
integer operations much faster than floating points. In the integers,
the ones with a smaller width (number of bits) can be processed much
faster. You can see this with the two commands below where you will add
the integer images with each other and the floats with each other:
$ astarithmetic flt-1.fits flt-2.fits + -oflt-sum.fits -g1
$ astarithmetic int-1.fits int-2.fits + -oint-sum.fits -g1
Have a look at the running time of the two commands above (that is
printed on their last line). On the system that this paragraph was
written on, the floating point and integer image sums were respectively
done in 0.481 and 0.089 seconds (the integer operation was almost 5
times faster!).
*If your data does not have decimal points, use integer types:* integer
types are much faster and can take much less space in your storage or
RAM (while the program is running).
*Select the smallest width that can host the range/precision of values:*
for example, if the largest possible value in your dataset is 1000 and
all numbers are integers, store it as a 16-bit integer. Also, if you
know the values can never become negative, store it as an unsigned
16-bit integer. For floating point types, if you know you will not need
a precision of more than 6 significant digits, use the 32-bit floating
point type. For more on the range (for integers) and precision (for
floats), see *note Numeric data types::.
There is a price to be paid for this improved efficiency in integers:
your wisdom! If you have not selected your types wisely, strange
situations may happen. For example, try the command below:
$ astarithmetic 125 10 +
You expect the output to be $135$, but it will be $-121$! The reason is
that when Arithmetic (or column-arithmetic in Table) confronts a number
on the command-line, it use the principles above to select the most
efficient type for each number. Both $125$ and $10$ can safely fit
within a signed, 8-bit integer type, so arithmetic will store both as an
8-bit integer. However, the sum ($135$) is larger than the maximum
possible value of an 8-bit signed integer ($127$). Therefore an integer
overflow will occur, and the bits will be over-written. As a result,
the value will be $135-128=7$ more than the minimum value of this type
($-128$), which is $-128+7=-121$.
When you know situations like this may occur, you can simply use
*note Numerical type conversion operators::, to set just one of the
inputs to a wider data type (the smallest, wider type to avoid wasting
resources). In the example above, this would be ‘uint16’:
$ astarithmetic 125 uint16 10 +
The reason this worked is that $125$ is now converted into an
unsigned 16-bit integer before the ‘+’ operator. Since this is larger
than an 8-bit integer, the C programming language's automatic type
conversion will treat both as the wider type and store the result of the
binary operation (‘+’) in that type.
For such a basic operation like the command above, a faster hack
would be any of the two commands below (which are equivalent). This is
because ‘125.0’ or ‘125.’ are interpreted as floating-point types and
they do not suffer from such issues (converting only on one input is
enough):
$ astarithmetic 125. 10 +
$ astarithmetic 125.0 10 +
For this particular command, the fix above will be as fast as the
‘uint16’ solution. This is because there are only two numbers, and the
overhead of Arithmetic (reading configuration files, etc.) dominates
the running time. However, for large datasets, the ‘uint16’ solution
will be faster (as you saw above), Arithmetic will consume less RAM
while running, and the output will consume less storage in your system
(all major benefits)!
It is possible to do internal checks in Gnuastro and catch integer
overflows and correct them internally. However, we have not opted for
this solution because all those checks will consume significant
resources and slow down the program (especially with large datasets
where RAM, storage and running time become important). To be optimal,
we therefore trust that you (the wise Gnuastro user!) make the
appropriate type conversion in your commands where necessary (recall
that the operators are available in *note Numerical type conversion
operators::).
6.2.3 Noise basics
------------------
Deep astronomical images, like those used in extragalactic studies,
seriously suffer from noise in the data. Generally speaking, the
sources of noise in an astronomical image are photon counting noise and
Instrumental noise which are discussed in *note Photon counting noise::
and *note Instrumental noise::. This review finishes with *note
Generating random numbers:: which is a short introduction on how random
numbers are generated. We will see that while software random number
generators are not perfect, they allow us to obtain a reproducible
series of random numbers through setting the random number generator
function and seed value. Therefore in this section, we will also
discuss how you can set these two parameters in Gnuastro's programs
(including the arithmetic operators in *note Random number
generators::).
6.2.3.1 Photon counting noise
.............................
With the very accurate electronics used in today's detectors, photon
counting noise(1) is the most significant source of uncertainty in most
datasets. To understand this noise (error in counting) and its effect
on the images of astronomical targets, let's start by reviewing how a
distribution produced by counting can be modeled as a parametric
function.
Counting is an inherently discrete operation, which can only produce
positive integer outputs (including zero). For example, we cannot count
$3.2$ or $-2$ of anything. We only count $0$, $1$, $2$, $3$ and so on.
The distribution of values, as a result of counting efforts is formally
known as the Poisson distribution
(https://en.wikipedia.org/wiki/Poisson_distribution). It is associated
to Siméon Denis Poisson, because he discussed it while working on the
number of wrongful convictions in court cases in his 1837 book(2).
Let's take $\lambda$ to represent the expected mean count of
something. Furthermore, let's take $k$ to represent the output of a
counting attempt (hence $k$ is a positive integer). The probability
density function of getting $k$ counts (in each attempt, given the
expected/mean count of $\lambda$) can be written as:
$$f(k)={\lambda^k \over k!} e^{-\lambda},\quad k\in {0, 1, 2, 3, \dots }$$
Because the Poisson distribution is only applicable to positive
integer values (note the factorial operator, which only applies to
non-negative integers), naturally it is very skewed when $\lambda$ is
near zero. One qualitative way to understand this behavior is that for
smaller values near zero, there simply are not enough integers smaller
than the mean, than integers that are larger. Therefore to accommodate
all possibilities/counts, it has to be strongly skewed to the positive
when the mean is small. For more on Skewness, see *note Skewness caused
by signal and its measurement::.
As $\lambda$ becomes larger, the distribution becomes more and more
symmetric, and the variance of that distribution is equal to its mean.
In other words, the standard deviation is the square root of the mean.
It can also be proved that when the mean is large, say $\lambda>1000$,
the Poisson distribution approaches the Normal (Gaussian) distribution
(https://en.wikipedia.org/wiki/Normal_distribution) with mean
$\mu=\lambda$ and standard deviation $\sigma=\sqrt{\lambda}$. In other
words, a Poisson distribution (with a sufficiently large $\lambda$) is
simply a Gaussian that has one free parameter ($\mu=\lambda$ and
$\sigma=\sqrt{\lambda}$), instead of the two parameters that the
Gaussian distribution originally has (independent $\mu$ and $\sigma$).
In real situations, the photons/flux from our targets are combined
with photons from a certain background (observationally, the _Sky_
value). The Sky value is defined to be the average flux of a region in
the dataset with no targets. Its physical origin can be the brightness
of the atmosphere (for ground-based instruments), possible stray light
within the imaging instrument, the average flux of undetected targets,
etc. The Sky value is thus an ideal definition, because in real
datasets, what lies deep in the noise (far lower than the detection
limit) is never known(3). To account for all of these, the sky value is
defined to be the average count/value of the undetected regions in the
image. In a mock image/dataset, we have the luxury of setting the
background (Sky) value.
In summary, the value in each element of the dataset (pixel in an
image) is the sum of contributions from various galaxies and stars
(after convolution by the PSF, see *note PSF::). Let's name the
convolved sum of possibly overlapping objects in each pixel as $I_{nn}$.
$nn$ represents 'no noise'. For now, let's assume the background ($B$)
is constant and sufficiently high for the Poisson distribution to be
approximated by a Gaussian. Then the flux of that pixel, after adding
noise, is _a random value_ taken from a Gaussian distribution with the
following mean ($\mu$) and standard deviation ($\sigma$):
$$\mu=B+I_{nn}, \quad \sigma=\sqrt{B+I_{nn}}$$
In astronomical instruments, $B$ is enhanced by adding a "bias" level
to each pixel before the shutter is even opened (for the exposure to
start). As the exposure is ongoing and photo-electrons are accumulating
from the astronomical objects, a "dark" current (due to thermal
radiation of the instrument) also builds up in the pixels. The "dark"
current will accumulate even when the shutter is closed, but the CCD
electronics are working (hence the name "dark"). This added dark level
further enhances the mean value in a real observation compared to the
raw background value (from the atmosphere for example).
Since this type of noise is inherent in the objects we study, it is
usually measured on the same scale as the astronomical objects, namely
the magnitude system, see *note Brightness flux magnitude::. It is then
internally converted to the flux scale for further processing.
The equations above clearly show the importance of the background
value and its effect on the final signal to noise ratio in each pixel of
a science image. It is therefore, one of the most important factors in
understanding the noise (and properly simulating observations where
necessary). An inappropriately bright background value can hide the
signal of the mock profile hide behind the noise. In other words, a
brighter background has larger standard deviation and vice versa. As a
result, the only necessary parameter to define photon-counting noise
over a mock image of simulated profiles is the background. For a
complete example, see *note Sufi simulates a detection::.
To better understand the correlation between the mean (or background)
value and the noise standard deviation, let's use an analogy. Consider
the profile of your galaxy to be analogous to the profile of a ship that
is sailing in the sea. The height of the ship would therefore be
analogous to the maximum flux difference between your galaxy's minimum
and maximum values. Furthermore, let's take the depth of the sea to
represent the background value: a deeper sea, corresponds to a brighter
background. In this analogy, the "noise" would be the height of the
waves that surround the ship: in deeper waters, the waves would also be
taller (the square root of the mean depth at the ship's position).
If the ship is in deep waters, the height of waves are greater than
when the ship is near to the beach (at lower depths). Therefore, when
the ship is in the middle of the sea, there are high waves that are
capable of hiding a significant part of the ship from our perspective.
This corresponds to a brighter background value in astronomical images:
the resulting noise from that brighter background can completely wash
out the signal from a fainter galaxy, star or solar system object.
---------- Footnotes ----------
(1) In practice, we are actually counting the electrons that are
produced by each photon, not the actual photons.
(2) [From Wikipedia] Poisson's result was also derived in a previous
study by Abraham de Moivre in 1711. Therefore some people suggest it
should rightly be called the de Moivre distribution.
(3) In a real image, a relatively large number of very faint objects
can be fully buried in the noise and never detected. These undetected
objects will bias the background measurement to slightly larger values.
Our best approximation is thus to simply assume they are uniform, and
consider their average effect. See Figure 1 (a.1 and a.2) and Section
2.2 in Akhlaghi and Ichikawa 2015 (https://arxiv.org/abs/1505.01664).
6.2.3.2 Instrumental noise
..........................
While taking images with a camera, a bias current is fed to the pixels,
the variation of the value of this bias current over the pixels, also
adds to the final image noise. Another source of noise is the readout
noise that is produced by the electronics in the detector.
Specifically, the parts that attempt to digitize the voltage produced by
the photo-electrons in the analog to digital converter. With the
current generation of instruments, this source of noise is not as
significant as the noise due to the background Sky discussed in *note
Photon counting noise::.
Let $C$ represent the combined standard deviation of all these
instrumental sources of noise. When only this source of noise is
present, the noised pixel value would be a random value chosen from a
Gaussian distribution with
$$\mu=I_{nn}, \quad \sigma=\sqrt{C^2+I_{nn}}$$
This type of noise is independent of the signal in the dataset, it is
only determined by the instrument. So the flux scale (and not magnitude
scale) is most commonly used for this type of noise. In practice, this
value is usually reported in analog-to-digital units or ADUs, not flux
or electron counts. The gain value of the device can be used to convert
between these two, see *note Brightness flux magnitude::.
6.2.3.3 Final noised pixel value
................................
Based on the discussions in *note Photon counting noise:: and *note
Instrumental noise::, depending on the values you specify for $B$ and
$C$ from the above, the final noised value for each pixel is a random
value chosen from a Gaussian distribution with
$$\mu=B+I_{nn}, \quad \sigma=\sqrt{C^2+B+I_{nn}}$$
6.2.3.4 Generating random numbers
.................................
As discussed above, to generate noise we need to make random samples of
a particular distribution. So it is important to understand some
general concepts regarding the generation of random numbers. For a very
complete and nice introduction we strongly advise reading Donald Knuth's
"The art of computer programming", volume 2, chapter 3(1). Quoting from
the GNU Scientific Library manual, "If you do not own it, you should
stop reading right now, run to the nearest bookstore, and buy it"(2)!
Using only software, we can only produce what is called a
psuedo-random sequence of numbers. A true random number generator is a
hardware (let's assume we have made sure it has no systematic biases),
for example, throwing dice or flipping coins (which have remained from
the ancient times). More modern hardware methods use atmospheric noise,
thermal noise or other types of external electromagnetic or quantum
phenomena. All pseudo-random number generators (software) require a
seed to be the basis of the generation. The advantage of having a seed
is that if you specify the same seed for multiple runs, you will get an
identical sequence of random numbers which allows you to reproduce the
same final noised image.
The programs in GNU Astronomy Utilities (for example, MakeNoise or
MakeProfiles) use the GNU Scientific Library (GSL) to generate random
numbers. GSL allows the user to set the random number generator through
environment variables, see *note Installation directory:: for an
introduction to environment variables. In the chapter titled "Random
Number Generation" they have fully explained the various random number
generators that are available (there are a lot of them!). Through the
two environment variables ‘GSL_RNG_TYPE’ and ‘GSL_RNG_SEED’ you can
specify the generator and its seed respectively.
If you do not specify a value for ‘GSL_RNG_TYPE’, GSL will use its
default random number generator type. The default type is sufficient
for most general applications. If no value is given for the
‘GSL_RNG_SEED’ environment variable and you have asked Gnuastro to read
the seed from the environment (through the ‘--envseed’ option), then GSL
will use the default value of each generator to give identical outputs.
If you do not explicitly tell Gnuastro programs to read the seed value
from the environment variable, then they will use the system time
(accurate to within a microsecond) to generate (apparently random)
seeds. In this manner, every time you run the program, you will get a
different random number distribution.
There are two ways you can specify values for these environment
variables. You can call them on the same command-line for example:
$ GSL_RNG_TYPE="taus" GSL_RNG_SEED=345 astarithmetic input.fits \
mknoise-sigma \
--envseed
In this manner the values will only be used for this particular
execution of Arithmetic. However, it makes your code hard to read!
Alternatively, you can define them for the full period of your terminal
session or script, using the shell's ‘export’ command with the two
separate commands below (for a script remove the ‘$’ signs):
$ export GSL_RNG_TYPE="taus"
$ export GSL_RNG_SEED=345
The subsequent programs which use GSL's random number generators will
hence forth use these values in this session of the terminal you are
running or while executing this script. In case you want to set fixed
values for these parameters every time you use the GSL random number
generator, you can add these two lines to your ‘.bashrc’ startup
script(3), see *note Installation directory::.
*IMPORTANT NOTE:* If the two environment variables ‘GSL_RNG_TYPE’ and
‘GSL_RNG_SEED’ are defined, GSL will report them by default, even if you
do not use the ‘--envseed’ option. For example, see this call to
MakeProfiles:
$ export GSL_RNG_TYPE=taus
$ export GSL_RNG_SEED=345
$ astmkprof -s1 --kernel=gaussian,2,5
GSL_RNG_TYPE=taus
GSL_RNG_SEED=345
MakeProfiles V.VV started on DDD MMM DDD HH:MM:SS YYYY
- Building one gaussian kernel
- Random number generator (RNG) type: taus
- Basic RNG seed: 1618960836
---- ./kernel.fits created.
-- Output: ./kernel.fits
MakeProfiles finished in 0.068945 seconds
The first two output lines (showing the names and values of the GSL
environment variables) are printed by GSL before MakeProfiles actually
starts generating random numbers. Gnuastro's programs will report the
actual values they use independently (after the name of the program),
you should check them for the final values used, not GSL's printed
values. In the example above, did you notice how the random number
generator seed above is different between GSL and MakeProfiles?
However, if ‘--envseed’ was given, both printed seeds would be the same.
---------- Footnotes ----------
(1) Knuth, Donald. 1998. The art of computer programming.
Addison-Wesley. ISBN 0-201-89684-2
(2) For students, running to the library might be more affordable!
(3) Do not forget that if you are going to give your scripts (that
use the GSL random number generator) to others you have to make sure you
also tell them to set these environment variable separately. So for
scripts, it is best to keep all such variable definitions within the
script, even if they are within your ‘.bashrc’.
6.2.4 Arithmetic operators
--------------------------
In this section, list of recognized operators in Arithmetic (and the
Table program's *note Column arithmetic::) and discussed in detail with
examples. As mentioned before, to be able to easily do complex
operations on the command-line, the Reverse Polish Notation is used
(where you write '$4\quad5\quad+$' instead of '$4 + 5$'), if you are not
already familiar with it, before continuing, please see *note Reverse
polish notation::.
The operands to all operators can be a data array (for example, a
FITS image or data cube) or a number, the output will be an array or
number according to the inputs. For example, a number multiplied by an
array will produce an array. The numerical data type of the output of
each operator is described within it. Here are some generic tips and
tricks (relevant to all operators):
Multiple operators in one command
When you need to use arithmetic commands in several consecutive
operations, you can use one command instead of multiple commands
and perform all calculations in the same command. For example,
assume you want to apply a threshold of 10 on your image, and label
the connected groups of pixel above this threshold. You need two
operators for this: ‘gt’ (for "greater than", see *note Conditional
operators::) and ‘connected-components’ (see *note Mathematical
morphology operators::). The bad (non-optimized and slow) way of
doing this is to call Arithmetic two times:
$ astarithmetic image.fits 10 gt --output=thresh.fits
$ astarithmetic thresh.fits 2 connected-components \
--output=labeled.fits
$ rm thresh.fits
The good (optimal) way is to call them after each other (remember
*note Reverse polish notation::):
$ astarithmetic image.fits 10 gt 2 connected-components \
--output=labeled.fits
You can similarly add any number of operations that must be done
sequentially in a single command and benefit from the speed and
lack of intermediate files. When your commands become long, you
can use the ‘set-AAA’ operator to make it more readable, see *note
Operand storage in memory or a file::.
Blank pixels in Arithmetic
Blank pixels in the image (see *note Blank pixels::) will be stored
based on the data type. When the input is floating point type,
blank values are NaN. One aspect of NaN values is that by
definition they will fail on _any_ comparison. Also, any operator
that includes a NaN as a an operand will produce a NaN
(irrespective of its other operands). Hence both equal and
not-equal operators will fail when both their operands are NaN!
Therefore, the only way to guarantee selection of blank pixels is
through the ‘isblank’ operator explained above.
One way you can exploit this property of the NaN value to your
advantage is when you want a fully zero-valued image (even over the
blank pixels) based on an already existing image (with same size
and world coordinate system settings). The following command will
produce this for you:
$ astarithmetic input.fits nan eq --output=all-zeros.fits
Note that on the command-line you can write NaN in any case (for
example, ‘NaN’, or ‘NAN’ are also acceptable). Reading NaN as a
floating point number in Gnuastro is not case-sensitive.
6.2.4.1 Basic mathematical operators
....................................
These are some of the most common operations you will be doing on your
data and include, so no further explanation is necessary. If you are
new to Gnuastro, just read the description of each carefully.
‘+’
Addition, so "‘4 5 +’" is equivalent to $4+5$. For example, in the
command below, the value 20000 is added to each pixel's value in
‘image.fits’:
$ astarithmetic 20000 image.fits +
You can also use this operator to sum the values of one pixel in
two images (which have to be the same size). For example, in the
commands below (which are identical, see paragraph after the
commands), each pixel of ‘sum.fits’ is the sum of the same pixel's
values in ‘a.fits’ and ‘b.fits’.
$ astarithmetic a.fits b.fits + -h1 -h1 --output=sum.fits
$ astarithmetic a.fits b.fits + -g1 --output=sum.fits
The HDU/extension has to be specified for each image with ‘-h’.
However, if the HDUs are the same in all inputs, you can use ‘-g’
to only specify the HDU once
If you need to add more than one dataset, one way is to use this
operator multiple times, for example, see the two commands below
that are identical in the Reverse Polish Notation (*note Reverse
polish notation::):
$ astarithmetic a.fits b.fits + c.fits + -osum.fits
$ astarithmetic a.fits b.fits c.fits + + -osum.fits
However, this can get annoying/buggy if you have more than three or
four images, in that case, a better way to sum data is to use the
‘sum’ operator (which also ignores blank pixels), that is discussed
in *note Stacking operators::.
*NaN values:* if a single argument of ‘+’ has a NaN value, the
output will also be NaN. To ignore NaN values, use the ‘sum’
operator of *note Stacking operators::. You can see the difference
with the two commands below:
$ astarithmetic --quiet 1.0 2.0 3.0 nan + + +
nan
$ astarithmetic --quiet 1.0 2.0 3.0 nan 4 sum
6.000000e+00
The same goes for all the *note Stacking operators:: so if your
data may include NaN pixels, be sure to use the stacking operators.
‘-’
Subtraction, so "‘4 5 -’" is equivalent to $4-5$. Usage of this
operator is similar to ‘+’ operator, for example:
$ astarithmetic 20000 image.fits -
$ astarithmetic a.fits b.fits - -g1 --output=sub.fits
‘x’
Multiplication, so "‘4 5 x’" is equivalent to $4\times5$. For
example, in the command below, the value of each output pixel is 5
times its value in ‘image.fits’:
$ astarithmetic image.fits 5 x
And you can multiply the value of each pixel in two images, like
this:
$ astarithmetic a.fits a.fits x -g1 –output=multip.fits
‘/’
Division, so "‘4 5 /’" is equivalent to $4/5$. Like the
multiplication, for example
$ astarithmetic image.fits 5 -h1 /
$ astarithmetic a.fits b.fits / -g1 –output=div.fits
‘%’
Modulo (remainder), so "‘3 2 %’" will return $1$. Note that the
modulo operator only works on integer types (see *note Numeric data
types::). This operator is therefore not defined for most
processed astronomical astronomical images that have floating-point
value. However it is useful in labeled images, for example, *note
Segment output::). In such cases, each pixel is the integer label
of the object it is associated with hence with the example command
below, we can change the labels to only be between 1 and 4 and
decrease all objects on the image to 4/5th (all objects with a
label that is a multiple of 5 will be set to 0).
$ astarithmetic label.fits 5 1 %
‘abs’
Absolute value of first operand, so "‘4 abs’" is equivalent to
$|4|$. For example, the output of the command bellow will not have
any negative pixels (all negative pixels will be multiplied by $-1$
to become positive)
$ astarithmetic image.fits abs
‘pow’
First operand to the power of the second, so "‘4.3 5 pow’" is
equivalent to $4.3^{5}$. For example, with the command below all
pixels will be squared
$ astarithmetic image.fits 2 pow
‘sqrt’
The square root of the first operand, so "‘5 sqrt’" is equivalent
to $\sqrt{5}$. Since the square root is only defined for positive
values, any negative-valued pixel will become NaN (blank). The
output will have a floating point type, but its precision is
determined from the input: if the input is a 64-bit floating point,
the output will also be 64-bit. Otherwise, the output will be
32-bit floating point (see *note Numeric data types:: for the
respective precision). Therefore if you require 64-bit precision
in estimating the square root, convert the input to 64-bit floating
point first, for example, with ‘5 float64 sqrt’. For example, each
pixel of the output of the command below will be the square root of
that pixel in the input.
$ astarithmetic image.fits sqrt
If you just want to scale an image with negative values using this
operator (for better visual inspection, and the actual values do
not matter for you), you can subtract the image from its minimum
value, then take its square root:
$ astarithmetic image.fits image.fits minvalue - sqrt -g1
Alternatively, to avoid reading the image into memory two times,
you can use the ‘set-’ operator to read it into the variable ‘i’
and use ‘i’ two times to speed up the operation (described below):
$ astarithmetic image.fits set-i i i minvalue - sqrt
‘log’
Natural logarithm of first operand, so "‘4 log’" is equivalent to
$ln(4)$. Negative pixels will become NaN, and the output type is
determined from the input, see the explanation under ‘sqrt’ for
more on these features. For example, the command below will take
the natural logarithm of every pixel in the input.
$ astarithmetic image.fits log --output=log.fits
‘log10’
Base-10 logarithm of first popped operand, so "‘4 log’" is
equivalent to $log_{10}(4)$. Negative pixels will become NaN, and
the output type is determined from the input, see the explanation
under ‘sqrt’ for more on these features. For example, the command
below will take the base-10 logarithm of every pixel in the input.
$ astarithmetic image.fits log10
6.2.4.2 Trigonometric and hyperbolic operators
..............................................
All the trigonometric and hyperbolic functions are described here. One
good thing with these operators is that they take inputs and outputs in
degrees (which we usually need as input or output), not radians (like
most other programs/libraries).
‘sin’
‘cos’
‘tan’
Basic trigonometric functions. They take one operand, in units of
degrees.
‘asin’
‘acos’
‘atan’
Inverse trigonometric functions. They take one operand and the
returned values are in units of degrees.
‘atan2’
Inverse tangent (output in units of degrees) that uses the signs of
the input coordinates to distinguish between the quadrants. This
operator therefore needs two operands: the first popped operand is
assumed to be the X axis position of the point, and the second
popped operand is its Y axis coordinate.
For example, see the commands below. To be more clear, we are
using Table's *note Column arithmetic:: which uses exactly the same
internal library function as the Arithmetic program for images. We
are showing the results for four points in the four quadrants of
the 2D space (if you want to try running them, you do not need to
type/copy the parts after <#>). The first point (2,2) is in the
first quadrant, therefore the returned angle is 45 degrees. But
the second, third and fourth points are in the quadrants of the
same order, and the returned angles reflect the quadrant.
$ echo " 2 2" | asttable -c'arith $2 $1 atan2' # --> 45
$ echo " 2 -2" | asttable -c'arith $2 $1 atan2' # --> -45
$ echo "-2 -2" | asttable -c'arith $2 $1 atan2' # --> -135
$ echo "-2 2" | asttable -c'arith $2 $1 atan2' # --> 135
However, if you simply use the classic arc-tangent operator
(‘atan’) for the same points, the result will only be in two
quadrants as you see below:
$ echo " 2 2" | asttable -c'arith $2 $1 / atan' # --> 45
$ echo " 2 -2" | asttable -c'arith $2 $1 / atan' # --> -45
$ echo "-2 -2" | asttable -c'arith $2 $1 / atan' # --> 45
$ echo "-2 2" | asttable -c'arith $2 $1 / atan' # --> -45
‘sinh’
‘cosh’
‘tanh’
Hyperbolic sine, cosine, and tangent. These operators take a
single operand.
‘asinh’
‘acosh’
‘atanh’
Inverse Hyperbolic sine, cosine, and tangent. These operators take
a single operand.
6.2.4.3 Constants
.................
During your analysis it is often necessary to have certain constants
like the number $\pi$. The "operators" in this section do not actually
take any operand, they just replace the desired constant into the stack.
So in effect, these are actually operands. But since their value is not
inserted by the user, we have placed them in the list of operators.
‘e’
Euler’s number, or the base of the natural logarithm (no units).
See Wikipedia
(https://en.wikipedia.org/wiki/E_(mathematical_constant)).
‘pi’
Ratio of circle’s circumference to its diameter (no units). See
Wikipedia (https://en.wikipedia.org/wiki/Pi).
‘c’
The speed of light in vacuum, in units of $m/s$. see Wikipedia
(https://en.wikipedia.org/wiki/Speed_of_light).
‘G’
The gravitational constant, in units of $m^3/kg/s^2$. See
Wikipedia (https://en.wikipedia.org/wiki/Gravitational_constant).
‘h’
Plank's constant, in units of $J/Hz$ or $kg\times m^2/s$. See
Wikipedia (https://en.wikipedia.org/wiki/Planck_constant).
‘au’
Astronomical Unit, in units of meters. See Wikipedia
(https://en.wikipedia.org/wiki/Astronomical_unit).
‘ly’
Distance covered by light in vacuum in one year, in units of
meters. See Wikipedia (https://en.wikipedia.org/wiki/Light-year).
‘avogadro’
Avogadro's constant, in units of $1/mol$. See Wikipedia
(https://en.wikipedia.org/wiki/Avogadro_constant).
‘fine-structure’
The fine-structure constant (no units). See Wikipedia
(https://en.wikipedia.org/wiki/Fine-structure_constant).
6.2.4.4 Coordinate conversion operators
.......................................
Different celestial coordinate systems are useful for different
scenarios. For example, assume you have the RA and Dec of large sample
of galaxies that you plan to study the halos of galaxies from. For such
studies, you prefer to stay as far away as possible from the Galactic
plane, because the density of stars and interstellar filaments (cirrus)
significantly increases as you get close to the Milky way's disk. But
the Equatorial coordinate system
(https://en.wikipedia.org/wiki/Equatorial_coordinate_system) which
defines the RA and Dec and is based on Earth's equator; and does not
show the position of your objects in relation to the galactic disk.
The best way forward in the example above is to convert your RA and
Dec table into the Galactic coordinate system
(https://en.wikipedia.org/wiki/Galactic_coordinate_system); and select
those with a large (positive or negative) Galactic latitude.
Alternatively, if you observe a bright point on a galaxy and want to
confirm if it was actually a super-nova and not a moving asteroid, a
first step is to convert your RA and Dec to the Ecliptic coordinate
system (https://en.wikipedia.org/wiki/Ecliptic_coordinate_system) and
confirm if you are sufficiently distant from the ecliptic (plane of the
Solar System; where fast moving objects are most common).
The operators described in this section are precisely for the purpose
above: to convert various celestial coordinate systems that are
supported within Gnuastro into each other. For example, if you want to
convert the RA and Dec equatorial (at the Julian year 2000 equinox)
coordinates (within the ‘RA’ and ‘DEC’ columns) of ‘points.fits’ into
Galactic longitude and latitude, you can use the command below (the
column metadata are not mandatory, but to avoid later confusion, it is
always good to have them in your output.
$ asttable points.fits -c'arith RA DEC eq-j2000-to-galactic' \
--colmetadata=1,GLON,deg,"Galactic longitude" \
--colmetadata=2,GLAT,deg,"Galactic latitude" \
--output=points-gal.fits
One important thing to consider is that the equatorial and ecliptic
coordinates are not static: they include the dynamics of Earth in the
solar system: in particular, the reference point on the equator moves
over decades. Therefore these two (equatorial and ecliptic) coordinate
systems are defined within epochs: the 1950 epoch is defined by
Besselian years
(https://en.wikipedia.org/wiki/Epoch_(astronomy)#Besselian_years), while
the 2000 epoch is defined in Julian years
(https://en.wikipedia.org/wiki/Epoch_(astronomy)#Julian_years_and_J2000).
So when dealing with these coordinates one of the '‘-b1950’' or
'‘-j2000’' suffixes are necessary (for example ‘eq-j2000’ or
‘ec-b1950’).
The Galactic or Supergalactic coordinates are not defined based on
the Earth's dynamics; therefore they do not have any epoch associated
with them. Extra-galactic studies do not depend on the dynamics of the
earth, but the equatorial coordinate system is the most dominant in that
field. Therefore in its 23rd General Assembly, the International
Astronomical Union approved the International Celestial Reference System
(https://en.wikipedia.org/wiki/International_Celestial_Reference_System_and_its_realizations)
or ICRS based on quasars (which are static within our observational
limitations)viewed through long baseline radio interferometry (the most
accurate method of observation that we currently have). ICRS is
designed to be within the errors of the Equatorial J2000 coordinate
system, so they are currently very similar; but ICRS has much better
accuracy. We will be adding ICRS in the operators below soon.
*Floating point errors:* The operation to convert between the
coordinate systems involves many sines, cosines (and their inverse).
Therefore, floating point errors (due to the limited precision of the
definition of floating points in bits) can cause small offsets. For
example see the code below were we convert equatorial to galactic and
back, then compare the input and output (which is in the 5th and 6th
decimal of a degree; or about 0.2 or 0.01 arcseconds).
$ sys1=eq-j2000
$ sys2=galactic
$ echo "10.2345689 45.6789012" \
| asttable -Afixed -B8 \
-c'arith $1 $2 '$sys1'-to-'$sys2' \
'$sys2'-to-'$sys1' set-lat set-lng \
lng $1 - lat $2 -'
0.00000363 -0.00007725
If you set ‘sys2=ec-j2000’ or ‘sys2=supergalactic’, it will be zero
until the full set of 8 decimals that are printed here (the displayed
precision can be changed with the value of the ‘-B’ option above). It
is therefore useful to have your original coordinates (in the same table
for example) and not do too many conversions on conversions (to
propagate this problem).
‘eq-b1950-to-eq-j2000’
‘eq-b1950-to-ec-b1950’
‘eq-b1950-to-ec-j2000’
‘eq-b1950-to-galactic’
‘eq-b1950-to-supergalactic’
Convert Equatorial (B1950 equinox) coordinates into the respective
coordinate system within each operator's name.
‘eq-j2000-to-eq-b1950’
‘eq-j2000-to-ec-b1950’
‘eq-j2000-to-ec-j2000’
‘eq-j2000-to-galactic’
‘eq-j2000-to-supergalactic’
Convert Equatorial (J2000 equinox) coordinates into the respective
coordinate system within each operator's name.
‘ec-b1950-to-eq-b1950’
‘ec-b1950-to-eq-j2000’
‘ec-b1950-to-ec-j2000’
‘ec-b1950-to-galactic’
‘ec-b1950-to-supergalactic’
Convert Ecliptic (B1950 equinox) coordinates into the respective
coordinate system within each operator's name.
‘ec-j2000-to-eq-b1950’
‘ec-j2000-to-eq-j2000’
‘ec-j2000-to-ec-b1950’
‘ec-j2000-to-galactic’
‘ec-j2000-to-supergalactic’
Convert Ecliptic (J2000 equinox) coordinates into the respective
coordinate system within each operator's name.
‘galactic-to-eq-b1950’
‘galactic-to-eq-j2000’
‘galactic-to-ec-b1950’
‘galactic-to-ec-j2000’
‘galactic-to-supergalactic’
Convert Galactic coordinates into the respective coordinate system
within each operator's name.
‘supergalactic-to-eq-b1950’
‘supergalactic-to-eq-j2000’
‘supergalactic-to-ec-b1950’
‘supergalactic-to-ec-j2000’
‘supergalactic-to-galactic’
Convert Supergalactic coordinates into the respective coordinate
system within each operator's name.
6.2.4.5 Unit conversion operators
.................................
It often happens that you have data in one unit (for example, counts on
your CCD), but would like to convert it into another (for example,
magnitudes, to measure the brightness of a galaxy). While the equations
for the unit conversions can be easily found on the internet, the
operators in this section are designed to simplify the process and let
you do it easily and fast without having to remember constants and
relations.
‘counts-to-mag’
Convert counts (usually CCD outputs) to magnitudes using the given
zero point. The zero point is the first popped operand and the
count image or value is the second popped operand.
For example, assume you have measured the standard deviation of the
noise in an image to be ‘0.1’ counts, and the image's zero point is
‘22.5’ and you want to measure the _per-pixel_ surface brightness
limit of the dataset(1). To apply this operator on an image,
simply replace ‘0.1’ with the image name, as described below.
$ astarithmetic 0.1 22.5 counts-to-mag --quiet
Of course, you can also convert every pixel in an image (or table
column in Table's *note Column arithmetic::) with this operator if
you replace the second popped operand with an image/column name.
For an example of applying this operator on an image, see the
description of surface brightness in *note Brightness flux
magnitude::, where we will convert an image's pixel values to
surface brightness.
‘mag-to-counts’
Convert magnitudes to counts (usually CCD outputs) using the given
zero point. The zero point is the first popped operand and the
magnitude value is the second. For example, if an object has a
magnitude of 20, you can estimate the counts corresponding to it
(when the image has a zero point of 24.8) with this command: Note
that because the output is a single number, we are using ‘--quiet’
to avoid printing extra information.
$ astarithmetic 20 24.8 mag-to-counts --quiet
‘counts-to-sb’
Convert counts to surface brightness using the zero point and area
(in units of arcsec$^2$). The first popped operand is the area (in
arcsec$^2$), the second popped operand is the zero point and the
third are the count values. Estimating the surface brightness
involves taking the logarithm. Therefore this operator will
produce NaN for counts with a negative value.
For example, with the commands below, we read the zero point from
the image headers (assuming it is in the ‘ZPOINT’ keyword), we
calculate the pixel area from the image itself, and we call this
operator to convert the image pixels (in counts) to surface
brightness (mag/arcsec$^2$).
$ zeropoint=$(astfits image.fits --keyvalue=ZPOINT -q)
$ pixarea=$(astfits image.fits --pixelareaarcsec2)
$ astarithmetic image.fits $zeropoint $pixarea counts-to-sb \
--output=image-sb.fits
For more on the definition of surface brightness see *note
Brightness flux magnitude::, and for a fully tutorial on optimal
usage of this, see *note FITS images in a publication::.
‘sb-to-counts’
Convert surface brightness using the zero point and area (in units
of arcsec$^2$) to counts. The first popped operand is the area (in
arcsec$^2$), the second popped operand is the zero point and the
third are the surface brightness values. See the description of
‘counts-to-sb’ for more.
‘mag-to-sb’
Convert magnitudes to surface brightness over a certain area (in
units of arcsec$^2$). The first popped operand is the area and the
second is the magnitude. For example, let's assume you have a
table with the two columns of magnitude (called ‘MAG’) and area
(called ‘AREAARCSEC2’). In the command below, we will use *note
Column arithmetic:: to return the surface brightness.
$ asttable table.fits -c'arith MAG AREAARCSEC2 mag-to-sb'
‘sb-to-mag’
Convert surface brightness to magnitudes over a certain area (in
units of arcsec$^2$). The first popped operand is the area and the
second is the magnitude. See the description of ‘mag-to-sb’ for
more.
‘counts-to-jy’
Convert counts (usually CCD outputs) to Janskys through an
AB-magnitude based zero point. The top-popped operand is assumed
to be the AB-magnitude zero point and the second-popped operand is
assumed to be a dataset in units of counts (an image in Arithmetic,
and a column in Table's *note Column arithmetic::). For the full
equation and basic definitions, see *note Brightness flux
magnitude::.
For example, SDSS images are calibrated in units of nanomaggies,
with a fixed zero point magnitude of 22.5. Therefore you can
convert the units of SDSS image pixels to Janskys with the command
below:
$ astarithmetic sdss-image.fits 22.5 counts-to-jy
‘jy-to-counts’
Convert Janskys to counts (usually CCD outputs) through an
AB-magnitude based zero point. This is the inverse operation of
the ‘counts-to-jy’, see there for usage example.
‘zeropoint-change’
Change the zero point of the input dataset to a new zero point
(output will always be 64-bit floating point). For example, with
the command below, we are changing the zero point of an image from
26.892 to 27.0:
$ astarithmetic image.fits 26.892 27.0 zeropoint-change
Note that the input or output zero points can also be an image
(with the same size as the input image). This is very useful to
correct situations where the zero point can change over the image
(happens in single exposures) and you want to unify it across the
whole image (to build a deep stacked image).
‘counts-to-nanomaggy’
Convert counts to Nanomaggy (with fixed zero point of 22.5, used as
the pixel units of many surveys like SDSS). For example if your
image has a zero point of 24.93, you can convert it to Nanomaggies
with the command below:
$ astarithmetic image.fits 24.93 counts-to-nanomaggy
This is just a wrapper over the ‘zeropoint-change’ operator where
the output zero point is 22.5.
‘nanomaggy-to-counts’
Convert Nanomaggy to counts. Nanomaggy is defined to have a fixed
zero point of 22.5 and is the pixel units of many surveys like
SDSS. For example if you would like to convert an image in units of
Nanomaggy (for example from SDSS) to the counts of a camera with a
zero point of 25.92, you can use the command below:
$ astarithmetic image.fits 25.92 nanomaggy-to-counts
This is just a wrapper over the ‘zeropoint-change’ operator where
the input zero point is 22.5.
‘mag-to-jy’
Convert AB magnitudes to Janskys, see *note Brightness flux
magnitude::.
‘jy-to-mag’
Convert Janskys to AB magnitude, see *note Brightness flux
magnitude::.
‘jy-to-wavelength-flux-density’
Convert Janskys to wavelength flux density (in units of
$erg/cm^2/s/\AA$) at a certain wavelength (given in units of
Angstroms). Recall that Jansky is also a unit of flux density, but
it is in units of frequency ($erg/cm^2/s/Hz$).
For example at the wavelength of 5556, Vega's frequency flux
density is 3500 Janskys. To convert this to wavelength flux
density, you can use this command:
$ astarithmetic 3.5e3 5556 jy-to-wavelength-flux-density
If your input values are in units of counts or magnitudes, you can
use the ‘mag-to-jy’ or ‘count-to-jy’ operators that are described
above. For example, if you want the wavelength flux density of a
source with magnitude 21 in the J-PAS J0660 filter (centered at
approximately $6600\AA$), you can use this command:
$ astarithmetic 21 mag-to-jy 6600 \
jy-to-wavelength-flux-density
‘au-to-pc’
Convert Astronomical Units (AUs) to Parsecs (PCs). This operator
takes a single argument which is interpreted to be the input AUs.
The conversion is based on the definition of Parsecs: $1 \rm{PC} =
1/tan(1^{\prime\prime}) \rm{AU}$, where $1^{\prime\prime}$ is one
arcseconds. In other words, $1 (\rm{PC}) = 648000/\pi (\rm{AU})$.
For example, if we take Pluto's average distance to the Sun to be
40 AUs, we can obtain its distance in Parsecs using this command:
echo 40 | asttable -c'arith $1 au-to-pc'
‘pc-to-au’
Convert Parsecs (PCs) to Astronomical Units (AUs). This operator
takes a single argument which is interpreted to be the input PCs.
For more on the conversion equation, see description of ‘au-to-pc’.
For example, Proxima Centauri (the nearest star to the Solar
system) is 1.3020 Parsecs from the Sun, we can calculate this
distance in units of AUs with the command below:
echo 1.3020 | asttable -c'arith $1 pc-to-au'
‘ly-to-pc’
Convert Light-years (LY) to Parsecs (PCs). This operator takes a
single argument which is interpreted to be the input LYs. The
conversion is done from IAU's definition of the light-year
(9460730472580800 m $\approx$ 63241.077 AU = 0.306601 PC, for the
conversion of AU to PC, see the description of ‘au-to-pc’).
For example, the distance of Andromeda galaxy to our galaxy is 2.5
million light-years, so its distance in kilo-Parsecs can be
calculated with the command below (note that we want the output in
kilo-parsecs, so we are dividing the output of this operator by
1000):
echo 2.5e6 | asttable -c'arith $1 ly-to-pc 1000 /'
‘pc-to-ly’
Convert Parsecs (PCs) to Light-years (LY). This operator takes a
single argument which is interpreted to be the input PCs. For the
conversion and an example of the inverse of this operator, see the
description of ‘ly-to-pc’.
‘ly-to-au’
Convert Light-years (LY) to Astronomical Units (AUs). This
operator takes a single argument which is interpreted to be the
input LYs. For the conversion and a similar example, see the
description of ‘ly-to-pc’.
‘au-to-ly’
Convert Astronomical Units (AUs) to Light-years (LY). This operator
takes a single argument which is interpreted to be the input AUs.
For the conversion and a similar example, see the description of
‘ly-to-pc’.
---------- Footnotes ----------
(1) The _per-pixel_ surface brightness limit is the magnitude of the
noise standard deviation. For more on surface brightness see *note
Brightness flux magnitude::. In the example command, because the output
is a single number, we are using ‘--quiet’ to avoid printing extra
information.
6.2.4.6 Statistical operators
.............................
The operators in this section take a single dataset as input, and will
return the desired statistic as a single value.
‘minvalue’
Minimum value in the first popped operand, so "‘a.fits minvalue’"
will push the minimum pixel value in this image onto the stack.
When this operator acts on a single image, the output (operand that
is put back on the stack) will no longer be an image, but a number.
The output of this operand is in the same type as the input. This
operator is mainly intended for multi-element datasets (for
example, images or data cubes), if the popped operand is a number,
it will just return it without any change.
Note that when the final remaining/output operand is a single
number, it is printed onto the standard output. For example, with
the command below the minimum pixel value in ‘image.fits’ will be
printed in the terminal:
$ astarithmetic image.fits minvalue
However, the output above also includes a lot of extra information
that are not relevant in this context. If you just want the final
number, run Arithmetic in quiet mode:
$ astarithmetic image.fits minvalue -q
Also see the description of ‘sqrt’ for other example usages of this
operator.
‘maxvalue’
Maximum value of first operand in the same type, similar to
‘minvalue’, see the description there for more. For example
$ astarithmetic image.fits maxvalue -q
‘numbervalue’
Number of non-blank elements in first operand in the ‘uint64’ type
(since it is always a positive integer, see *note Numeric data
types::). Its usage is similar to ‘minvalue’, for example
$ astarithmetic image.fits numbervalue -q
‘sumvalue’
Sum of non-blank elements in first operand in the ‘float32’ type.
Its usage is similar to ‘minvalue’, for example
$ astarithmetic image.fits sumvalue -q
‘meanvalue’
Mean value of non-blank elements in first operand in the ‘float32’
type. Its usage is similar to ‘minvalue’, for example
$ astarithmetic image.fits meanvalue -q
‘stdvalue’
Standard deviation of non-blank elements in first operand in the
‘float32’ type. Its usage is similar to ‘minvalue’, for example
$ astarithmetic image.fits stdvalue -q
‘medianvalue’
Median of non-blank elements in first operand with the same type.
Its usage is similar to ‘minvalue’, for example
$ astarithmetic image.fits medianvalue -q
‘madclip-maskfilled’
‘sigclip-maskfilled’
Mask (set to blank/NaN) all the outlying elements (defined by
$\sigma$ or MAD clipping) in the inputs and put all the inputs back
on the stack. The first popped operand is the termination criteria
of the clipping, the second popped operand is the multiple of
$\sigma$ or MAD and the third is the number of input datasets that
will be popped for the actual operation. If you are not yet
familiar with $\sigma$ or MAD clipping, it is recommended to read
this tutorial: *note Clipping outliers::.
When more than 95$\%$ of the area of an operand is masked, the full
operand will be masked. This is necessary in like this: one of
your inputs has many outliers (for example it is much more noisy
than the rest or its sky level has not been subtracted properly).
Because this operator fills holes between outlying pixels, most of
the area of the input will be masked, but the thin edges (where
there are no "holes") will remain, causing different statistics in
those thin edges of that input in your final stack. Through this
mask coverage fraction (which is currently hard-coded(1)), we
ensure that such thin edges do not cause artifacts in the final
stack.
For example, with the second command below, we are masking the MAD
clipped pixels of the 9 inputs (that are generated in *note
Clipping outliers::) and writing them as separate HDUs of the
output. The clipping is done with 5 times the MAD and the clipping
starts when the relative difference between subsequent MADs is
0.01. Finally, with the third command, we see 10 HDUs in the
output (because the first, or 0-th, is just metadata).
$ ls in-*.fits
in-1.fits in-3.fits in-5.fits in-7.fits in-9.fits
in-2.fits in-4.fits in-6.fits in-8.fits
$ astarithmetic in-*.fits 9 5 0.01 madclip-maskfilled \
-g1 --writeall --output=clipped.fits
$ astfits clipped.fits --numhdus
10
In the Arithmetic command above, ‘--writeall’ is necessary because
this operator puts all its inputs back on the stack of operands.
This is because these are usually just intermediate operators. For
example, after masking the outliers from each input, you may want
to stack them into one deeper image (with the *note Stacking
operators::. After the stacking is done, only one operand will be
on the stack, and ‘--writeall’ will no longer be necessary. For
example if you want to see how many images were used in the final
stack's pixels, you can use the ‘number’ operator like below:
$ astarithmetic in-*.fits 9 5 0.01 madclip-maskfilled \
9 number -g1 --output=num-good.fits
‘unique’
Remove all duplicate (and blank) elements from the first popped
operand. The unique elements of the dataset will be stored in a
single-dimensional dataset.
Recall that by default, single-dimensional datasets are stored as a
table column in the output. But you can use ‘--onedasimage’ or
‘--onedonstdout’ to respectively store them as a single-dimensional
FITS array/image, or to print them on the standard output.
Although you can use this operator on the floating point dataset,
due to floating-point errors it may give non-reasonable values:
because the tenth digit of the decimal point is also considered
although it may be statistically meaningless, see *note Numeric
data types::. It is therefore better/recommended to use it on the
integer dataset like the labeled images of *note Segment output::
where each pixel has the integer label of the object/clump it is
associated with. For example, let's assume you have cropped a
region of a larger labeled image and want to find the
labels/objects that are within the crop. With this operator, this
job is trivial:
$ astarithmetic seg-crop.fits unique
‘noblank’
Remove all blank elements from the first popped operand. Since the
blank pixels are being removed, the output dataset will always be
single-dimensional, independent of the dimensionality of the input.
Recall that by default, single-dimensional datasets are stored as a
table column in the output. But you can use ‘--onedasimage’ or
‘--onedonstdout’ to respectively store them as a single-dimensional
FITS array/image, or to print them on the standard output.
For example, with the command below, the non-blank pixel values of
‘cropped.fits’ are printed on the command-line (the ‘--quiet’
option is used to remove the extra information that Arithmetic
prints as it reads the inputs, its version and its running time).
$ astarithmetic cropped.fits noblank --onedonstdout --quiet
---------- Footnotes ----------
(1) Please get in touch with us at ‘bug-gnuastro@gnu.org’ if you
notice this problem and feel the fraction needs to be lowered (or
generally to be set in each run).
6.2.4.7 Stacking operators
..........................
The operators in this section are used when you have multiple datasets
that you would like to merge into one, commonly known as "stacking" or
"coaddition". For example, you have taken ten exposures of your
scientific target, and you would like to combine them all into one deep
stacked image that is deeper.
*Masking outliers (before stacking):* Outliers in one of the inputs (for
example star ghosts, satellite trails, or cosmic rays, can leave their
imprints in the final stack. One good way to remove them is the
‘madclip-maskfilled’ operator that can be called before the operators
here. It is described in *note Statistical operators::; and a full
tutorial on understanding outliers and how best to remove them is
available in *note Clipping outliers::.
*Hundreds or thousands of images to stack:* It can happen that you need
to stack hundreds or thousands of images. Added with the possibly long
file/directory names, this can lead to an extremely long shell command
that may cause an "Argument list too long" error in your shell. To
avoid this, you should use Arithmetic's ‘--arguments’ option, see *note
Invoking astarithmetic::.
When calling the stacking operators you should determine how many
operands they should take in: unlike the rest of the operators that have
a fixed number of input operands, these operators have a variable number
of input operators. As described in the first operand below, you do
this through their first (or early) popped operands (which should be a
single integer number that is larger than one). Below are Some
important points for all the stacking operators described in this
section:
• NaN/blank pixels will be ignored, see *note Blank pixels::.
• The operation will be multi-threaded, greatly speeding up the
process if you have large and numerous data to stack. You can
disable multi-threaded operations with the ‘--numthreads=1’ option
(see *note Multi-threaded operations::).
‘min’
‘max’
‘sum’
‘std’
‘mad’
‘mean’
‘median’
‘number’
For each pixel, calculate the respective statistic from in all
given datasets. For the ‘min’ and ‘max’ operators, the output will
have the same type as the input. For the ‘number’ operator, the
output will have an unsigned 32-bit integer type and the rest will
be 32-bit floating point.
The first popped operand to this operator must be a positive
integer number which specifies how many further operands should be
popped from the stack. All the subsequently popped operands must
have the same type and size. This operator (and all the
variable-operand operators similar to it that are discussed below)
will work in multi-threaded mode unless Arithmetic is called with
the ‘--numthreads=1’ option, see *note Multi-threaded operations::.
For example, the following command will produce an image with the
same size and type as the three inputs, but each output pixel value
will be the minimum of the same pixel's values in all three input
images.
$ astarithmetic a.fits b.fits c.fits 3 min --output=min.fits
Regarding the ‘number’ operator: some datasets may have blank
values (which are also ignored in all similar operators like ‘min’,
‘sum’, ‘mean’ or ‘median’). Hence, the final pixel values of this
operator will not, in general, be equal to the number of inputs.
This operator is therefore mostly called in parallel with those
operators to know the "weight" of each pixel (in case you want to
only keep pixels that had the full exposure for example).
‘quantile’
For each pixel, find the quantile from all given datasets. The
output will have the same numeric data type and size as the input
datasets. Besides the input datasets, the quantile operator also
needs a single parameter (the requested quantile). The parameter
should be the first popped operand, with a value between (and
including) 0 and 1. The second popped operand must be the number
of datasets to use.
In the example below, the first-popped operand (‘0.7’) is the
quantile, the second-popped operand (‘3’) is the number of datasets
to pop.
astarithmetic a.fits b.fits c.fits 3 0.7 quantile
‘sigclip-mad’
‘sigclip-std’
‘sigclip-mean’
‘sigclip-median’
Return the respective statistic after $\sigma$-clipping the values
of the same pixel of all the input operands. The respective
statistic will be stored in a 32-bit floating point number. The
number of inputs used to make the desired measurement for each
pixel is also returned as a second output operand; see below for
more on how to deal with the second output operand.
For a complete tutorial on clipping outliers when stacking images
see *note Clipping outliers:: (if you haven't read it yet, we
encourage you to read through it before continuing). In
particular, the most robust solution is to first use
‘madclip-maskfilled’ (described in *note Statistical operators::),
then use any of these.
This operator is very similar to ‘min’, with the exception that it
expects two extra operands (parameters for MAD-clipping) before the
total number of inputs. The first popped operand is the
termination criteria and the second is the multiple of the median
absolute deviation.
For example, in the command below, the first popped operand of
‘sigclip-mean’ (‘0.1’) is the $\sigma$-clipping termination
criteria. If the termination criteria is larger than, or equal to
1, it is interpreted as the total number of clips. But if it is
between 0 and 1, then it is the tolerance level on the change in
the median absolute deviation (see *note Sigma clipping::). The
second popped operand (‘4’) is the multiple of sigma (STD) to use.
The third popped operand (‘3’) is number of datasets that should be
stacked (similar to the first popped operand to ‘min’). Two other
side-notes should be mentioned here:
• As mentioned above, before this operator, we are masking the
filled MAD-clipped elements with ‘madclip-maskfilled’. As
described in *note Clipping outliers::, this is very important
for removing the types of outliers that we have in
astronomical imaging.
• We are using ‘--writeall’ because this operator places two
operands on the stack: your desired statistics, and the number
of inputs that were used in it (after clipping).
$ astarithmetic a.fits b.fits c.fits -g1 --writeall \
3 5 0.01 madclip-maskfilled \
3 4 0.1 sigclip-mean
The numbers image has the smallest unsigned integer type that fits
the total number of your input datasets (see *note Numeric data
types::). For example if you have less than 255 input operands
(not pixels!), then it will have an unsigned 8-bit integer type, if
you have 1000 input operands (or any number less than 65534
inputs), it will be an unsigned 16-bit integer. Recall that when
you have many input files to stack, it may be necessary to write
the arguments into a text file and use ‘--arguments’ (see *note
Invoking astarithmetic::).
The numbers image is included by default because it is usually
important in clipping based stacks (where the number of inputs used
in the calculation of each pixel can be different from another
pixel, and this affects the final output noise). In case you are
not interested in the numbers image, you should first ‘swap’ the
two output operands, then ‘free’ the top operand like below.
$ astarithmetic a.fits b.fits c.fits -g1 \
3 5 0.01 madclip-maskfilled \
3 4 0.1 sigclip-mean swap free \
--output=single-hdu.fits
In case you just want the numbers image, you can use
‘sigclip-median’ (which is always calculated as part of the
clipping process: no extra overhead), and ‘free’ the top operand
(without the ‘swap’: the median-stack image), leaving only the
numbers image:
$ astarithmetic a.fits b.fits c.fits 3 5 0.01 \
madclip-fill-median -g1 free \
--output=single-hdu-only-numbers.fits
‘madclip-mad’
‘madclip-std’
‘madclip-mean’
‘madclip-median’
Similar to the ‘sigclip-*’ operators, but using Median Absolute
Deviation (MAD) clipping. See the description of ‘sigclip-*’ for
usage details; just remember that o1 MAD is equivalent to
$0.67449\sigma$; see *note MAD clipping:: and more generally *note
Clipping outliers::.
6.2.4.8 Filtering (smoothing) operators
.......................................
Image filtering is commonly used for smoothing: every pixel value in the
output image is created by applying a certain statistic to the pixels in
its vicinity.
‘filter-mean’
Apply mean filtering (or moving average
(https://en.wikipedia.org/wiki/Moving_average)) on the input
dataset. During mean filtering, each pixel (data element) is
replaced by the mean value of all its surrounding pixels (excluding
blank values). The number of surrounding pixels in each dimension
(to calculate the mean) is determined through the earlier operands
that have been pushed onto the stack prior to the input dataset.
The number of necessary operands is determined by the dimensions of
the input dataset (first popped operand). The order of the
dimensions on the command-line is the order in FITS format. Here
is one example:
$ astarithmetic 5 4 image.fits filter-mean
In this example, each pixel is replaced by the mean of a 5 by 4 box
around it. The box is 5 pixels along the first FITS dimension
(horizontal when viewed in ds9) and 4 pixels along the second FITS
dimension (vertical).
Each pixel will be placed in the center of the box that the mean is
calculated on. If the given width along a dimension is even, then
the center is assumed to be between the pixels (not in the center
of a pixel). When the pixel is close to the edge, the pixels of
the box that fall outside the image are ignored. Therefore, on the
edge, less points will be used in calculating the mean.
The final effect of mean filtering is to smooth the input image, it
is essentially a convolution with a kernel that has identical
values for all its pixels (is flat), see *note Convolution
process::.
Note that blank pixels will also be affected by this operator: if
there are any non-blank elements in the box surrounding a blank
pixel, in the filtered image, it will have the mean of the
non-blank elements, therefore it will not be blank any more. If
blank elements are important for your analysis, you can use the
‘isblank’ operator with the ‘where’ operator to set them back to
blank after filtering.
For example in the command below, we are first filtering the image,
then setting its original blank elements back to blank in the
output of filtering (all within one Arithmetic command). Note how
we are using the ‘set-’ operator to give names to the temporary
outputs of steps and simplify the code (see *note Operand storage
in memory or a file::).
$ astarithmetic image.fits -h1 set-in \
5 4 in filter-mean set-filtered \
filtered in isblank nan where \
--output=out.fits
‘filter-median’
Apply median filtering
(https://en.wikipedia.org/wiki/Median_filter) on the input dataset.
This is very similar to ‘filter-mean’, except that instead of the
mean value of the box pixels, the median value is used to replace a
pixel value. For more on how to use this operator, please see
‘filter-mean’.
The median is less susceptible to outliers compared to the mean.
As a result, after median filtering, the pixel values will be more
discontinuous than mean filtering.
‘filter-sigclip-mean’
Apply a $\sigma$-clipped mean filtering onto the input dataset.
This is very similar to ‘filter-mean’, except that all outliers
(identified by the $\sigma$-clipping algorithm) have been removed,
see *note Sigma clipping:: for more on the basics of this
algorithm. As described there, two extra input parameters are
necessary for $\sigma$-clipping: the multiple of $\sigma$ and the
termination criteria. ‘filter-sigclip-mean’ therefore needs to pop
two other operands from the stack after the dimensions of the box.
For example, the line below uses the same box size as the example
of ‘filter-mean’. However, all elements in the box that are
iteratively beyond $3\sigma$ of the distribution's median are
removed from the final calculation of the mean until the change in
$\sigma$ is less than $0.2$.
$ astarithmetic 3 0.2 5 4 image.fits filter-sigclip-mean
The median (which needs a sorted dataset) is necessary for
$\sigma$-clipping, therefore ‘filter-sigclip-mean’ can be
significantly slower than ‘filter-mean’. However, if there are
strong outliers in the dataset that you want to ignore (for
example, emission lines on a spectrum when finding the continuum),
this is a much better solution. Note that $\sigma$-clipping is
easily biased by outliers, so ‘filter-madclip-mean’ is more robust.
‘filter-sigclip-median’
Apply a $\sigma$-clipped median filtering onto the input dataset.
This operator and its necessary operands are almost identical to
‘filter-sigclip-mean’, except that after $\sigma$-clipping, the
median value (which is less affected by outliers than the mean) is
added back to the stack. Note that $\sigma$-clipping is easily
biased by outliers, so ‘filter-madclip-median’ is more robust.
‘filter-madclip-mean’
Apply a MAD-clipped mean filtering onto the input dataset. MAD
stands for Median Absolute Deviation, this is much more robust to
outliers than the $\sigma$-clipping. This operator is called in a
similar way to ‘filter-madclip-mean’, for example in the command
below, we use $4\times$MAD clipping with a termination criteria of
0.01 on an image:
$ astarithmetic 4 0.01 5 5 image.fits filter-madclip-mean
‘filter-madclip-median’
Apply a MAD-clipped median filtering onto the input dataset; see
the description of ‘filter-madclip-mean’ for more.
6.2.4.9 Pooling operators
.........................
Pooling is one way of reducing the complexity of the input image by
grouping multiple input pixels into one output pixel (using any
statistical measure). As a result, the output image has fewer pixels
(less complexity). In Computer Vision, Pooling is commonly used in
Convolutional Neural Networks
(https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNNs).
In pooling, the inputs are an image (e.g., a FITS file) and a square
window pixel size that is known as a pooling window. The window has to
be smaller than the input's number of pixels in both dimensions and its
width is called the "pool size". The pooling window starts at the
top-left corner pixel of the input and calculates statistical operations
on the pixels that overlap with it. It slides forward by the "stride"
pixels, moving over all pixels in the input from the top-left corner to
the bottom-right corner, and repeats the same calculation for the
overlapping pixels in each position.
Usually, the stride (or spacing between the windows as they slide
over the input) is equal to the window-size. In other words, in
pooling, the separate "windows" do not overlap with each other on the
input. However, you can choose any size for the stride. Remember this,
It's crucial to ensure that the stride size is less than the pool size.
If not, some pixels may be missed during the pooling process. Therefore
there are two major differences with *note Spatial domain convolution::
or *note Filtering operators::, but pooling has some similarities to the
*note Warp::.
• In convolution or filtering the input and output sizes are the
same. However, when the stride is larger than 1 then, the output
of pooling must have fewer pixels.
• In convolution or filters, the kernels slide over the input in a
pixel-by-pixel manner. As a result, the same pixel's value will be
used in many of the output pixels. However, in pooling each input
pixel may be only used in a single output pixel (if the stride and
the pool size are the same).
• Special cases of Warping an image are similar to pooling. For
example calling ‘pool-sum’ with pool size of 2 will give the same
pixel values (except the outer edges) as giving the same image to
‘astwarp’ with ‘--scale=1/2 --centeroncorner’. However, warping
will only provide the sum of the input pixels, there is no easy way
to generically define something like ‘pool-max’ in Warp (which is
far more general than pooling). Also, due to its generic features
(for example for non-linear warps), Warp is slower than the
‘pool-max’ that is introduced here.
*No WCS in output:* As of Gnuastro 0.23, the output of pooling will not
contain WCS information (primarily due to a lack of time by developers).
Please inform us of your interest in having it, by contacting us at
‘bug-gnuastro@gnu.org’. If you need ‘pool-sum’, you can use *note
Warp:: (which also modifies the WCS, see note above).
If the width or height of input is not divisible by the stride size,
the pool window will go beyond the input pixel grid. In this case, the
window pixels that do not overlap with the input are given a blank value
(and thus ignored in the calculation of the desired statistical
operation).
The simple ASCII figure below shows the pooling operation where the
input is a $3\times3$ pixel image with a pool size of 2 pixels. In the
center of the second row, you see the intermediate input matrix that
highlights how the input and output pixels relate with each other.
Since the input is $3\times3$ and we have a stride size of 2, as
mentioned above blank pseudo-pixels are added with a value of ‘B’ (for
blank).
Pool window: Input:
+-----------+ +-------------+
| | | | 10 12 9 |
| _ _ | _ _ |___________________________| 31 4 1 |
| | | || || | 16 5 8 |
| | | || || +-------------+
+-----------+ || ||
The pooling window 2*2 || ||
stride 2 \/ \/
+---------------------+
|/ 10 12\|/ 9 B \|
| | |
+-------+ pool-min |\ 31 4 /|\ 1 B /| pool-max +-------+
| 4 1 | /------ |---------------------| ------\ |31 9 |
| 5 8 | \------ |/ 16 5 \|/ 8 B \| ------/ |16 8 |
+-------+ | | | +-------+
|\ B B /.\ B B /|
+---------------------+
The choice of the statistic to use depends on the specific use case,
the characteristics of the input data, and the desired output. Each
statistic has its advantages and disadvantages and the choice of which
to use should be informed by the specific needs of the problem at hand.
Below, the various pool operators of arithmetic are listed:
‘pool-max’
Apply max-pooling on the input dataset. This operator takes three
operands: the first popped operand is the stride and the second is
the width of the square pooling window (which should be a single
integer). Also, The third operand should be the input image.
Within the pooling window, this operator will place the largest
value in the output pixel (any blank pixels will be ignored).
See the ASCII diagram above for a demonstration of how max-pooling
works. Here is an example of using this operator:
$ astarithmetic image.fits 2 2 pool-max
Max-pooling retains the largest value of the input window in the
output, so the returned image is sharper where you have strong
signal-to-noise ratio and more noisy in regions with no significant
signal (only noise). It is therefore useful when the background of
the image is dark and we are interested in only the highest
signal-to-noise ratio regions of the image.
‘pool-min’
Apply min-pooling on the input dataset. This operator takes three
operands: the first popped operand is the stride and the second is
the width of the square pooling window (which should be a single
integer). Also, The third operand should be the input image.
Except the used statistical measurement, this operator is similar
to ‘pool-max’, see the description there for more.
Min-pooling is mostly used when the image has a high
signal-to-noise ratio and a light background: min-pooling will
select darker (lower-valued) pixels. For low signal-to-noise
regions, this operator will increase the noise level (similar to
the maximum, the scatter in the minimum is very strong).
‘pool-sum’
Apply sum-pooling to the input dataset. This operator takes three
operands: the first popped operand is the stride and the second is
the width of the square pooling window (which should be a single
integer). Also, The third operand should be the input image.
Except the used statistical measurement, this operator is similar
to ‘pool-max’, see the description there for more.
Sum-pooling will increase the signal-to-noise ratio at the cost of
having a smoother output (less resolution).
‘pool-mean’
Apply mean pooling on the input dataset. This operator takes three
operands: the first popped operand is the stride and the second is
the width of the square pooling window (which should be a single
integer). Also, The third operand should be the input image.
Except the used statistical measurement, this operator is similar
to ‘pool-max’, see the description there for more.
The mean pooling method smooths out the image and hence the sharp
features may not be identified when this pooling method is used.
This therefore preserves more information than max-pooling, but may
also reduces the effect of the most prominent pixels. Mean is
often used where a more accurate representation of the input is
required.
‘pool-median’
Apply median pooling on the input dataset. This operator takes
three operands: the first popped operand is the stride and the
second is the width of the square pooling window (which should be a
single integer). Also, The third operand should be the input
image. Except the used statistical measurement, this operator is
similar to ‘pool-max’, see the description there for more.
In general, the mean is mathematically easier to interpret and more
susceptible to outliers, while the median outputs as being less
subject to the influence of outliers compared to the mean so we
have a smoother image. This is therefore better for low
signal-to-ratio (noisy) features and extended features (where you
don't want a single high or low valued pixel to affect the output).
6.2.4.10 Interpolation operators
................................
Interpolation is the process of removing blank pixels from a dataset (by
giving them a value based on the non-blank neighbors).
‘interpolate-medianngb’
Interpolate the blank elements of the second popped operand with
the median of nearest non-blank neighbors to each. The number of
the nearest non-blank neighbors used to calculate the median is
given by the first popped operand.
The distance of the nearest non-blank neighbors is irrelevant in
this interpolation. The neighbors of each blank pixel will be
parsed in expanding circular rings (for 2D images) or spherical
surfaces (for 3D cube) and each non-blank element over them is
stored in memory. When the requested number of non-blank neighbors
have been found, their median is used to replace that blank
element. For example, the line below replaces each blank element
with the median of the nearest 5 pixels.
$ astarithmetic image.fits 5 interpolate-medianngb
When you want to interpolate blank regions and you want each blank
region to have a fixed value (for example, the centers of saturated
stars) this operator is not good. Because the pixels used to
interpolate various parts of the region differ. For such
scenarios, you may use ‘interpolate-maxofregion’ or
‘interpolate-inofregion’ (described below).
‘interpolate-meanngb’
Similar to ‘interpolate-medianngb’, but will fill the blank values
of the dataset with the mean value of the requested number of
nearest neighbors.
‘interpolate-minngb’
Similar to ‘interpolate-medianngb’, but will fill the blank values
of the dataset with the minimum value of the requested number of
nearest neighbors.
‘interpolate-maxngb’
Similar to ‘interpolate-medianngb’, but will fill the blank values
of the dataset with the maximum value of the requested number of
nearest neighbors. One useful implementation of this operator is
to fill the saturated pixels of stars in images.
‘interpolate-minofregion’
Interpolate all blank regions (consisting of many blank pixels that
are touching) in the second popped operand with the minimum value
of the pixels that are immediately bordering that region (a single
value). The first popped operand is the connectivity (see
description in ‘connected-components’).
For example, with the command below all the connected blank regions
of ‘image.fits’ will be filled. Its an image (2D dataset), so a 2
connectivity means that the independent blank regions are defined
by 8-connected neighbors. If connectivity was 1, the regions would
be defined by 4-connectivity: blank regions that may only be
touching on the corner of one pixel would be identified as separate
regions.
$ astarithmetic image.fits 2 interpolate-minofregion
‘interpolate-maxofregion’
Similar to ‘interpolate-minofregion’, but the maximum is used to
fill the blank regions.
This operator can be useful in filling saturated pixels in stars
for example. Recall that the ‘interpolate-maxngb’ operator looks
for the maximum value with a given number of neighboring pixels and
is more useful in small noisy regions. Therefore as the blank
regions become larger, ‘interpolate-maxngb’ can cause a
fragmentation in the connected blank region because the nearest
neighbor to one part of the blank region, may not fall within the
pixels searched for the other regions. With this option, the size
of the blank region is irrelevant: all the pixels bordering the
blank region are parsed and their maximum value is used for the
whole region.
6.2.4.11 Dimensionality changing operators
..........................................
Through these operators you can change the dimensions of the output
through certain statistics on the dimensions that should be removed.
For example, let's assume you have a 3D data cube that has 300 by 300
pixels in the RA and Dec dimensions (first two dimensions), and 3600
slices along the wavelength (third dimension), so the whole cube is
$300\times300\times3600$ voxels (volume elements). To create a
narrow-band image that only contains 100 slices around a certain
wavelength, you can crop that section (using *note Crop::), giving you a
$300\times300\times100$ cube. You can now use the ‘collapse-sum’
operator below to "collapse" all the 100 slices into one 2D image that
has $300\times300$ pixels. Every pixel in this 2D image will have the
flux of the sum of the 100 slices.
‘to-1d’
Convert the input operand into a 1D array; irrespective of the
number of dimensions it has. This operator only takes a single
operand (the input array) and just updates the metadata. Therefore
it does not change the layout of the array contents in memory and
is very fast.
If no further operation is requested on the 1D array, recall that
Arithmetic will write a 1D array as a table column by default. In
case you want the output to be saved as a 1D image, or to see it on
the standard output, please use the ‘--onedasimage’ or
‘--onedonstdout’ options respectively (see *note Invoking
astarithmetic::).
This operator is useful in scenarios where after some operations on
a 2D image or 3D cube, the dimensionality is no longer relevant for
you and you just care about the values. In the example below, we
will first make a simple 2D image from a plain-text file, then
convert it to a 1D array:
## Contents of 'a.txt' to start with.
$ cat a.txt
# Image 1: DEMO [counts, uint8] An example image
1 2 3
4 5 6
7 8 9
## Convert the text image into a FITS image.
$ astconvertt a.txt -o a.fits
## Convert it into a table column (1D):
$ astarithmetic a.fits to-1d -o table.fits
## Convert it into a 1D image:
$ astarithmetic a.fits to-1d -o table.fits --onedasimage
A more real-world example would be the following: assume you want
to "flatten" two images into a single 1D array (as commonly done in
convolutional neural networks, or CNNs(1)). First, we show the
contents of a new $2\times2$ image in plain-text image, then
convert it to a 2D FITS image (‘b.fits’). We will then use
arithmetic to make both ‘a.fits’ (from the example above) and
‘b.fits’ into a 1D array and stitch them together into a single 1D
image with one call to Arithmetic. For a description of the
‘stitch’ operator, see below (same section).
## Contents of 'b.txt':
$ cat b.txt
# Image 1: DEMO [counts, uint8] An example image
10 11
12 13
## Convert the text image into a FITS image.
$ astconvertt b.txt -o b.fits
# Flatten the two images into a single 1D image:
$ astarithmetic a.fits to-1d b.fits to-1d 2 1 stitch -g1 \
--onedonstdout --quiet
1
2
3
4
5
6
7
8
9
10
11
12
13
‘stitch’
Stitch (connect) any number of given images together along the
given dimension. The output has the same number of dimensions as
the input, but the number of pixels along the requested dimension
will be different from the inputs. The ‘stitch’ operator takes at
least three operands:
• The first popped operand (placed just before ‘stitch’) is the
direction (dimension) that the images should be stitched
along. The first FITS dimension is along the horizontal,
therefore a value of ‘1’ will stitch them horizontally.
Similarly, giving a value of ‘2’ will result in a vertical
stitch.
• The second popped operand is the number of images that should
be stitched.
• Depending on the value given to the second popped operand,
‘stitch’ will pop the given number of datasets from the stack
and stitch them along the given dimension. The popped images
have to have the same number of pixels along the other
dimension. The order of the stitching is defined by how they
are placed in the command-line, not how they are popped (after
being popped, they are placed in a list in the same order).
For example, in the commands below, we will first crop out fixed
sized regions of $100\times300$ pixels of a larger image
(‘large.fits’) first. In the first call of Arithmetic below, we
will stitch the bottom set of crops together along the first
(horizontal) axis. In the second Arithmetic call, we will stitch
all 6 along both dimensions.
## Crop the fixed-size regions of a larger image ('-O' is the
## short form of the '--mode' option).
$ astcrop large.fits -Oimg --section=1:100,1:300 -oa.fits
$ astcrop large.fits -Oimg --section=101:200,1:300 -ob.fits
$ astcrop large.fits -Oimg --section=201:300,1:300 -oc.fits
$ astcrop large.fits -Oimg --section=1:100,301:600 -od.fits
$ astcrop large.fits -Oimg --section=101:200,301:600 -oe.fits
$ astcrop large.fits -Oimg --section=201:300,301:600 -of.fits
## Stitch the bottom three crops into one image.
$ astarithmetic a.fits b.fits c.fits 3 1 stitch -obottom.fits
# Stitch all the 6 crops along both dimensions
$ astarithmetic a.fits b.fits c.fits 3 1 stitch \
d.fits e.fits f.fits 3 1 stitch \
2 2 stitch -g1 -oall.fits
The start of the last command is like the one before it (stitching
the bottom three crops along the first FITS dimension, producing a
$300\times300$ image). Later in the same command, we then stitch
the top three crops horizontally (again, into a $300\times300$
image) This leaves the the two $300\times300$ images on the stack
(see *note Reverse polish notation::). We finally stitch those two
along the second (vertical) dimension. This operator is therefore
useful in scenarios like placing the CCD amplifiers into one image.
‘trim’
Trim all blank elements from the outer edges of the input operand
(it only takes a single operand). For example see the commands
below using Table's *note Column arithmetic:::
$ cat table.txt
nan
nan
nan
3
4
nan
5
6
nan
$ asttable table.txt -Y -c'arith $1 trim'
3.000000
4.000000
nan
5.000000
6.000000
Similarly, on 2D images or 3D cubes, all outer rows/columns or
slices that are fully blank get "trim"ed with this operator. This
is therefore a very useful operator for extracting a certain
feature within your dataset.
For example, let's assume that you have set *note NoiseChisel:: and
*note Segment:: on an image to extract all clumps and objects.
With the command below on Segment's output, you will have a smaller
image that only contains the sky-subtracted input pixels
corresponding to object 263.
$ astarithmetic seg.fits -hINPUT-NO-SKY seg.fits -hOBJECTS \
263 ne nan where trim --output=obj-263.fits
‘add-dimension-slow’
Build a higher-dimensional dataset from all the input datasets
stacked after one another (along the slowest dimension). The first
popped operand has to be a single number. It is used by the
operator to know how many operands it should pop from the stack
(and the size of the output in the new dimension). The rest of the
operands must have the same size and numerical data type. This
operator currently only works for 2D input operands, please contact
us if you want inputs to have different dimensions.
The output's WCS (which should have a different dimensionality
compared to the inputs) can be read from another file with the
‘--wcsfile’ option. If no file is specified for the WCS, the first
dataset's WCS will be used, you can later add/change the necessary
WCS keywords with the FITS keyword modification features of the
Fits program (see *note Fits::).
If your datasets do not have the same type, you can use the type
transformation operators of Arithmetic that are discussed below.
Just beware of overflow if you are transforming to a smaller type,
see *note Numeric data types::.
For example, let's assume you have 3 two-dimensional images
‘a.fits’, ‘b.fits’ and ‘c.fits’ (each with $200\times100$ pixels).
You can construct a 3D data cube with $200\times100\times3$ voxels
(volume-pixels) using the command below:
$ astarithmetic a.fits b.fits c.fits 3 add-dimension-slow
‘add-dimension-fast’
Similar to ‘add-dimension-slow’ but along the fastest dimension.
This operator currently only works for 1D input operands, please
contact us if you want inputs to have different dimensions.
For example, let's assume you have 3 one-dimensional datasets, each
with 100 elements. With this operator, you can construct a
$3\times100$ pixel FITS image that has 3 pixels along the
horizontal and 5 pixels along the vertical.
‘collapse-sum’
Collapse the given dataset (second popped operand), by summing all
elements along the first popped operand (a dimension in FITS
standard: counting from one, from fastest dimension). The returned
dataset has one dimension less compared to the input.
The output will have a double-precision floating point type
irrespective of the input dataset's type. Doing the operation in
double-precision (64-bit) floating point will help the collapse
(summation) be affected less by floating point errors. But
afterwards, single-precision floating points are usually enough in
real (noisy) datasets. So depending on the type of the input and
its nature, it is recommended to use one of the type conversion
operators on the returned dataset.
If any WCS is present, the returned dataset will also lack the
respective dimension in its WCS matrix. Therefore, when the WCS is
important for later processing, be sure that the input is aligned
with the respective axes: all non-diagonal elements in the WCS
matrix are zero.
One common application of this operator is the creation of pseudo
broad-band or narrow-band 2D images from 3D data cubes. For
example, integral field unit (IFU) data products that have two
spatial dimensions (first two FITS dimensions) and one spectral
dimension (third FITS dimension). The command below will collapse
the whole third dimension into a 2D array the size of the first two
dimensions, and then convert the output to single-precision
floating point (as discussed above).
$ astarithmetic cube.fits 3 collapse-sum float32
‘collapse-min’
‘collapse-max’
‘collapse-mean’
‘collapse-median’
Similar to ‘collapse-sum’, but the returned dataset will be the
desired statistic along the collapsed dimension, not the sum.
‘collapse-madclip-fill-mad’
‘collapse-madclip-fill-std’
‘collapse-madclip-fill-mean’
‘collapse-madclip-fill-median’
‘collapse-madclip-fill-number’
Collapse the input dataset (fourth popped operand) along the FITS
dimension given as the first popped operand by calculating the
desired statistic after median absolute deviation (MAD) filled
re-clipping. The MAD-clipping parameters (namely, the multiple of
sigma and termination criteria) are read as the third and second
popped operands respectively.
This is the most robust method to reject outliers; for more on
filled re-clipping and its advantages, see *note Contiguous
outliers::. For a more general tutorial on rejecting outliers, see
*note Clipping outliers::. If you have not done this tutorial yet,
we recommend you to take an hour or so and go through that tutorial
for optimal understanding and results.
When more than 95$\%$ of the area of an operand is masked, the full
operand will be masked. This is necessary in like this: one of
your inputs has many outliers (for example it is much more noisy
than the rest or its sky level has not been subtracted properly).
Because this operator fills holes between outlying pixels, most of
the area of the input will be masked, but the thin edges (where
there are no "holes") will remain, causing different statistics in
those thin edges of that input in your final stack. Through this
mask coverage fraction (which is currently hard-coded(2)), we
ensure that such thin edges do not cause artifacts in the final
stack.
For example, with the command below, the pixels of the input 2
dimensional ‘image.fits’ will be collapsed to a single dimension
output. The first popped operand is ‘2’, so it will collapse all
the pixels that are vertically on top of each other. Such that the
output will have the same number of pixels as the horizontal axis
of the input. During the collapsing, all pixels that are more than
$3\sigma$ (third popped operand) are rejected, and the clipping
will continue until the standard deviation changes less than $0.2$
between clips. Finally the ‘counter’ operator is used to have a
two-column table with the first one being a simple counter starting
from one (see *note Size and position operators::).
$ astarithmetic image.fits 3 0.2 2 collapse-sigclip-mean \
counter --output=collapsed-vertical.fits
*Printing output of collapse in plain-text:* the default datatype
of ‘collapse-sigclip-mean’ is 32-bit floating point. This is
sufficient for any observed astronomical data. However, if you
request a plain-text output, or decide to print/view the output as
plain-text on the standard output, the full set of decimals may not
be printed in some situations. This can lead to apparently
discrete values in the output of this operator when viewed in
plain-text! The FITS format is always superior (since it stores
the value in binary, therefore not having the problem above). But
if you are forced to save the output in plain-text, use the
‘float64’ operator after this to change the type to 64-bit floating
point (which will print more decimals).
‘collapse-madclip-mad’
‘collapse-madclip-std’
‘collapse-madclip-mean’
‘collapse-madclip-median’
‘collapse-madclip-number’
Collapse the input dataset (fourth popped operand) along the FITS
dimension given as the first popped operand by calculating the
desired statistic after median absolute deviation (MAD) clipping.
This operator is called similarly to the ‘collapse-madclip-fill-*’
operators, see the description there for more.
‘collapse-sigclip-fill-mad’
‘collapse-sigclip-fill-std’
‘collapse-sigclip-fill-mean’
‘collapse-sigclip-fill-median’
‘collapse-sigclip-fill-number’
Collapse the input dataset (fourth popped operand) along the FITS
dimension given as the first popped operand by calculating the
desired statistic after filled $\sigma$ re-clipping. This operator
is called similarly to the ‘collapse-madclip-fill-*’ operators, see
the description there for more.
‘collapse-sigclip-mad’
‘collapse-sigclip-std’
‘collapse-sigclip-mean’
‘collapse-sigclip-median’
‘collapse-sigclip-number’
Collapse the input dataset (fourth popped operand) along the FITS
dimension given as the first popped operand by calculating the
desired statistic after $\sigma$-clipping. This operator is called
similarly to the ‘collapse-madclip-fill-*’ operators, see the
description there for more.
---------- Footnotes ----------
(1)
(2) Please get in touch with us at ‘bug-gnuastro@gnu.org’ if you
notice this problem and feel the fraction needs to be lowered (or
generally to be set in each run).
6.2.4.12 Conditional operators
..............................
Conditional operators take two inputs and return a binary output that
can only have two values 0 (for pixels where the condition was false) or
1 (for the pixels where the condition was true). Because of the binary
(2-valued) nature of their outputs, the output is therefore stored in an
‘unsigned char’ data type (see *note Numeric data types::) to speed up
process and take less space in your storage. There are two exceptions
to the general features above: ‘isblank’ only takes one input, and
‘where’ takes three, while not returning a binary output, see their
description for more.
‘lt’
Less than: creates a binary output (values either 0 or 1) where
each pixel will be 1 if the second popped operand is smaller than
the first popped operand and 0 otherwise. If both operands are
images, then all the pixels will be compared with their
counterparts in the other image.
For example, the pixels in the output of the command below will
have a value of 1 (true) if their value in ‘image1.fits’ is less
than their value in ‘image2.fits’. Otherwise, their value will be
0 (false).
$ astarithmetic image1.fits image2.fits lt
If only one operand is an image, then all the pixels will be
compared with the single value (number) of the other operand. For
example:
$ astarithmetic image1.fits 1000 lt
Finally if both are numbers, then the output is also just one
number (0 or 1).
$ astarithmetic 4 5 lt
‘le’
Less or equal: similar to ‘lt’ ('less than' operator), but
returning 1 when the second popped operand is smaller or equal to
the first. For example
$ astarithmetic image1.fits 1000 le
‘gt’
Greater than: similar to ‘lt’ ('less than' operator), but returning
1 when the second popped operand is greater than the first. For
example
$ astarithmetic image1.fits 1000 gt
‘ge’
Greater or equal: similar to ‘lt’ ('less than' operator), but
returning 1 when the second popped operand is larger or equal to
the first. For example
$ astarithmetic image1.fits 1000 ge
‘eq’
Equality: similar to ‘lt’ ('less than' operator), but returning 1
when the two popped operands are equal (to double precision
floating point accuracy).
$ astarithmetic image1.fits 1000 eq
‘ne’
Non-Equality: similar to ‘lt’ ('less than' operator), but returning
1 when the two popped operands are _not_ equal (to double precision
floating point accuracy).
$ astarithmetic image1.fits 1000 ne
‘and’
Logical AND: returns 1 if both operands have a non-zero value and 0
if both are zero. Both operands have to be the same kind: either
both images or both numbers and it mostly makes meaningful values
when the inputs are binary (with pixel values of 0 or 1).
$ astarithmetic image1.fits image2.fits -g1 and
For example, if you only want to see which pixels in an image have
a value _between_ 50 (greater equal, or inclusive) and 200 (less
than, or exclusive), you can use this command:
$ astarithmetic image.fits set-i i 50 ge i 200 lt and
‘or’
Logical OR: returns 1 if either one of the operands is non-zero and
0 only when both operators are zero. Both operands have to be the
same kind: either both images or both numbers. The usage is
similar to ‘and’.
For example, if you only want to see which pixels in an image have
a value _outside of_ -100 (greater equal, or inclusive) and 200
(less than, or exclusive), you can use this command:
$ astarithmetic image.fits set-i i -100 lt i 200 ge or
‘not’
Logical NOT: returns 1 when the operand is 0 and 0 when the operand
is non-zero. The operand can be an image or number, for an image,
it is applied to each pixel separately. For example, if you want
to know which pixels are not blank (and assuming that we didn't
have the ‘isnotblank’ operator), you can use this ‘not’ operator on
the output of the ‘isblank’ operator described below:
$ astarithmetic image.fits isblank not
‘isblank’
Test each pixel for being a blank value (see *note Blank pixels::).
This is a conditional operator: the output has the same size and
dimensions as the input, but has an unsigned 8-bit integer type
with two possible values: either 1 (for a pixel that was blank) or
0 (for a pixel that was not blank). See the description of ‘lt’
operator above). The difference is that it only needs one operand.
For example:
$ astarithmetic image.fits isblank
Because of the definition of a blank pixel, a blank value is not
even equal to itself, so you cannot use the equal operator above to
select blank pixels. See the "Blank pixels" box below for more on
Blank pixels in Arithmetic.
In case you want to set non-blank pixels to an output pixel value
of 1, it is better to use ‘isnotblank’ instead of '‘isblank not’'
(for more, see the description of ‘isnotblank’).
‘isnotblank’
The inverse of the ‘isblank’ operator above (see that description
for more). Therefore, if a pixel has a blank value, the output of
this operator will have a 0 value for it. This operator is
therefore similar to running '‘isblank not’', but slightly more
efficient (won't need the intermediate product of two operators).
‘where’
Change the input (pixel) value _where_/if a certain condition
holds. The conditional operators above can be used to define the
condition. Three operands are required for ‘where’. The input
format is demonstrated in this simplified example:
$ astarithmetic modify.fits binary.fits if-true.fits where
The value of any pixel in ‘modify.fits’ that corresponds to a
non-zero _and_ non-blank pixel of ‘binary.fits’ will be changed to
the value of the same pixel in ‘if-true.fits’ (this may also be a
number). The 3rd and 2nd popped operands (‘modify.fits’ and
‘binary.fits’ respectively, see *note Reverse polish notation::)
have to have the same dimensions/size. ‘if-true.fits’ can be
either a number, or have the same dimension/size as the other two.
The 2nd popped operand (‘binary.fits’) has to have ‘uint8’ (or
‘unsigned char’ in standard C) type (see *note Numeric data
types::). It is treated as a binary dataset (with only two values:
zero and non-zero, hence the name ‘binary.fits’ in this example).
However, commonly you will not be dealing with an actual FITS file
of a condition/binary image. You will probably define the
condition in the same run based on some other reference image and
use the conditional and logical operators above to make a
true/false (or one/zero) image for you internally. For example,
the case below:
$ astarithmetic in.fits reference.fits 100 gt new.fits where
In the example above, any of the ‘in.fits’ pixels that has a value
in ‘reference.fits’ greater than ‘100’, will be replaced with the
corresponding pixel in ‘new.fits’. Effectively the ‘reference.fits
100 gt’ part created the condition/binary image which was added to
the stack (in memory) and later used by ‘where’. The command above
is thus equivalent to these two commands:
$ astarithmetic reference.fits 100 gt --output=binary.fits
$ astarithmetic in.fits binary.fits new.fits where
Finally, the input operands are read and used independently, so you
can use the same file more than once as any of the operands.
When the 1st popped operand to ‘where’ (‘if-true.fits’) is a single
number, it may be a NaN value (or any blank value, depending on its
type) like the example below (see *note Blank pixels::). When the
number is blank, it will be converted to the blank value of the
type of the 3rd popped operand (‘in.fits’). Hence, in the example
below, all the pixels in ‘reference.fits’ that have a value greater
than 100, will become blank in the natural data type of ‘in.fits’
(even though NaN values are only defined for floating point types).
$ astarithmetic in.fits reference.fits 100 gt nan where
6.2.4.13 Mathematical morphology operators
..........................................
From Wikipedia: "Mathematical morphology (MM) is a theory and technique
for the analysis and processing of geometrical structures, based on set
theory, lattice theory, topology, and random functions. MM is most
commonly applied to digital images". In theory it extends a very large
body of research and methods in image processing, but currently in
Gnuastro it mainly applies to images that are binary (only have a value
of 0 or 1). For example, you have applied the greater-than operator
(‘gt’, see *note Conditional operators::) to select all pixels in your
image that are larger than a value of 100. But they will all have a
value of 1, and you want to separate the various groups of pixels that
are connected (for example, peaks of stars in your image). With the
‘connected-components’ operator, you can give each connected region of
the output of ‘gt’ a separate integer label.
‘erode’
Erode the foreground pixels (with value ‘1’) of the input dataset
(second popped operand). The first popped operand is the
connectivity (see description in ‘connected-components’). Erosion
is simply a flipping of all foreground pixels (with value ‘1’) to
background (with value ‘0’) that are "touching" background pixels.
"Touching" is defined by the connectivity.
In effect, this operator "carves off" the outer borders of the
foreground, making them thinner. This operator assumes a binary
dataset (all pixels are ‘0’ or ‘1’). For example, imagine that you
have an astronomical image with a mean/sky value of 0 units and a
standard deviation ($\sigma$) of 100 units and many galaxies in it.
With the first command below, you can apply a threshold of
$2\sigma$ on the image (by only keeping pixels that are greater
than 200 using the ‘gt’ operator). The output of thresholding the
image is a binary image (each pixel is either smaller or equal to
the threshold or larger than it). You can then erode the binary
image with the second command below to remove very small false
positives (one or two pixel peaks).
$ astarithmetic image.fits 100 gt -obinary.fits
$ astarithmetic binary.fits 2 erode -oout.fits
In fact, you can merge these operations into one command thanks to
the reverse polish notation (see *note Reverse polish notation::):
$ astarithmetic image.fits 100 gt 2 erode -oout.fits
To see the effect of connectivity, try this:
$ astarithmetic image.fits 100 gt 1 erode -oout-con-1.fits
‘dilate’
Dilate the foreground pixels (with value ‘1’) of the binary input
dataset (second popped operand). The first popped operand is the
connectivity (see description in ‘connected-components’). Dilation
is simply a flipping of all background pixels (with value ‘0’) to
foreground (with value ‘1’) that are "touching" foreground pixels.
"Touching" is defined by the connectivity. In effect, this expands
the outer borders of the foreground. This operator assumes a
binary dataset (all pixels are ‘0’ and ‘1’). The usage is similar
to ‘erode’, for example:
$ astarithmetic binary.fits 2 dilate -oout.fits
‘number-neighbors’
Return a dataset of the same size as the second popped operand, but
where each non-zero and non-blank input pixel is replaced with the
number of its non-zero and non-blank neighbors. The first popped
operand is the connectivity (see above) and must be a single-value
of an integer type. The dataset is assumed to be binary (having an
unsigned, 8-bit dataset).
For example with the command below, you can select all pixels above
a value of 100 in your image with the "greater-than" or ‘gt’
operator (see *note Conditional operators::). Recall that the
output of all conditional operators is a binary output (having a
value of 0 or 1). In the same command, we will then find how many
neighboring pixels of each pixel (that was originally above the
threshold) are also above the threshold.
$ astarithmetic image.fits 100 gt 2 number-neighbors
‘connected-components’
Find the connected components in the input dataset (second popped
operand). The first popped is the connectivity used in the
connected components algorithm. The second popped operand is the
dataset where connected components are to be found. It is assumed
to be a binary image (with values of 0 or 1). It must have an
8-bit unsigned integer type which is the format produced by
conditional operators. This operator will return a labeled dataset
where the non-zero pixels in the input will be labeled with a
counter (starting from 1).
The connectivity is a number between 1 and the number of dimensions
in the dataset (inclusive). 1 corresponds to the weakest
(symmetric) connectivity between elements and the number of
dimensions the strongest. For example, on a 2D image, a
connectivity of 1 corresponds to 4-connected neighbors and 2
corresponds to 8-connected neighbors.
One example usage of this operator can be the identification of
regions above a certain threshold, as in the command below. With
this command, Arithmetic will first separate all pixels greater
than 100 into a binary image (where pixels with a value of 1 are
above that value). Afterwards, it will label all those that are
connected.
$ astarithmetic in.fits 100 gt 2 connected-components
If your input dataset does not have a binary type, but you know all
its values are 0 or 1, you can use the ‘uint8’ operator (below) to
convert it to binary.
‘fill-holes’
Flip background (0) pixels surrounded by foreground (1) in a binary
dataset. This operator takes two operands (similar to
‘connected-components’): the second is the binary (0 or 1 valued)
dataset to fill holes in and the first popped operand is the
connectivity (to define a hole). Imagine that in your dataset
there are some holes with zero value inside the objects with one
value (for example, the output of the thresholding example of
‘erode’) and you want to fill the holes:
$ astarithmetic binary.fits 2 fill-holes
‘invert’
Invert an unsigned integer dataset (will not work on other data
types, see *note Numeric data types::). This is the only operator
that ignores blank values (which are set to be the maximum values
in the unsigned integer types).
This is useful in cases where the target(s) has(have) been imaged
in absorption as raw formats (which are unsigned integer types).
With this option, the maximum value for the given type will be
subtracted from each pixel value, thus "inverting" the image, so
the target(s) can be treated as emission. This can be useful when
the higher-level analysis methods/tools only work on emission
(positive skew in the noise, not negative).
$ astarithmetic image.fits invert
6.2.4.14 Bitwise operators
..........................
Astronomical images are usually stored as an array multi-byte pixels
with different sizes for different precision levels (see *note Numeric
data types::). For example, images from CCDs are usually in the
unsigned 16-bit integer type (each pixel takes 16 bits, or 2 bytes, of
memory) and fully reduced deep images have a 32-bit floating point type
(each pixel takes 32 bits or 4 bytes).
On the other hand, during the data reduction, we need to preserve a
lot of meta-data about some pixels. For example, if a cosmic ray had
hit the pixel during the exposure, or if the pixel was saturated, or is
known to have a problem, or if the optical vignetting is too strong on
it. A crude solution is to make a new image when checking for each one
of these things and make a binary image where we flag (set to 1) pixels
that satisfy any of these conditions above, and set the rest to zero.
However, processing pipelines sometimes need more than 20 flags to store
important per-pixel meta-data, and recall that the smallest numeric data
type is one byte (or 8 bits, that can store up to 256 different values),
while we only need two values for each flag! This is a major waste of
storage space!
A much more optimal solution is to use the bits within each pixel to
store different flags! In other words, if you have an 8-bit pixel, use
each bit as a flag to mark if a certain condition has happened on a
certain pixel or not. For example, let's set the following standard
based on the four cases mentioned above: the first bit will show that a
cosmic ray has hit that pixel. So if a pixel is only affected by cosmic
rays, it will have this sequence of bits (note that the bit-counting
starts from the right): ‘00000001’. The second bit shows that the pixel
was saturated (‘00000010’), the third bit shows that it has known
problems (‘00000100’) and the fourth bit shows that it was affected by
vignetting (‘00001000’).
Since each bit is independent, we can thus mark multiple metadata
about that pixel in the actual image, within a single "flag" or "mask"
pixel of a flag or mask image that has the same number of pixels. For
example, a flag-pixel with the following bits ‘00001001’ shows that it
has been affected by cosmic rays _and_ it has been affected by
vignetting at the same time. The common data type to store these
flagging pixels are unsigned integer types (see *note Numeric data
types::). Therefore when you open an unsigned 8-bit flag image in a
viewer like DS9, you will see a single integer in each pixel that
actually has 8 layers of metadata in it! For example, the integer you
will see for the bit sequences given above will respectively be: $2^0=1$
(for a pixel that only has cosmic ray), $2^1=2$ (for a pixel that was
only saturated), $2^2=4$ (for a pixel that only has known problems),
$2^3=8$ (for a pixel that is only affected by vignetting) and $2^0 + 2^3
= 9$ (for a pixel that has a cosmic ray _and_ was affected by
vignetting).
You can later use this bit information to mark objects in your final
analysis or to mask certain pixels. For example, you may want to set
all pixels affected by vignetting to NaN, but can interpolate over
cosmic rays. You therefore need ways to separate the pixels with a
desired flag(s) from the rest. It is possible to treat a flag pixel as
a single integer (and try to define certain ranges in value to select
certain flags). But a much more easier and robust way is to actually
look at each pixel as a sequence of bits (not as a single integer!) and
use the bitwise operators below for this job. For more on the theory
behind bitwise operators, see Wikipedia
(https://en.wikipedia.org/wiki/Bitwise_operation).
‘bitand’
Bitwise AND operator: only bits with values of 1 in both popped
operands will get the value of 1, the rest will be set to 0. For
example, (assuming numbers can be written as bit strings on the
command-line): ‘00101000 00100010 bitand’ will give ‘00100000’.
Note that the bitwise operators only work on integer type datasets.
‘bitor’
Bitwise inclusive OR operator: The bits where at least one of the
two popped operands has a 1 value get a value of 1, the others 0.
For example, (assuming numbers can be written as bit strings on the
command-line): ‘00101000 00100010 bitand’ will give ‘00101010’.
Note that the bitwise operators only work on integer type datasets.
‘bitxor’
Bitwise exclusive OR operator: A bit will be 1 if it differs
between the two popped operands. For example, (assuming numbers
can be written as bit strings on the command-line): ‘00101000
00100010 bitand’ will give ‘00001010’. Note that the bitwise
operators only work on integer type datasets.
‘lshift’
Bitwise left shift operator: shift all the bits of the first
operand to the left by a number of times given by the second
operand. For example, (assuming numbers can be written as bit
strings on the command-line): ‘00101000 2 lshift’ will give
‘10100000’. This is equivalent to multiplication by 4. Note that
the bitwise operators only work on integer type datasets.
‘rshift’
Bitwise right shift operator: shift all the bits of the first
operand to the right by a number of times given by the second
operand. For example, (assuming numbers can be written as bit
strings on the command-line): ‘00101000 2 rshift’ will give
‘00001010’. Note that the bitwise operators only work on integer
type datasets.
‘bitnot’
Bitwise not (more formally known as one's complement) operator:
flip all the bits of the popped operand (note that this is the only
unary, or single operand, bitwise operator). In other words, any
bit with a value of ‘0’ is changed to ‘1’ and vice-versa. For
example, (assuming numbers can be written as bit strings on the
command-line): ‘00101000 bitnot’ will give ‘11010111’. Note that
the bitwise operators only work on integer type datasets/numbers.
6.2.4.15 Numerical type conversion operators
............................................
With the operators below you can convert the numerical data type of your
input, see *note Numeric data types::. Type conversion is particularly
useful when dealing with integers, see *note Integer benefits and
pitfalls::.
As an example, let's assume that your colleague gives you many single
exposure images for processing, but they have a double-precision
floating point type! You know that the statistical error a
single-exposure image can never exceed 6 or 7 significant digits, so you
would prefer to archive them as a single-precision floating point and
save space on your computer (a double-precision floating point is also
double the file size!). You can do this with the ‘float32’ operator
described below.
‘u8’
‘uint8’
Convert the type of the popped operand to 8-bit unsigned integer
type (see *note Numeric data types::). The internal conversion of
C will be used.
‘i8’
‘int8’
Convert the type of the popped operand to 8-bit signed integer type
(see *note Numeric data types::). The internal conversion of C
will be used.
‘u16’
‘uint16’
Convert the type of the popped operand to 16-bit unsigned integer
type (see *note Numeric data types::). The internal conversion of
C will be used.
‘i16’
‘int16’
Convert the type of the popped operand to 16-bit signed integer
(see *note Numeric data types::). The internal conversion of C
will be used.
‘u32’
‘uint32’
Convert the type of the popped operand to 32-bit unsigned integer
type (see *note Numeric data types::). The internal conversion of
C will be used.
‘i32’
‘int32’
Convert the type of the popped operand to 32-bit signed integer
type (see *note Numeric data types::). The internal conversion of
C will be used.
‘u64’
‘uint64’
Convert the type of the popped operand to 64-bit unsigned integer
(see *note Numeric data types::). The internal conversion of C
will be used.
‘f32’
‘float32’
Convert the type of the popped operand to 32-bit (single precision)
floating point (see *note Numeric data types::). The internal
conversion of C will be used. For example, if ‘f64.fits’ is a
64-bit floating point image, and you want to store it as a 32-bit
floating point image, you can use the command below (the second
command is to show that the output file consumes half the storage)
$ astarithmetic f64.fits float32 --output=f32.fits
$ ls -lh f64.fits f32.fits
‘f64’
‘float64’
Convert the type of the popped operand to 64-bit (double precision)
floating point (see *note Numeric data types::). The internal
conversion of C will be used.
6.2.4.16 Random number generators
.................................
When you simulate data (for example, see *note Sufi simulates a
detection::), everything is ideal and there is no noise! The final step
of the process is to add simulated noise to the data. The operators in
this section are designed for that purpose. To learn more about the
definition and implementation "noise", see *note Noise basics::.
In case each data element's random distribution should have an
independent parameter (for example $\sigma$ in a Gaussian distribution),
the first popped operand can be a dataset of the same size as the
second. In this case (when the parameter is not a single value, but an
array), each element will have a different parameter.
When ‘--quiet’ is not given, a statement will be printed on each
invocation of these operators (if there are multiple calls to the
‘mknoise-*’ operators, the statement will be printed multiple times).
It will show the random number generator function and seed that was used
in that invocation. These are necessary for the future reproducibility
of the outputs using the ‘--envseed’ option, for more, see *note
Generating random numbers::. For example, with the first command below,
‘image.fits’ will be degraded by a noise of standard deviation 3 units.
$ astarithmetic image.fits 3 mknoise-sigma
Alternatively, you can use the operators in this section within the
*note Column arithmetic:: feature of the Table program. For example,
with the command below, you can generate a random number (centered on 0,
with $\sigma=3$). With the second command, you can put it into a shell
variable for later usage.
$ echo 0 | asttable -c'arith $1 3 mknoise-sigma'
$ value=$(echo 0 | asttable -c'arith $1 3 mknoise-sigma' --quiet)
$ echo $value
You can also use the operators here in combination with AWK to easily
generate an arbitrarily large table with random columns. In the example
below, we will create a two column table with 20 rows. The first column
will be centered on 5 and $\sigma_1=2$, the second will be centered on
10 and $\sigma_2=3$:
$ echo 5 10 \
| awk '{for(i=0;i<20;++i) print $1, $2}' \
| asttable -c'arith $1 2 mknoise-sigma' \
-c'arith $2 3 mknoise-sigma'
By adding an extra ‘--output=random.fits’, the table will be saved
into a file called ‘random.fits’, and you can change the ‘i<20’ to
‘i<5000’ to have 5000 rows instead. Of course, if your input table has
different values in the desired column the noisy distribution will be
centered on each input element, but all will have the same
scatter/sigma.
As mentioned above, you can use the ‘--envseed’ option to pre-define
the random number generator seed (and thus get a reproducible result).
For more on ‘--envseed’, see *note Generating random numbers::. When
using column arithmetic in Table, it may happen that multiple columns
need random numbers (with any of the ‘mknoise-*’ operators) in one call
of ‘asttable’. In such cases, the value given to ‘GSL_RNG_SEED’ is
incremented by one on every call to the ‘mknoise-*’ operators. Without
this increment, when the column values are the same (happens a lot, for
no-noised datasets), the returned values for all columns will be
identical. But this feature has a side-effect: that if the order of
calling the ‘mknoise-*’ operators changes, the seeds used for each
operator will change(1).
‘mknoise-sigma’
Add a Gaussian noise with pre-defined $\sigma$ to each element of
the input dataset (independent of the input pixel value). $\sigma$
is the standard deviation of the Gaussian or Normal distribution
(https://en.wikipedia.org/wiki/Normal_distribution). This operator
takes two arguments: the top/first popped operand is the noise
standard deviation, the next popped operand is the dataset that the
noise should be added to.
For example, with the first command below, let's put a Sérsic
profile with Sérsic index 1 and effective radius 10 pixels,
truncated at 5 times the effective radius in the center of a mock
image that is $100\times100$ pixels wide. We will also give it a
position angle of 45 degrees and an axis ratio of 0.8, and set it
to have a total electron count of 10000 (‘1e4’ in the command).
Note that this example is focused on this operator, for a robust
simulation, see the tutorial in *note Sufi simulates a detection::.
With the second command, let's add noise to this image and with the
third command, we'll subtract the raw image from the noised image.
Finally, we'll view them both together:
$ echo "1 50 50 1 10 1 45 0.8 1e4 5" \
| astmkprof --mergedsize=100,100 --oversample=1 \
--mcolissum --output=raw.fits
$ astarithmetic raw.fits 2 mknoise-sigma --output=sigma.fits
$ astarithmetic raw.fits sigma.fits - -g1 \
--output=diff-sigma.fits
$ astscript-fits-view raw.fits sigma.fits diff-sigma.fits
You see that the final ‘diff-sigma.fits’ distribution was
independent of the pixel values of the input. You will also notice
that within ‘sigma.fits’ the noisy pixels that had a zero value in
‘raw.fits’, the noise fluctuates around zero (is negative in half
of those pixels). These behaviors will be different in the case
for ‘mknoise-sigma-from-mean’ below, which is more "realistic" (or
Poisson-like).
‘mknoise-sigma-from-mean’
Replace each input element (e.g., pixel in an image) of the input
with a random value taken from a Gaussian distribution (for pixel
$i$) with mean $\mu_i$ and standard deviation $\sigma_i$. Where,
$\sigma_i=\sqrt{I_i+B_i}$ and $\mu_i=I_i+B_i$ and $I_i$ and $B_i$
are respectively the values of the input image, and background in
that same pixel. In other words, this can be seen as approximating
a Poisson distribution at high mean values (where the Poisson
distribution becomes identical to the Gaussian distribution).
This operator takes two arguments: 1. the first popped operand
(just before the operator) is the _per-pixel_ background value (in
units of electron counts). 2. The second popped operand is the
dataset that the noise should be added to.
To demonstrate the effect of this noise pattern, please run the
example commands in the description of ‘mknoise-sigma’. With the
first command below, let's add this Poisson-like noise (assuming a
background level of 4 electron counts, to be similar to a
$\sigma=2$ of the example in ‘mknoise-sigma’). With the second
command, let's subtract the raw image from this noise pattern:
$ astarithmetic raw.fits 4 mknoise-sigma-from-mean \
--output=sigma-from-mean.fits
$ astarithmetic raw.fits sigma-from-mean.fits - -g1 \
--output=diff-sigma-from-mean.fits
$ astscript-fits-view diff-sigma.fits \
diff-sigma-from-mean.fits
You clearly see how the noise in the center of the Sérsic profile
is much stronger than the outer parts. As described, above, this
is behavior we would expect in a "real" observation: the regions
with stronger signal, also have stronger noise as defined through
the Poisson distribution
(https://en.wikipedia.org/wiki/Poisson_distribution)! The reason
we described this operator as "Poisson-like" is that, it has some
shortcomings as opposed to the ‘mknoise-poisson’ operator (that is
described below):
• For low mean values (less than 3 for example), this will
produce a symmetric Gaussian distribution, while the Poisson
distribution will not be symmetric.
• The random values from this distribution are floating point
(unlike the Poisson distribution that produces integers.
• The random values can be negative (which is not possible in a
Poisson distribution).
Therefore to simulate photon-starved images (for example UV or
X-ray data), the ‘mknoise-poisson’ operator should always be used,
not this one. However, in optical (or redder bands) data, the
background is very bright (much brighter than 10 counts for
example). In such cases (as the mean increases), the Poisson
distributions becomes identical to the Gaussian distribution.
Furthermore, processed co-add/stacked images are no longer
integers, but floating points with the Sky-level already subtracted
(see *note Sky value::). Therefore if you are trying to simulate a
processed, photon-rich dataset, you can safely use this operator.
Recall that the background values reported by observatories (for
example, to define dark or gray nights), or in papers, is usually
reported in units of magnitudes per arcseconds square. You need to
do the conversion to counts per pixel manually. The conversion of
magnitudes to counts is described below. For converting arcseconds
squared to number of pixels, you can use the ‘--pixelscale’ option
of *note Fits::. For example, ‘astfits image.fits --pixelscale’.
Except for the noise-model, this operator is very similar to
‘mknoise-sigma’ and the examples there apply here too. The main
difference with ‘mknoise-sigma’ is that in a Poisson distribution
the scatter/sigma will depend on each element's value.
For example, let's assume you have made a mock image called
‘mock.fits’ with *note MakeProfiles:: and it is assumed zero point
is 22.5 (for more on the zero point, see *note Brightness flux
magnitude::). Let's assume the background level for the Poisson
noise has a value of 19 magnitudes. You can first use the
‘mag-to-counts’ operator to convert this background magnitude into
counts, then feed the background value in counts to
‘mknoise-sigma-from-mean’ operator:
$ astarithmetic mock.fits 19 22.5 mag-to-counts \
mknoise-sigma-from-mean
Try changing the background value from 19 to 10 to see the effect!
Recall that the tutorial *note Sufi simulates a detection:: shows
how you can use MakeProfiles to build mock images.
‘mknoise-poisson’
Replace every pixel of the input with a random integer taken from a
Poisson distribution with the mean value of that input pixel.
Similar to ‘mknoise-sigma-from-mean’, it takes two operands: 1.
The first popped operand (just before the operator) is the
per-pixel background value (in units of electron counts). 2. The
second popped operand is the dataset that the noise should be added
to.
To demonstrate this noise pattern, let's use ‘mknoise-poisson’ in
the example of the description of ‘mknoise-sigma-from-mean’ with
the first command below. The second command below will show you
the two images side-by-side, you will notice that the Poisson
distribution's undetected regions are slightly darker (this is
because of the skewness of the Poisson distribution). Finally,
with the last two commands, you can see the histograms of the two
distributions:
$ astarithmetic raw.fits 4 mknoise-poisson \
--output=poisson.fits
$ astscript-fits-view sigma-from-mean.fits poisson.fits
$ aststatistics sigma-from-mean.fits --lessthan=10
-------
Histogram:
| ***
| ******
| **********
| ***********
| **************
| ****************
| ******************
| **********************
| **************************
| ********************************** *
|* **********************************************************
|------------------------------------------------------------
$ aststatistics poisson.fits --lessthan=10
-------
Histogram:
| * *
| * *
| * * *
| * * * *
| * * * * *
| * * * * *
| * * * * * * *
| * * * * * * *
| * * * * * * * *
| * * * * * * * *
| * * * * * * * *
|------------------------------------------------------------
The extra skewness in the Poisson distribution, and the fact that
it only returns integers is therefore clear with the commands
above. The comparison was further made above in the description of
‘mknoise-sigma-from-mean’. In summary, you should prefer the
Poisson distribution when you are simulating the following
scenarios:
• A photon-starved image (as in UV or X-ray).
• A raw exposure of a photon-rich image (which may be
photon-rich, but always integers).
‘mknoise-uniform’
Add uniform noise to each element of the input dataset. This
operator takes two arguments: the top/first popped operand is the
width of the interval, the second popped operand is the dataset
that the noise should be added to (each element will be the center
of the interval). The returned random values may happen to be the
minimum interval value, but will never be the maximum. Except for
the noise-model, this operator behaves very similar to
‘mknoise-sigma’, see the explanation there for more.
For example, with the command below, a random value will be
selected between 10 to 14 (centered on 12, which is the only input
data element, with a total width of 4).
echo 12 | asttable -c'arith $1 4 mknoise-uniform'
Similar to the example in ‘mknoise-sigma’, you can pipe the output
of ‘echo’ to ‘awk’ before passing it to ‘asttable’ to generate a
full column of uniformly selected values within the same interval.
‘random-from-hist-raw’
Generate random values from a custom distribution (defined by a
histogram). The output will have a double-precision floating point
type (see *note Numeric data types::). This operator takes three
operands:
• The first popped operand (nearest to the operator) is the
histogram values. The histogram is a 1-dimensional dataset (a
table column) and contains the probability of obtaining a
certain interval of values. The histogram does not have to be
normalized: the GNU Scientific Library (or GSL, which is used
by Gnuastro for this operator), will normalize it internally.
The value of each bin (whose probability is given in the
histogram) is given in the second popped operand. Therefore
these two operands have to have the same number of rows.
• The second popped operand is the bin value (mostly the bin
center, but it can be anything). The probability of each bin
is defined in the histogram operand (first popped operand).
The bins can have any width (do not have to be evenly spaced),
and any order. Just make sure that the same row in the bins
column corresponds to the same row in the histogram: the
number of rows in the bins and histogram must be equal.
• The third popped operand is the dataset that the random values
should be written over. Effectively only its size will be
used by this operator (all values will be over-written as a
double-precision floating point number).
The first two operands have to be single-dimensional (a table
column) and have the same number of rows, but the last popped
operand can have any number of dimensions. You can use the
‘load-col-’ operator to load the two bins and histogram columns
from an external file (see *note Loading external columns::).
For example, in the command below, we first construct a fake
histogram to represent a $y=x^2$ distribution with AWK. We aim to
distribute random values from this distribution in a $100\times100$
image. Therefore, we use the ‘makenew’ operator to construct an
empty image of that size, use the ‘load-col-’ operator to load the
histogram columns into Arithmetic and put the output in
‘random.fits’. Finally we visually inspect ‘random.fits’ with DS9
and also have a look at its pixel distribution with
‘aststatistics’.
$ echo "" | awk '{for(i=1;i<5;++i) print i, i*i}' \
> histogram.txt
$ cat histogram.txt
1 1
2 4
3 9
4 16
$ astarithmetic 100 100 2 makenew \
load-col-1-from-histogram.txt \
load-col-2-from-histogram.txt \
random-from-hist-raw \
--output=random.fits
$ astscript-fits-view random.fits
$ aststatistics random.fits --asciihist --numasciibins=50
| *
| *
| *
| *
| * *
| * *
| * *
| * * *
| * * *
|* * * *
|* * * *
|--------------------------------------------------
As you see, the 10000 pixels in the image only have values 1, 2, 3
or 4 (which were the values in the bins column of ‘histogram.txt’),
and the number of times each of these values occurs follows the
$y=x^2$ distribution.
Generally, any value given in the bins column will be used for the
final output values. For example, in the command below (for
generating a histogram from an analytical function), we are adding
the bins by 20 (while keeping the same probability distribution of
$y=x^2$). If you re-run the Arithmetic command above after this,
you will notice that the pixels values are now one of the following
21, 22, 23 or 24 (instead of 1, 2, 3, or 4). But the shape of the
histogram of the resulting random distribution will be unchanged.
$ echo "" | awk '{for(i=1;i<5;++i) print 20+i, i*i}' \
> histogram.txt
If you do not want the outputs to have exactly the value of the bin
identifier, but be a randomly selected value from a uniform
distribution within the bin, you should use ‘random-from-hist’ (see
below).
As mentioned above, the output will have a double-precision
floating point type (see *note Numeric data types::). Therefore,
by default each element of the output will consume 8 bytes
(64-bits) of storage. This is usually far more than the
statistical error/precision of your data (and just results in
wasted storage in your file system, or wasted RAM when a program
that uses the data is being run, and a slower running time of the
program).
It is therefore recommended to use a type-conversion operator after
this operator to put the output in the smallest type that can be
used to safely store your data without wasting storage, RAM or
time. For the list of type conversion operators, see *note
Numerical type conversion operators::. Recall that you already
know the values returned by this operator (they are one of the
values in the bins column).
For example, in the example above, the whole image only has values
1, 2, 3 or 4. Since they are always positive and are below 255, we
can safely place them in an unsigned 8-bit integer (see *note
Numeric data types::) with the command below (note the ‘uint8’
after the operator name, and that we are using a different name for
the output). After building the new image, let's have a look at
the sizes of the two images with ‘ls -l’:
$ astarithmetic 100 100 2 makenew \
load-col-1-from-histogram.txt \
load-col-2-from-histogram.txt \
random-from-hist-raw uint8 \
--output=random-u8.fits
$ ls -lh random.fits random-u8.fits
-rw-r--r-- 1 name name 85K Jan 01 13:40 random.fits
-rw-r--r-- 1 name name 17K Jan 01 13:45 random-u8.fits
As you see, when using a suitable data type, we can shrink the size
of the file significantly without loosing any information (from 85
kilobytes to 17 kilobytes). This difference can be felt much
better for larger (real-world) datasets, so be sure to always set
the output data type after calling this operator.
‘random-from-hist’
Similar to ‘random-from-hist-raw’, but do not return the exact bin
value, instead return a random value from a uniform distribution
within each bin. Therefore the following limitations have to be
taken into account (compared to ‘random-from-hist-raw’):
• The number associated with each bin (in the bin column) should
be its center.
• The bins have to be in descending order (so the second row in
the bin column is larger than the first).
• The bin widths (distance from one bin to another) have to be
fixed.
For a demonstration, let's replace ‘random-from-hist-raw’ with
‘random-from-hist’ in the example of the description of
‘random-from-hist-raw’. Note how we are manually converting the
output of this operator into single-precision floating point
(32-bit, since the default 64-bit precision is statistically
meaningless in this scenario and we do not want to waste storage,
memory and running time):
$ echo "" | awk '{for(i=1;i<5;++i) print i, i*i}' \
> histogram.txt
$ astarithmetic 100 100 2 makenew \
load-col-1-from-histogram.txt \
load-col-2-from-histogram.txt \
random-from-hist float32 \
--output=random.fits
$ aststatistics random.fits --asciihist --numasciibins=50
| *
| *** ********
| ************
| *************
| * * *************
| * ***********************
| *************************
| *************************
| *************************************
|********* * **************************************
|**************************************************
|--------------------------------------------------
You can see that the pixels of ‘histogram.fits’ are no longer just
1, 2, 3 or 4. Instead, the values within each bin are selected
from a uniform distribution covering that bin. This creates the
step-like feature in the histogram of the output.
Of course, this extra uniform random number generation can make
your program slower so be sure to check if it is worth it. In
particular, one way to avoid this (and use ‘random-from-hist-raw’
with a more contiguous-looking output distribution) is to simply
use a higher-resolution histogram (assuming it is possible: you
have a sufficient number of data points, or you have an analytical
expression that you can sample at smaller bin sizes).
To better demonstrate this operator and its practical usage in
everyday research, let's look at another example: Assume you want
to get 100 random star magnitudes that follow the real-world Gaia
Data release 3 magnitude distribution within a radius of 2 degrees
around the (RA,Dec) coordinate of (1.23,4.56). Let's further
assume that you want to distribute them uniformly over an image of
size 1000 by 1000 pixels. So your desired output table should have
three columns, the first two are pixel positions of each star, and
the third is the magnitude.
First, we need to query the Gaia database and ask for all the
magnitudes in this region of the sky. We know that Gaia is not
complete for stars fainter than the 20th magnitude, so we will use
the ‘--range’ option and only ask for those stars that are brighter
than magnitude 20.
$ astquery gaia --dataset=dr3 --center=1.23,3.45 --radius=2 \
--column=phot_g_mean_mag --output=gaia.fits \
--range=phot_g_mean_mag,-inf,20
We now have more than 25000 magnitudes in ‘gaia.fits’! To get a
more accurate random sampling of our stars, let's construct a
histogram with 500 bins, and generate our three desired randomly
selected columns:
$ aststatistics gaia.fits --histogram --numbins=500 \
--output=gaia-hist.fits
$ asttable gaia-hist.fits -i
$ echo 1000 \
| awk '{for(i=0;i<100;++i) print $1/2}' \
| asttable -c'arith $1 500 mknoise-uniform' \
-c'arith $1 500 mknoise-uniform' \
-c'arith $1 \
load-col-1-from-gaia-hist.fits-hdu-1 \
load-col-2-from-gaia-hist.fits-hdu-1 \
random-from-hist float32'
These columns can easily be placed in the format for *note
MakeProfiles:: to be inserted into an image automatically.
---------- Footnotes ----------
(1) We have defined Task 15971 (https://savannah.gnu.org/task/?15971)
in Gnuastro's project management system to address this. If you need
this feature please send us an email at ‘bug-gnuastro@gnu.org’ (to
motivate us in its implementation).
6.2.4.17 Coordinate and border operators
........................................
The operators here help you in defining or manipulating coordinates.
For examples to define the "box" (a rectangular region) that surrounds
an ellipse or to rotate a point around a reference point.
‘rotate-coord’
Rotate the given point (horizontal and vertical coordinates given
in 5th and 4th popped operands) around a center/reference point
(coordinates given in the 3rd and 2nd popped operands) by a given
angle (first popped operand).
For example, if you want to trace the outer edge of a circle
centered on (1.23,45.6) with a radius of 0.78, you can use this
operator like below. The logic is that we assume a single point
that is located on 0.78 units after the center on the horizontal
axis (the point's vertical axis position is the same as the
center). We then rotate this point in each row by one degree to
build the circle's circumference.
$ cx=1.23
$ cy=45.6
$ rad=0.78
$ seq 0 360 \
| awk '{print '$rad'+'$cx', '$cy', $1}' \
| asttable -c'arith $1 $2 '$cx' '$cy' $3 rotate-coord' \
--output=circle.fits
## Within TOPCAT, after opening "Plane Plot", within "Axes" select
## "Aspect lock" so the steps in both axis is the same.
$ astscript-fits-view circle.fits
If you want the points to create a circle on the celestial sphere,
you can use the ‘eq-j2000-from-flat’ operator after this one (see
*note Column arithmetic::):
$ seq 0 360 \
| awk '{print '$rad'+'$cx', '$cy', $1}' \
| asttable -c'arith $1 $2 '$cx' '$cy' $3 rotate-coord \
'$cx' '$cy' TAN eq-j2000-from-flat' \
--output=circle-on-sky.fits
When you open TOPCAT, if you open the "Plane Plot", you will see an
ellipse. However, if you open "Sky Plot" (from the "Graphics"
menu), and select the first and second columns respectively, you
will see a circle.
The center coordinates and angle can be fixed for all the rows (as
in the example above) or be different for every row. Recall that
if you want these to change on every row, you should give the
column name (or number followed by ‘$’) for these operands instead
of the constant number above.
‘box-around-ellipse’
Return the width (along horizontal) and height (along vertical) of
a box that encompasses an ellipse with the same center point. The
top-popped operand is assumed to be the position angle (angle from
the horizontal axis) in _degrees_. The second and third popped
operands are the minor and major radii of the ellipse respectively.
This operator outputs two operands on the general stack. The first
one is the width and the second (which will be the top one when
this operator finishes) is the height.
If the value to the second popped operand (minor axis) is larger
than the third (major axis), a NaN value will be written for both
the width and height of that element and a warning will be printed
(the warning can be disabled with the ‘--quiet’ option).
As an example, if your ellipse has a major axis radius of 10 units,
a minor axis radius of 4 units and a position angle of 20 degrees,
you can estimate the bounding box with this command:
$ echo "10 4 20" \
| asttable -c'arith $1 $2 $3 box-around-ellipse'
Alternatively if your three values are in separate FITS
arrays/images, you can use the command below to have the width and
height in similarly sized fits arrays. In this example ‘a.fits’
and ‘b.fits’ are respectively the major and minor axis lengths and
‘pa.fits’ is the position angle (in degrees). Also, in all three,
we assume the first extension is used. After it is done, the
height of the box will be put in ‘h.fits’ and the width will be in
‘w.fits’. Just note that because this operator has two output
datasets, you need to first write the height (top output operand)
into a file and free it with the ‘tofilefree-’ operator, then write
the width in the file given to ‘--output’.
$ astarithmetic a.fits b.fits pa.fits box-around-ellipse \
tofilefree-h.fits -ow.fits -g1
Finally, if you need to treat the width and height separately for
further processing, you can call the ‘set-’ operator two times
afterwards like below. Recall that the ‘set-’ operator will pop
the top operand, and put it in memory with a certain name, bringing
the next operand to the top of the stack.
For example, let's assume ‘catalog.fits’ has at least three columns
‘MAJOR’, ‘MINOR’ and ‘PA’ which specify the major axis, minor axis
and position angle respectively. But you want the final width and
height in 32-bit floating point numbers (not the default 64-bit,
which may be too much precision in many scenarios). You can do
this with the command below (note you can also break lines with
<\>, within the single-quote environment)
$ asttable catalog.fits \
-c'arith MAJOR MINOR PA box-around-ellipse \
set-height set-width \
width float32 height float32'
‘box-vertices-on-sphere’
Convert a box center and width to the coordinates of the vertices
of the box on a left-hand spherical coordinate system. In a
left-handed spherical coordinate system, the longitude increases
towards the left while north is up (as in the RA and Dec direction
of the equatorial coordinate system used in astronomy). This
operator therefore takes four input operands (the RA and Dec of the
box's center, as well as the width of the box in each direction).
After it is complete, this operator places 8 operands on the stack
which contain the RA and Dec of the four vertices of the box in the
following anti-clockwise order:
1. Bottom-left vertice Longitude (RA)
2. Bottom-left vertice Latitude (Dec)
3. Bottom-right vertice Longitude (RA)
4. Bottom-right vertice Latitude (Dec)
5. Top-right vertice Longitude (RA)
6. Top-right vertice Latitude (Dec)
7. Top-left vertice Longitude (RA)
8. Top-left vertice Latitude (Dec)
For example, with the command below, we will retrieve the vertice
coordinates of a rectangle around a point with RA=20 and Dec=0 (on
the equator). The rectangle will have a 1 degree edge along the RA
direction and a 2 degree edge along the declination. In this
example, we are using the ‘-Afixed -B2’ only for demonstration
purposes here due to the round numbers! In general, it is best to
write your outputs to a binary FITS table to preserve the full
precision (see *note Printing floating point numbers::).
$ echo "20 0 1 2" \
| asttable -Afixed -B2 \
-c'arith $1 $2 $3 $4 box-vertices-on-sphere'
20.50 -1.00 19.50 -1.00 19.50 1.00 20.50 1.00
We see that the bottom-left vertice is at (RA,Dec) of
$(20.50,-1.0)$ and the top-right vertice is at $(19.50,1.00)$.
These could have easily been done by manually adding and
subtracting! But you will see that the complexity arises at
higher/lower declinations. For example, with the command below,
let's see how vertice coordinates of the same box, but after moving
its center to (RA,Dec) of (20,85):
$ echo "20 85 1 2" \
| asttable -Afixed -B2 \
-c'arith $1 $2 $3 $4 box-vertices-on-sphere'
24.78 84.00 15.22 84.00 12.83 86.00 27.17 86.00
Even though, we didn't change the central RA (20) or the size of
the box along the RA (1 degree), the RA of the bottom-left vertice
is now at 24.78; almost 5 degrees away! This occurs because of the
spherical coordinate system, we measure the longitude (e.g., RA)
with the following way:
1. Draw a meridian that passes your point. The meridian is half
of a great-circle (https://en.wikipedia.org/wiki/Great_circle)
(which has a diameter that is equal to the sphere's diameter)
passes both poles.
2. Find the intersection of that meridian with the equator.
3. The distance of the intersection and the reference point
(along the equator) defines the longitude angle.
As you get more distant from the equator (declination becomes
non-zero), any change along the RA (towards the east; 1 degree in
the example above) will on longer be on a great circle, but along a
"small circle (https://en.wikipedia.org/wiki/Circle_of_a_sphere)".
On a small circle that is defined by the fixed declination
$\delta$, the distance of two points is closer than the distances
of their projection on the equator (as described in the definition
of longitude above). It is smaller by a factor of
$\cos({\delta})$.
Therefore, an angular change (let's call it $\Delta_{lon}$) along
the small circle defined by the fixed declination of $\delta$
corresponds to $\Delta_{lon}/\cos(\delta)$ on the equator.
6.2.4.18 Loading external columns
.................................
In the Arithmetic program, you can always load new dataset by simply
giving their name. However, they can only be images, not a column. In
the Table program, you can load columns in *note Column arithmetic::,
but it has to be columns within the same table (and thus the same number
of rows). However, in some situations, it is necessary to use certain
columns of a table in the Arithmetic program, or columns of different
rows (from the main input) in Table.
‘load-col-%-from-%’
‘load-col-%-from-%-hdu-%’
Load the requested column (first ‘%’) from the requested file
(second ‘%’). If the file is a FITS file, it is also necessary to
specify a HDU using the second form (where the HDU identifier is
the third ‘%’. For example, ‘load-col-MAG-from-catalog.fits-hdu-1’
will load the ‘MAG’ column from HDU 1 of ‘catalog.fits’.
For example, let's assume you have the following two tables, and
you would like to add the first column of the first with the
second:
$ asttable tab-1.fits
1 43.23
2 21.91
3 71.28
4 18.10
$ cat tab-2.txt
5
6
7
8
$ asttable tab-1.txt -c'arith $1 load-col-1-from-tab-2.txt +'
6
8
10
12
6.2.4.19 Size and position operators
....................................
With the operators below you can get metadata about the top dataset on
the stack.
‘index’
Add a new operand to the stack with an integer type and the same
size (in all dimensions) as top operand on the stack (before it was
called; it is not popped!). The first pixel in the returned
operand is zero, and every later pixel's value is incremented by
one. It is important to remember that the top operand is not
popped by this operand, so it remains on the stack. After this
operand is finished, it adds a new operand to the stack. To pop
the previous operand, you can use the ‘indexonly’ operator.
The data type of the output is always an unsigned integer, and its
width is determined from the number of pixels/rows in the top
operand. For example if there are only 108 rows in a table, the
returned column will have an unsigned 8-bit integer type (that can
keep 256 separate values). But if the top operand is a
$1000\times1000=10^6$ pixel image, the output will be a 32-bit
unsigned integer. For the various types of integers, see *note
Numeric data types::.
To see the index image along with the actual image, you can use the
‘--writeall’ operator to have a multi-HDU output (without
‘--writeall’, Arithmetic will complain if more than one operand is
left at the end). After DS9 opens with the second command, flip
between the two extensions.
$ astarithmetic image.fits index --writeall
$ astscript-fits-view image_arith.fits
Below is a review some usage examples of this operator:
Image: masking margins
With the command below, we will be masking all pixels that are
20 pixels away from the edges of the image (on the margin).
Here is a description of the command below (for the basics of
Arithmetic's notation, see *note Reverse polish notation::):
• The ‘index’ operator just adds a new dataset on the
stack: unlike almost all other operators in Arithmetic,
‘index’ doesn't remove its input dataset from the stack
(use ‘indexonly’ for the "normal" behavior). This is
because ‘index’ returns the pixel metadata not data. As
a result, after ‘index’, we have two operands on the
stack: the input image and the index image.
• With the ‘set-i’ operator, the top operand (the image
containing the index of each pixel) is popped from the
stack and associated to the name ‘i’. Therefore after
this, the stack only has the input image. For more on
the ‘set-’ operator, see *note Operand storage in memory
or a file::.
• We need three values from the commands before Arithmetic
(for the width and height of the image and the size of
the margin). To make the rest of the command easier to
read/use, we'll define them in Arithmetic as three named
operators (respectively called ‘w’, ‘h’ and ‘m’). All
three are integers that will have a positive value lower
than $2^{16}=65536$ (for a "normal" image!). Therefore,
we will store them as 16-bit unsigned integers with the
‘uint16’ operator (this will help optimal processing in
later steps). For more the type changing operators, see
*note Numerical type conversion operators::.
• Using the modulo ‘%’ and division (‘/’) operators on the
index image and the width, we extract the horizontal (X)
and vertical (Y) positions of each pixel in separately
named operands called ‘X’ and ‘Y’. The maximum value in
these two will also fit within an unsigned 16-bit
integer, so we'll also store these in that type.
• For the horizontal (X) dimension, we select pixels that
are less than the margin (‘X m lt’) and those that are
more than the width subtracted by the margin (‘X w m -
gt’).
• The output of the ‘lt’ and ‘gt’ conditional operators
above is a binary (0 or 1 valued) image. We therefore
merge them into one binary image using the ‘or’ operator.
For more, see *note Conditional operators::.
• We repeat the two steps above for the vertical (Y)
dimension.
• Once the images containing the to-be-masked pixels in
each dimension are made, we combine them into one binary
image with a final ‘or’ operator. At this point, the
stack only has two operands: 1) the input image and 2)
the binary image that has a value of 1 for all pixels
whose value should be changed.
• A single-element operand (‘nan’) is added on the stack.
• Using the ‘where’ operator, we replace all the pixels
that are non-zero in the second operand (on the margins)
to the top operand's value (NaN) in the third popped
operand (image that was read from ‘image.fits’). For
more on the ‘where’ operator, see *note Conditional
operators::.
$ margin=20
$ width=$(astfits image.fits --keyvalue=NAXIS1 -q)
$ height=$(astfits image.fits --keyvalue=NAXIS2 -q)
$ astarithmetic image.fits index set-i \
$width uint16 set-w \
$height uint16 set-h \
$margin uint16 set-m \
i w % uint16 set-X \
i w / uint16 set-Y \
X m lt X w m - gt or \
Y m lt Y h m - gt or \
or nan where
Image: Masking regions outside a circle
As another example for usage on an image, in the command below
we are using ‘index’ to define an image where each pixel
contains the distance to the pixel with X,Y coordinates of
345,250. We are then using that distance image to only keep
the pixels that are within a 50 pixel radius of that point.
The basic concept behind this process is very similar to the
previous example, with a different mathematical definition for
pixels to mask. The major difference is that we want the
distance to a pixel within the image, we need to have negative
values and the center coordinates can be in a sub-pixel
positions. The best numeric datatype for intermediate steps
is therefore floating point. 64-bit floating point can have a
precision of up to 15 digits after the decimal point. This is
far too much for what we need here: in astronomical imaging,
the PSF is usually on the scale of 1 or more pixels (see *note
Sampling theorem::). So even reaching a precision of one
millionth of a pixel (offered by 32-bit floating points) is
beyond our wildest dreams (see *note Numeric data types::).
We will also define the horizontal (X) and vertical (Y)
operands after shifting to the desired central point.
$ radius=50
$ centerx=345.2
$ centery=250.3
$ width=$(astfits image.fits --keyvalue=NAXIS1 -q)
$ astarithmetic image.fits index set-i \
$width uint16 set-w \
$radius float32 set-r \
$centerx float32 set-cx \
$centery float32 set-cy \
i w % cx - set-X \
i w / cy - set-Y \
X X x Y Y x + sqrt r gt \
nan where --output=arith-masked.fits
*Optimal data types have significant benefits:* choosing the
minimum required datatype for your operation is very important
to avoid wasting your CPU and RAM. Don't simply default to
64-bit floating points for everything! Integer operations are
much faster than floating points, and within floating point
types, 32-bit is faster and will use half the RAM/storage!
For more, see *note Numeric data types::.
The example above was just a demo for usage of the ‘index’
operator and some important concepts. But it is not the
easiest way to achieve the desired result above! An easier
way for the scenario above (to keep a circle within an image
and set everything else to NaN) is to use MakeProfiles in
combination with Arithmetic, like below:
$ radius=50
$ centerx=345.2
$ centery=250.3
$ echo "1 $centerx $centery 5 $radius 0 0 1 1 1" \
| astmkprof --background=image.fits \
--mforflatpix --clearcanvas \
-omkprof-mask.fits --type=uint8
$ astarithmetic image.fits mkprof-mask.fits not \
nan where -g1 -omkprof-masked.fits
Tables: adding new columns with row index
Within Table, you can use this operator to add an index column
like below (see the ‘counter’ operator for starting the count
from one).
## The index will be the second column.
$ asttable table.fits -c'arith $1 index'
## The index will be the first column
$ asttable table.fits -c'arith $1 index swap'
‘indexonly’
Similar to ‘index’, except that the top operand is popped from the
stack and is no longer available afterwards.
‘counter’
Similar to ‘index’, except that counting starts from one (not zero
as in ‘index’). Counting from one is usually necessary when adding
row counters in tables, like below:
$ asttable table.fits -c'arith $1 counter swap'
‘counteronly’
Similar to ‘counter’, but the top operand before it is popped (no
longer available).
‘size’
Size of the dataset along a given FITS (or FORTRAN) dimension
(counting from 1). The desired dimension should be the first
popped operand and the dataset must be the second popped operand.
The output will be a single unsigned integer (dimensions cannot be
negative). For example, the following command will produce the
size of the first extension/HDU (the default HDU) of ‘a.fits’ along
the second FITS axis.
$ astarithmetic a.fits 2 size
*Not optimal:* This operator reads the top element on the stack and
then simply reads its size along the given dimension. On a small
dataset this won't consume much RAM, but if you want to put this in
a pipeline or use it on large image, the extra RAM and slow
operation can become meaningful. To avoid such issues, you can
read the size along the given dimension using the ‘--keyvalue’
option of *note Keyword inspection and manipulation::. For
example, in the code below, the X axis position of every pixel is
returned:
$ width=$(astfits image.fits --keyvalue=NAXIS1 -q)
$ astarithmetic image.fits indexonly $width % -opix-x.fits
6.2.4.20 New operands
.....................
With the operator here, you can create a new dataset from scratch to
start certain operations without any input data.
‘makenew’
Create a new dataset that only has zero values. The number of
dimensions is read as the first popped operand and the number of
elements along each dimension are the next popped operand (in
reverse of the popping order). The type of the new dataset is an
unsigned 8-bit integer and all pixel values have a value of zero.
For example, if you want to create a new 100 by 200 pixel image,
you can run this command:
$ astarithmetic 100 200 2 makenew
To further extend the example, you can use any of the noise-making
operators to add noise to this new dataset (see *note Random number
generators::), like the command below:
$ astarithmetic 100 200 2 makenew 5 mknoise-sigma
‘constant’
Return an operand that will have a constant value (first popped
operand) in all its elements. The number of elements is read from
the second popped operand. The second popped operand is only used
for its number of elements, its numeric data type, or its values
are fully ignored and it is later freed.
Here is one useful scenario for this operator in tables: you want
to merge the objects/rows of some catalogs together, but you first
want to give each source catalog a label/counter that distinguishes
between the source of each rows in the merged/final catalog (using
*note Invoking asttable::). The steps below show the the usage of
this.
## Add label 1 to the RA, Dec, magnitude and magnitude error
## rows of the first catalog.
$ asttable cat-1.fits -cRA,DEC,MAG,MAG_ERR \
-c'arith $1 1 constant' --output=tab-1.fits
## Similar to above, but for the second catalog.
$ asttable cat-2.fits -cRA,DEC,MAG,MAG_ERR \
-c'arith $1 2 constant' --output=tab-2.fits
## Concatenate (merge/blend) the rows of the two tables into
## one for the 5 columns, but also add a counter for each
## object or row in the final catalog.
$ asttable tab-1.fits --catrowfile=tab-2.fits \
-c'arith $1 counteronly' \
-cRA,DEC,MAG,MAG_ERR,5 --output=merged.fits \
--colmetadata=1,ID_MERGED,counter,"Merged ID." \
--colmetadata=6,SOURCE-CAT,counter,"Source ID."
## Add keyword information on each input. It is very important
## to preserve this within the merged catalog. If the tables
## came from public databases (for example on VizieR), give
## their public identifier as the value.
$ astfits merged.fits --write=/,"Source catalogs" \
--write=CATSRC1,"I/355/gaiadr3","VizieR ID." \
--write=CATSRC2,"Jane Doe","Name of source."
## Check the metadata in 'merged.fits' and clean the
## temporary files.
$ rm tab-1.fits tab-2.fits
$ astfits merged.fits -h1
Like most operators, ‘constant’ is not limited to tables, you can
also apply it on images. In the example below, we'll use
‘constant’ to set all the pixels of the input image to NaN (which
is necessary in scenarios that you need to include in an image in
an analysis, but you don't want its pixels to affect the
processing):
$ astarithmetic image.fits nan constant
6.2.4.21 Operand storage in memory or a file
............................................
In your early days of using Gnuastro, to do multiple operations, it is
likely that you will simply call Arithmetic (or Table, with column
arithmetic) multiple times: feed the output file of the first call to
the second call. But as you get more proficient in the reverse polish
notation, you will find yourself combining many operations into one
call. This greatly speeds up your operation, because instead of writing
the dataset to a file in one command, and reading it in the next
command, it will just keep the intermediate dataset in memory!
But adding more complexity to your operations, can make them much
harder to debug, or extend even further. Therefore in this section we
have some special operators that behave differently from the rest: they
do not touch the contents of the data, only where/how they are stored.
They are designed to do complex operations, without necessarily having a
complex command.
‘swap’
Swap the top two operands on the stack. For example the ‘index’
operator doesn't pop with the top operand (the input to ‘index’),
it just adds the index image to the stack. In case you want your
next operation to be on the input to ‘index’, you can simply call
‘swap’ and continue the operations on that image, while keeping the
indexed pixels for later steps. In the example below we are using
the ‘--writeall’ option to write the full stack and if you open the
outputs you will see that the stack order has changed.
## Index image is written in HDU 1.
$ astarithmetic image.fits index --writeall \
--output=ind-first.fits
## image.fits in HDU 1.
$ astarithmetic image.fits index swap --writeall \
--output=img-first.fits
‘repeat’
Add N copies of the second popped operand to the stack of operands.
N is the first popped operand. For example, let's assume
‘image.fits’ is a $100\times100$ image. The output of the command
below will be a 3D data cube of size $100\times100\times20$ voxels
(volume-pixels):
$ astarithmetic image.fits 20 repeat 20 add-dimension-slow
‘free’
Free the top operand from the stack and memory. This is useful in
cases where the operator adds more than one operand on the stack.
For example operators that do stacking by clipping in *note
Stacking operators::; see the examples there for more.
‘set-AAA’
Set the characters after the dash (‘AAA’ in the case shown here) as
a name for the first popped operand on the stack. The named
dataset will be freed from memory as soon as it is no longer
needed, or if the name is reset to refer to another dataset later
in the command. This operator thus enables reusability of a
dataset without having to reread it from a file every time it is
necessary during a process. When a dataset is necessary more than
once, this operator can thus help simplify reading/writing on the
command-line (thus avoiding potential bugs), while also speeding up
the processing.
Like all operators, this operator pops the top operand off of the
main processing stack, but unlike other operands, it will not add
anything back to the stack immediately. It will keep the popped
dataset in memory through a separate list of named datasets (not on
the main stack). That list will be used to add/copy any requested
dataset to the main processing stack when the name is called.
The name to give the popped dataset is part of the operator's name.
For example, the ‘set-a’ operator of the command below, gives the
name "‘a’" to the contents of ‘image.fits’. This name is then used
instead of the actual filename to multiply the dataset by two.
$ astarithmetic image.fits set-a a 2 x
The name can be any string, but avoid strings ending with standard
filename suffixes (for example, ‘.fits’)(1).
One example of the usefulness of this operator is in the ‘where’
operator. For example, let's assume you want to mask all pixels
larger than ‘5’ in ‘image.fits’ (extension number 1) with a NaN
value. Without setting a name for the dataset, you have to read
the file two times from memory in a command like this:
$ astarithmetic image.fits image.fits 5 gt nan where -g1
But with this operator you can simply give ‘image.fits’ the name
‘i’ and simplify the command above to the more readable one below
(which greatly helps when the filename is long):
$ astarithmetic image.fits set-i i i 5 gt nan where
‘tofile-AAA’
Write the top operand on the operands stack into a file called
‘AAA’ (can be any FITS file name) without changing the operands
stack. If you do not need the dataset any more and would like to
free it, see the ‘tofilefree’ operator below.
By default, any file that is given to this operator is deleted
before Arithmetic actually starts working on the input datasets.
The deletion can be deactivated with the ‘--dontdelete’ option (as
in all Gnuastro programs, see *note Input output options::). If
the same FITS file is given to this operator multiple times, it
will contain multiple extensions (in the same order that it was
called.
For example, the operator ‘tofile-check.fits’ will write the top
operand to ‘check.fits’. Since it does not modify the operands
stack, this operator is very convenient when you want to debug, or
understanding, a string of operators and operands given to
Arithmetic: simply put ‘tofile-AAA’ anywhere in the process to see
what is happening behind the scenes without modifying the overall
process.
‘tofilefree-AAA’
Similar to the ‘tofile’ operator, with the only difference that the
dataset that is written to a file is popped from the operand stack
and freed from memory (cannot be used any more).
---------- Footnotes ----------
(1) A dataset name like ‘a.fits’ (which can be set with ‘set-a.fits’)
will cause confusion in the initial parser of Arithmetic. It will
assume this name is a FITS file, and if it is used multiple times,
Arithmetic will abort, complaining that you have not provided enough
HDUs.
6.2.5 Invoking Arithmetic
-------------------------
Arithmetic will do pixel to pixel arithmetic operations on the
individual pixels of input data and/or numbers. For the full list of
operators with explanations, please see *note Arithmetic operators::.
Any operand that only has a single element (number, or single pixel FITS
image) will be read as a number, the rest of the inputs must have the
same dimensions. The general template is:
$ astarithmetic [OPTION...] ASTRdata1 [ASTRdata2] OPERATOR ...
One line examples:
## Calculate (10.32-3.84)^2.7 quietly (will just print 155.329):
$ astarithmetic -q 10.32 3.84 - 2.7 pow
## Inverse the input image (1/pixel):
$ astarithmetic 1 image.fits / --out=inverse.fits
## Multiply each pixel in image by -1:
$ astarithmetic image.fits -1 x --out=negative.fits
## Subtract extension 4 from extension 1 (counting from zero):
$ astarithmetic image.fits image.fits - --out=skysub.fits \
--hdu=1 --hdu=4
## Add two images, then divide them by 2 (2 is read as floating point):
## Note that without the '.0', the '2' will be read/used as an integer.
$ astarithmetic image1.fits image2.fits + 2.0 / --out=average.fits
## Use Arithmetic's average operator:
$ astarithmetic image1.fits image2.fits average --out=average.fits
## Calculate the median of three images in three separate extensions:
$ astarithmetic img1.fits img2.fits img3.fits median \
-h0 -h1 -h2 --out=median.fits
Arithmetic's notation for giving operands to operators is fully
described in *note Reverse polish notation::. The output dataset is
last remaining operand on the stack. When the output dataset a single
number, and ‘--output’ is not called, it will be printed on the standard
output (command-line). When the output is an array, it will be stored
as a file.
The name of the final file can be specified with the ‘--output’
option, but if it is not given (and the output dataset has more than one
element), Arithmetic will use "automatic output" on the name of the
first FITS image encountered to generate an output file name, see *note
Automatic output::. By default, if the output file already exists, it
will be deleted before Arithmetic starts operation. However, this can
be disabled with the ‘--dontdelete’ option (see below). At any point
during Arithmetic's operation, you can also write the top operand on the
stack to a file, using the ‘tofile’ or ‘tofilefree’ operators, see *note
Arithmetic operators::.
By default, the world coordinate system (WCS) information of the
output dataset will be taken from the first input image (that contains a
WCS) on the command-line. This can be modified with the ‘--wcsfile’ and
‘--wcshdu’ options described below. When the ‘--quiet’ option is not
given, the name and extension of the dataset used for the output's WCS
is printed on the command-line.
Through operators like those starting with ‘collapse-’, the
dimensionality of the inputs may not be the same as the outputs. By
default, when the output is 1D, Arithmetic will write it as a table, not
an image/array. The format of the output table (plain text or FITS
ASCII or binary) can be set with the ‘--tableformat’ option, see *note
Input output options::). You can disable this feature (write 1D arrays
as FITS images/arrays, or to the standard output) with the
‘--onedasimage’ or ‘--onedonstdout’ options.
See *note Common options:: for a review of the options in all
Gnuastro programs. Arithmetic just redefines the ‘--hdu’ and
‘--dontdelete’ options as explained below.
‘--arguments=STR’
A plain-text file containing the command-line arguments that will
be used by Arithmetic. This option is only relevant when no
arguments are given on the command-line: if any arguments are
given, this option is ignored.
This is necessary when the set of of input files and operators
(arguments; see *note Arguments and options::) are very long
(thousands of long file names for example; usually generated within
large pipelines). Such long arguments will cause the shell to
abort with an ‘Argument list too long’ error. In such cases, you
can put the list into a plain-text file and use this option like
below. Here we are assuming you want to stack all the files in a
certain directory with the ‘mean’ operator but after masking
outliers; see *note Stacking operators:: and *note Statistical
operators:::
$ counter=0
$ for f in $(pwd)/*.fits; do \
echo $f; counter=$((counter+1)); \
done > arguments.txt; \
echo "$counter 4.5 0.01 madclip-maskfilled $counter mean" \
>> arguments.txt
$ astarithmetic --arguments=arguments.txt -g1
‘-h INT/STR’
‘--hdu INT/STR’
The header data unit of the input FITS images, see *note Input
output options::. Unlike most options in Gnuastro (which will
ultimately only have one value for this option), Arithmetic allows
‘--hdu’ to be called multiple times and the value of each
invocation will be stored separately (for the unlimited number of
input images you would like to use). Recall that for other
programs this (common) option only takes a single value. So in
other programs, if you specify it multiple times on the
command-line, only the last value will be used and in the
configuration files, it will be ignored if it already has a value.
The order of the values to ‘--hdu’ has to be in the same order as
input FITS images. Options are first read from the command-line
(from left to right), then top-down in each configuration file, see
*note Configuration file precedence::.
If the number of HDUs is less than the number of input images,
Arithmetic will abort and notify you. However, if there are more
HDUs than FITS images, there is no problem: they will be used in
the given order (every time a FITS image comes up on the stack) and
the extra HDUs will be ignored in the end. So there is no problem
with having extra HDUs in the configuration files and by default
several HDUs with a value of ‘0’ are kept in the system-wide
configuration file when you install Gnuastro.
‘-g INT/STR’
‘--globalhdu INT/STR’
Use the value to this option as the HDU of all input FITS files.
This option is very convenient when you have many input files and
the dataset of interest is in the same HDU of all the files. When
this option is called, any values given to the ‘--hdu’ option
(explained above) are ignored and will not be used.
‘-w FITS’
‘--wcsfile FITS’
FITS Filename containing the WCS structure that must be written to
the output. The HDU/extension should be specified with ‘--wcshdu’.
When this option is used, the respective WCS will be read before
any processing is done on the command-line and directly used in the
final output. If the given file does not have any WCS, then the
default WCS (first file on the command-line with WCS) will be used
in the output.
This option will mostly be used when the default file (first of the
set of inputs) is not the one containing your desired WCS. But with
this option, you can also use Arithmetic to rewrite/change the WCS
of an existing FITS dataset from another file:
$ astarithmetic data.fits --wcsfile=other.fits -ofinal.fits
‘-W STR’
‘--wcshdu STR’
HDU/extension to read the WCS within the file given to ‘--wcsfile’.
For more, see the description of ‘--wcsfile’.
‘--envseed’
Use the environment for the random number generator settings in
operators that need them (for example, ‘mknoise-sigma’). This is
very important for obtaining reproducible results, for more see
*note Generating random numbers::.
‘--append’
If the output file already exists, do not delete it; add the output
data to new HDUs at the end of that file. You can use the
‘--meta*’ options below to give a name, unit or comments to this
HDUs (to easily distinguish it from other HDUs).
‘-n STR’
‘--metaname=STR’
Metadata (name) of the output dataset. For a FITS image or table,
the string given to this option is written in the ‘EXTNAME’ or
‘TTYPE1’ keyword (respectively).
If this keyword is present in a FITS extension, it will be printed
in the table output of a command like ‘astfits image.fits’ (for
images) or ‘asttable table.fits -i’ (for tables). This metadata
can be very helpful for yourself in the future (when you have
forgotten the details), so it is recommended to use this option for
files that should be archived or shared with colleagues.
‘-u STR’
‘--metaunit=STR’
Metadata (units) of the output dataset. For a FITS image or table,
the string given to this option is written in the ‘BUNIT’ or
‘TTYPE1’ keyword respectively. In the case of tables, recall that
the Arithmetic program only outputs a single column, you should use
column arithmetic in Table for more than one column (see *note
Column arithmetic::). For more on the importance of metadata, see
the description of ‘--metaname’.
‘-c STR’
‘--metacomment=STR’
Metadata (comments) of the output dataset. For a FITS image or
table, the string given to this option is written in the ‘COMMENT’
or ‘TCOMM1’ keyword respectively. In the case of tables, recall
that the Arithmetic program only outputs a single column, you
should use column arithmetic in Table for more than one column (see
*note Column arithmetic::). For more on the importance of
metadata, see the description of ‘--metaname’.
‘-O’
‘--onedasimage’
Write final dataset as a FITS image/array even if it has a single
dimension. By default, if the output is 1D, it will be written as
a table, see above. If the output has more than one dimension,
this option is redundant.
‘-s’
‘--onedonstdout’
Write final dataset (only when it is 1D) to standard output, not as
a file. By default 1D datasets will be written as a table, see
above. If the output has more than one dimension, this option is
redundant.
‘-D’
‘--dontdelete’
Do not delete the output file, or files given to the ‘tofile’ or
‘tofilefree’ operators, if they already exist. Instead append the
desired datasets to the extensions that already exist in the
respective file. Note it does not matter if the final output file
name is given with the ‘--output’ option, or determined
automatically.
Arithmetic treats this option differently from its default
operation in other Gnuastro programs (see *note Input output
options::). If the output file exists, when other Gnuastro
programs are called with ‘--dontdelete’, they simply complain and
abort. But when Arithmetic is called with ‘--dontdelete’, it will
appended the dataset(s) to the existing extension(s) in the file.
‘-a’
‘--writeall’
Write all datasets on the stack as separate HDUs in the output
file. This only affects datasets with multiple dimensions (or
single-dimension datasets when the ‘--onedasimg’ is called). This
option is useful to debug Arithmetic calls: to check all the images
on the stack while you are designing your operation. The top
dataset on the stack will be on HDU number 1 of the output, the
second dataset will be on HDU number 2 and so on.
Arithmetic accepts two kinds of input: images and numbers. Images
are considered to be any of the inputs that is a file name of a
recognized type (see *note Arguments::) and has more than one
element/pixel. Numbers on the command-line will be read into the
smallest type (see *note Numeric data types::) that can store them, so
‘-2’ will be read as a ‘char’ type (which is signed on most systems and
can thus keep negative values), ‘2500’ will be read as an ‘unsigned
short’ (all positive numbers will be read as unsigned), while
‘3.1415926535897’ will be read as a ‘double’ and ‘3.14’ will be read as
a ‘float’. To force a number to be read as float, put a ‘.’ after it
(possibly followed by a zero for easier readability), or add an ‘f’
after it. Hence while ‘5’ will be read as an integer, ‘5.’, ‘5.0’ or
‘5f’ will be added to the stack as ‘float’ (see *note Reverse polish
notation::).
Unless otherwise stated (in *note Arithmetic operators::), the
operators can deal with numeric multiple data types (see *note Numeric
data types::). For example, in "‘a.fits b.fits +’", the image types can
be ‘long’ and ‘float’. In such cases, C's internal type conversion will
be used. The output type will be set to the higher-ranking type of the
two inputs. Unsigned integer types have smaller ranking than their
signed counterparts and floating point types have higher ranking than
the integer types. So the internal C type conversions done in the
example above are equivalent to this piece of C:
size_t i;
long a[100];
float b[100], out[100];
for(i=0;i<100;++i) out[i]=a[i]+b[i];
Relying on the default C type conversion significantly speeds up the
processing and also requires less RAM (when using very large images).
Some operators can only work on integer types (of any length, for
example, bitwise operators) while others only work on floating point
types, (currently only the ‘pow’ operator). In such cases, if the
operand type(s) are different, an error will be printed. Arithmetic
also comes with internal type conversion operators which you can use to
convert the data into the appropriate type, see *note Arithmetic
operators::.
The hyphen (‘-’) can be used both to specify options (see *note
Options::) and also to specify a negative number which might be
necessary in your arithmetic. In order to enable you to do this,
Arithmetic will first parse all the input strings and if the first
character after a hyphen is a digit, then that hyphen is temporarily
replaced by the vertical tab character which is not commonly used. The
arguments are then parsed and these strings will not be specified as an
option. Then the given arguments are parsed and any vertical tabs are
replaced back with a hyphen so they can be read as negative numbers.
Therefore, as long as the names of the files you want to work on, do not
start with a vertical tab followed by a digit, there is no problem. An
important consequence of this implementation is that you should not
write negative fractions like this: ‘-.3’, instead write them as ‘-0.3’.
Without any images, Arithmetic will act like a simple calculator and
print the resulting output number on the standard output like the first
example above. If you really want such calculator operations on the
command-line, AWK (GNU AWK is the most common implementation) is much
faster, easier and much more powerful. For example, the numerical
one-line example above can be done with the following command. In
general AWK is a fantastic tool and GNU AWK has a wonderful manual
(). So if you often confront
situations like this, or have to work with large text tables/catalogs,
be sure to checkout AWK and simplify your life.
$ echo "" | awk '{print (10.32-3.84)^2.7}'
155.329
6.3 Convolve
============
On an image, convolution can be thought of as a process to blur or
remove the contrast in an image. If you are already familiar with the
concept and just want to run Convolve, you can jump to *note Convolution
kernel:: and *note Invoking astconvolve:: and skip the lengthy
introduction on the basic definitions and concepts of convolution.
There are generally two methods to convolve an image. The first and
more intuitive one is in the "spatial domain" or using the actual image
pixel values, see *note Spatial domain convolution::. The second method
is when we manipulate the "frequency domain", or work on the magnitudes
of the different frequencies that constitute the image, see *note
Frequency domain and Fourier operations::. Understanding convolution in
the spatial domain is more intuitive and thus recommended if you are
just starting to learn about convolution. However, getting a good grasp
of the frequency domain is a little more involved and needs some
concentration and some mathematical proofs. However, its reward is a
faster operation and more importantly a very fundamental understanding
of this very important operation.
Convolution of an image will generally result in blurring the image
because it mixes pixel values. In other words, if the image has sharp
differences in neighboring pixel values(1), those sharp differences will
become smoother. This has very good consequences in detection of signal
in noise for example. In an actual observed image, the variation in
neighboring pixel values due to noise can be very high. But after
convolution, those variations will decrease and we have a better hope in
detecting the possible underlying signal. Another case where
convolution is extensively used is in mock images and modeling in
general, convolution can be used to simulate the effect of the
atmosphere or the optical system on the mock profiles that we create,
see *note PSF::. Convolution is a very interesting and important topic
in any form of signal analysis (including astronomical observations).
So we have thoroughly(2) explained the concepts behind it in the
following sub-sections.
---------- Footnotes ----------
(1) In astronomy, the only major time we confront such sharp borders
in signal are cosmic rays. All other sources of signal in an image are
already blurred by the atmosphere or the optics of the instrument.
(2) A mathematician will certainly consider this explanation is
incomplete and inaccurate. However this text is written for an
understanding on the operations that are done on a real (not complex,
discrete and noisy) astronomical image, not any general form of abstract
function
6.3.1 Spatial domain convolution
--------------------------------
The pixels in an input image represent different "spatial" positions,
therefore when convolution is done only using the actual input pixel
values, we name the process as being done in the "Spatial domain". In
particular this is in contrast to the "frequency domain" that we will
discuss later in *note Frequency domain and Fourier operations::. In
the spatial domain (and in realistic situations where the image and the
convolution kernel do not extend to infinity), convolution is the
process of changing the value of one pixel to the _weighted_ average of
all the pixels in its _neighborhood_.
The 'neighborhood' of each pixel (how many pixels in which direction)
and the 'weight' function (how much each neighboring pixel should
contribute depending on its position) are given through a second image
which is known as a "kernel"(1).
---------- Footnotes ----------
(1) Also known as filter, here we will use 'kernel'.
6.3.1.1 Convolution process
...........................
In convolution, the kernel specifies the weight and positions of the
neighbors of each pixel. To find the convolved value of a pixel, the
central pixel of the kernel is placed on that pixel. The values of each
overlapping pixel in the kernel and image are multiplied by each other
and summed for all the kernel pixels. To have one pixel in the center,
the sides of the convolution kernel have to be an odd number. This
process effectively mixes the pixel values of each pixel with its
neighbors, resulting in a blurred image compared to the sharper input
image.
Formally, convolution is one kind of linear 'spatial filtering' in
image processing texts. If we assume that the kernel has $2a+1$ and
$2b+1$ pixels on each side, the convolved value of a pixel placed at $x$
and $y$ ($C_{x,y}$) can be calculated from the neighboring pixel values
in the input image ($I$) and the kernel ($K$) from
$$C_{x,y}=\sum_{s=-a}^{a}\sum_{t=-b}^{b}K_{s,t}\times{}I_{x+s,y+t}.$$
Formally, any pixel that is outside of the image in the equation
above will be considered to be zero (although, see *note Edges in the
spatial domain::). When the kernel is symmetric about its center the
blurred image has the same orientation as the original image. However,
if the kernel is not symmetric, the image will be affected in the
opposite manner, this is a natural consequence of the definition of
spatial filtering. In order to avoid this we can rotate the kernel
about its center by 180 degrees so the convolved output can have the
same original orientation (this is done by default in the Convolve
program). Technically speaking, only if the kernel is flipped the
process is known as _Convolution_. If it is not it is known as
_Correlation_.
To be a weighted average, the sum of the weights (the pixels in the
kernel) has to be unity. This will have the consequence that the
convolved image of an object and unconvolved object will have the same
brightness (see *note Brightness flux magnitude::), which is natural,
because convolution should not eat up the object photons, it only
disperses them.
The convolution of each pixel is independent of the other pixels, and
in some cases, it may be necessary to convolve different parts of an
image separately (for example, when you have different amplifiers on the
CCD). Therefore, to speed up spatial convolution, Gnuastro first defines
a tessellation over the input; assigning each group of pixels to
"tiles". It then does the convolution in parallel on each tile. For
more on how Gnuastro's programs create the tile grid (tessellation), see
*note Tessellation::.
6.3.1.2 Edges in the spatial domain
...................................
In purely 'linear' spatial filtering (convolution), there are problems
with the edges of the input image. Here we will explain the problem in
the spatial domain. For a discussion of this problem from the frequency
domain perspective, see *note Edges in the frequency domain::. The
problem originates from the fact that on the edges, in practice, the sum
of the weights we use on the actual image pixels is not unity(1). For
example, as discussed above, a profile in the center of an image will
have the same brightness before and after convolution. However, for
partially imaged profile on the edge of the image, the brightness (sum
of its pixel fluxes within the image, see *note Brightness flux
magnitude::) will not be equal, some of the flux is going to be 'eaten'
by the edges.
If you run ‘$ make check’ on the source files of Gnuastro, you can
see this effect by comparing the ‘convolve_frequency.fits’ with
‘convolve_spatial.fits’ in the ‘./tests/’ directory. In the spatial
domain, by default, no assumption will be made about pixels outside of
the image or any blank pixels in the image. The problem explained above
will also occur on the sides of blank regions (see *note Blank
pixels::). The solution to this edge effect problem is only possible in
the spatial domain. For pixels near the edge, we have to abandon the
assumption that the sum of the kernel pixels is unity during the
convolution process(2). So taking $W$ as the sum of the kernel pixels
that overlapped with non-blank and in-image pixels, the equation in
*note Convolution process:: will become:
$$C_{x,y}= { \sum_{s=-a}^{a}\sum_{t=-b}^{b}K_{s,t}\times{}I_{x+s,y+t} \over W}.$$
In this manner, objects which are near the edges of the image or blank
pixels will also have the same brightness (within the image) before and
after convolution. This correction is applied by default in Convolve
when convolving in the spatial domain. To disable it, you can use the
‘--noedgecorrection’ option. In the frequency domain, there is no way
to avoid this loss of flux near the edges of the image, see *note Edges
in the frequency domain:: for an interpretation from the frequency
domain perspective.
Note that the edge effect discussed here is different from the one in
*note If convolving afterwards::. In making mock images we want to
simulate a real observation. In a real observation, the images of the
galaxies on the sides of the CCD are first blurred by the atmosphere and
instrument, then imaged. So light from the parts of a galaxy which are
immediately outside the CCD will affect the parts of the galaxy which
are covered by the CCD. Therefore in modeling the observation, we have
to convolve an image that is larger than the input image by exactly half
of the convolution kernel. We can hence conclude that this correction
for the edges is only useful when working on actual observed images
(where we do not have any more data on the edges) and not in modeling.
---------- Footnotes ----------
(1) Because we assumed the overlapping pixels outside the input image
have a value of zero.
(2) Of course the sum of the kernel pixels still have to be unity in
general.
6.3.2 Frequency domain and Fourier operations
---------------------------------------------
Getting a good grip on the frequency domain is usually not an easy job!
So we have decided to give the issue a complete review here.
Convolution in the frequency domain (see *note Convolution theorem::)
heavily relies on the concepts of Fourier transform (*note Fourier
transform::) and Fourier series (*note Fourier series::) so we will be
investigating these important operations first. It has become something
of a cliché for people to say that the Fourier series "is a way to
represent a (wave-like) function as the sum of simple sine waves" (from
Wikipedia). However, sines themselves are abstract functions, so this
statement really adds no extra layer of physical insight.
Before jumping head-first into the equations and proofs, we will
begin with a historical background to see how the importance of
frequencies actually roots in our ancient desire to see everything in
terms of circles. A short review of how the complex plane should be
interpreted is then given. Having paved the way with these two basics,
we define the Fourier series and subsequently the Fourier transform.
The final aim is to explain discrete Fourier transform, however some
very important concepts need to be solidified first: The Dirac comb,
convolution theorem and sampling theorem. So each of these topics are
explained in their own separate sub-sub-section before going on to the
discrete Fourier transform. Finally we revisit (after *note Edges in
the spatial domain::) the problem of convolution on the edges, but this
time in the frequency domain. Understanding the sampling theorem and
the discrete Fourier transform is very important in order to be able to
pull out valuable science from the discrete image pixels. Therefore we
have included the mathematical proofs and figures so you can have a
clear understanding of these very important concepts.
6.3.2.1 Fourier series historical background
............................................
Ever since the ancient times, the circle has been (and still is) the
simplest shape for abstract comprehension. All you need is a center
point and a radius and you are done. All the points on a circle are at
a fixed distance from the center. However, the moment you try to
connect this elegantly simple and beautiful abstract construct (the
circle) with the real world (for example, compute its area or its
circumference), things become really hard (ideally, impossible) because
the irrational number $\pi$ gets involved.
The key to understanding the Fourier series (thus the Fourier
transform and finally the Discrete Fourier Transform) is our ancient
desire to express everything in terms of circles or the most
exceptionally simple and elegant abstract human construct. Most people
prefer to say the same thing in a more ahistorical manner: to break a
function into sines and cosines. As the term "ancient" in the previous
sentence implies, Jean-Baptiste Joseph Fourier (1768 - 1830 A.D.) was
not the first person to do this. The main reason we know this process
by his name today is that he came up with an ingenious method to find
the necessary coefficients (radius of) and frequencies ("speed" of
rotation on) the circles for any generic (integrable) function.
../gnuastro-figures//epicycles.png
Figure 6.1: Epicycles and the Fourier series. Left: A demonstration of
Mercury's epicycles relative to the "center of the world" by Qutb al-Din
al-Shirazi (1236 - 1311 A.D.) retrieved from Wikipedia
(https://commons.wikimedia.org/wiki/File:Ghotb2.jpg). Middle
(https://commons.wikimedia.org/wiki/File:Fourier_series_square_wave_circles_animation.gif)
and Right: How adding more epicycles (or terms in the Fourier series)
will approximate functions. The right
(https://commons.wikimedia.org/wiki/File:Fourier_series_sawtooth_wave_circles_animation.gif)
animation is also available.
Like most aspects of mathematics, this process of interpreting
everything in terms of circles, began for astronomical purposes. When
astronomers noticed that the orbit of Mars and other outer planets, did
not appear to be a simple circle (as everything should have been in the
heavens). At some point during their orbit, the revolution of these
planets would become slower, stop, go back a little (in what is known as
the retrograde motion) and then continue going forward again.
The correction proposed by Ptolemy (90 - 168 A.D.) was the most
agreed upon. He put the planets on Epicycles or circles whose center
itself rotates on a circle whose center is the earth. Eventually, as
observations became more and more precise, it was necessary to add more
and more epicycles in order to explain the complex motions of the
planets(1). *note Figure 6.1: epicycle.(Left) shows an example
depiction of the epicycles of Mercury in the late 13th century.
Of course we now know that if they had abdicated the Earth from its
throne in the center of the heavens and allowed the Sun to take its
place, everything would become much simpler and true. But there was not
enough observational evidence for changing the "professional consensus"
of the time to this radical view suggested by a small minority(2). So
the pre-Galilean astronomers chose to keep Earth in the center and find
a correction to the models (while keeping the heavens a purely
"circular" order).
The main reason we are giving this historical background which might
appear off topic is to give historical evidence that while such
"approximations" do work and are very useful for pragmatic reasons (like
measuring the calendar from the movement of astronomical bodies). They
offer no physical insight. The astronomers who were involved with the
Ptolemaic world view had to add a huge number of epicycles during the
centuries after Ptolemy in order to explain more accurate observations.
Finally the death knell of this world-view was Galileo's observations
with his new instrument (the telescope). So the physical insight, which
is what Astronomers and Physicists are interested in (as opposed to
Mathematicians and Engineers who just like proving and optimizing or
calculating!) comes from being creative and not limiting ourselves to
such approximations. Even when they work.
---------- Footnotes ----------
(1) See the Wikipedia page on "Deferent and epicycle" for a more
complete historical review.
(2) Aristarchus of Samos (310 - 230 B.C.) appears to be one of the
first people to suggest the Sun being in the center of the universe.
This approach to science (that the standard model is defined by
consensus) and the fact that this consensus might be completely wrong
still applies equally well to our models of particle physics and
cosmology today.
6.3.2.2 Circles and the complex plane
.....................................
Before going onto the derivation, it is also useful to review how the
complex numbers and their plane relate to the circles we talked about
above. The two schematics in the middle and right of *note Figure 6.1:
epicycle. show how a 1D function of time can be made using the 2D real
and imaginary surface. Seeing the animation in Wikipedia will really
help in understanding this important concept. At each point in time, we
take the vertical coordinate of the point and use it to find the value
of the function at that point in time. *note Figure 6.2: iandtime.
shows this relation with the axes marked.
Leonhard Euler(1) (1707 - 1783 A.D.) showed that the complex
exponential ($e^{iv}$ where $v$ is real) is periodic and can be written
as: $e^{iv}=\cos{v}+isin{v}$. Therefore $e^{iv+2\pi}=e^{iv}$. Later,
Caspar Wessel (mathematician and cartographer 1745 - 1818 A.D.) showed
how complex numbers can be displayed as vectors on a plane. Euler's
identity might seem counter intuitive at first, so we will try to
explain it geometrically (for deeper physical insight). On the
real-imaginary 2D plane (like the left hand plot in each box of *note
Figure 6.2: iandtime.), multiplying a number by $i$ can be interpreted
as rotating the point by $90$ degrees (for example, the value $3$ on the
real axis becomes $3i$ on the imaginary axis). On the other hand,
$e\equiv\lim_{n\rightarrow\infty}(1+{1\over n})^n$, therefore, defining
$m\equiv nu$, we get:
$$e^{u}=\lim_{n\rightarrow\infty}\left(1+{1\over n}\right)^{nu}
=\lim_{n\rightarrow\infty}\left(1+{u\over nu}\right)^{nu}
=\lim_{m\rightarrow\infty}\left(1+{u\over m}\right)^{m}$$
Taking $u\equiv iv$ the result can be written as a generic complex
number (a function of $v$):
$$e^{iv}=\lim_{m\rightarrow\infty}\left(1+i{v\over
m}\right)^{m}=a(v)+ib(v)$$
For $v=\pi$, a nice geometric animation of going to the limit can be
seen on Wikipedia (https://commons.wikimedia.org/wiki/File:ExpIPi.gif).
We see that $\lim_{m\rightarrow\infty}a(\pi)=-1$, while
$\lim_{m\rightarrow\infty}b(\pi)=0$, which gives the famous
$e^{i\pi}=-1$ equation. The final value is the real number $-1$,
however the distance of the polygon points traversed as
$m\rightarrow\infty$ is half the circumference of a circle or $\pi$,
showing how $v$ in the equation above can be interpreted as an angle in
units of radians and therefore how $a(v)=cos(v)$ and $b(v)=sin(v)$.
Since $e^{iv}$ is periodic (let's assume with a period of $T$), it is
more clear to write it as $v\equiv{2{\pi}n\over T}t$ (where $n$ is an
integer), so $e^{iv}=e^{i{2{\pi}n\over T}t}$. The advantage of this
notation is that the period ($T$) is clearly visible and the frequency
($2{\pi}n \over T$, in units of 1/cycle) is defined through the integer
$n$. In this notation, $t$ is in units of "cycle"s.
As we see from the examples in *note Figure 6.1: epicycle. and *note
Figure 6.2: iandtime, for each constituting frequency, we need a
respective 'magnitude' or the radius of the circle in order to
accurately approximate the desired 1D function. The concepts of
"period" and "frequency" are relatively easy to grasp when using
temporal units like time because this is how we define them in every-day
life. However, in an image (astronomical data), we are dealing with
spatial units like distance. Therefore, by one "period" we mean the
_distance_ at which the signal is identical and frequency is defined as
the inverse of that spatial "period". The complex circle of *note
Figure 6.2: iandtime. can be thought of the Moon rotating about Earth
which is rotating around the Sun; so the "Real (signal)" axis shows the
Moon's position as seen by a distant observer on the Sun as time goes
by. Because of the scalar (not having any direction or vector) nature
of time, *note Figure 6.2: iandtime. is easier to understand in units of
time. When thinking about spatial units, mentally replace the "Time
(sec)" axis with "Distance (meters)". Because length has direction and
is a vector, visualizing the rotation of the imaginary circle and the
advance along the "Distance (meters)" axis is not as simple as temporal
units like time.
../gnuastro-figures//iandtime.eps
Figure 6.2: Relation between the real (signal), imaginary
($i\equiv\sqrt{-1}$) and time axes at two snapshots of time.
---------- Footnotes ----------
(1) Other forms of this equation were known before Euler. For
example, in 1707 A.D. (the year of Euler's birth) Abraham de Moivre
(1667 - 1754 A.D.) showed that
$(\cos{x}+i\sin{x})^n=\cos(nx)+i\sin(nx)$. In 1714 A.D., Roger Cotes
(1682 - 1716 A.D. a colleague of Newton who proofread the second edition
of Principia) showed that: $ix=\ln(\cos{x}+i\sin{x})$.
6.3.2.3 Fourier series
......................
In astronomical images, our variable (brightness, or number of
photo-electrons, or signal to be more generic) is recorded over the 2D
spatial surface of a camera pixel. However to make things easier to
understand, here we will assume that the signal is recorded in 1D
(assume one row of the 2D image pixels). Also for this section and the
next (*note Fourier transform::) we will be talking about the signal
before it is digitized or pixelated. Let's assume that we have the
continuous function $f(l)$ which is integrable in the interval $[l_0,
l_0+L]$ (always true in practical cases like images). Take $l_0$ as the
position of the first pixel in the assumed row of the image and $L$ as
the width of the image along that row. The units of $l_0$ and $L$ can
be in any spatial units (for example, meters) or an angular unit (like
radians) multiplied by a fixed distance which is more common.
To approximate $f(l)$ over this interval, we need to find a set of
frequencies and their corresponding 'magnitude's (see *note Circles and
the complex plane::). Therefore our aim is to show $f(l)$ as the
following sum of periodic functions:
$$f(l)=\displaystyle\sum_{n=-\infty}^{\infty}c_ne^{i{2{\pi}n\over L}l} $$
Note that the different frequencies ($2{\pi}n/L$, in units of cycles per
meters for example) are not arbitrary. They are all integer multiples
of the fundamental frequency of $\omega_0=2\pi/L$. Recall that $L$ was
the length of the signal we want to model. Therefore, we see that the
smallest possible frequency (or the frequency resolution) in the end,
depends on the length we observed the signal or $L$. In the case of
each dimension on an image, this is the size of the image in the
respective dimension. The frequencies have been defined in this
"harmonic" fashion to insure that the final sum is periodic outside of
the $[l_0, l_0+L]$ interval too. At this point, you might be thinking
that the sky is not periodic with the same period as my camera's view
angle. You are absolutely right! The important thing is that since
your camera's observed region is the only region we are "observing" and
will be using, the rest of the sky is irrelevant; so we can safely
assume the sky is periodic outside of it. However, this working
assumption will haunt us later in *note Edges in the frequency domain::.
The frequencies are thus determined by definition. So all we need to
do is to find the coefficients ($c_n$), or magnitudes, or radii of the
circles for each frequency which is identified with the integer $n$.
Fourier's approach was to multiply both sides with a fixed term:
$$f(l)e^{-i{2{\pi}m\over L}l}=\displaystyle\sum_{n=-\infty}^{\infty}c_ne^{i{2{\pi}(n-m)\over L}l}
$$
where $m>0$(1). We can then integrate both sides over the observation
period:
$$\int_{l_0}^{l_0+L}f(l)e^{-i{2{\pi}m\over L}l}dl
=\int_{l_0}^{l_0+L}\displaystyle\sum_{n=-\infty}^{\infty}c_ne^{i{2{\pi}(n-m)\over L}l}dl=\displaystyle\sum_{n=-\infty}^{\infty}c_n\int_{l_0}^{l_0+L}e^{i{2{\pi}(n-m)\over L}l}dl
$$
Both $n$ and $m$ are positive integers. Also, we know that a complex
exponential is periodic so after one period ($L$) it comes back to its
starting point. Therefore $\int_{l_0}^{l_0+L}e^{2{\pi}k/L}dl=0$ for any
$k>0$. However, when $k=0$, this integral becomes:
$\int_{l_0}^{l_0+T}e^0dt=\int_{l_0}^{l_0+T}dt=T$. Hence since the
integral will be zero for all $n{\neq}m$, we get:
$$\displaystyle\sum_{n=-\infty}^{\infty}c_n\int_{l_0}^{l_0+T}e^{i{2{\pi}(n-m)\over L}l}dl=Lc_m $$
The origin of the axis is fundamentally an arbitrary position. So let's
set it to the start of the image such that $l_0=0$. So we can find the
"magnitude" of the frequency $2{\pi}m/L$ within $f(l)$ through the
relation:
$$c_m={1\over L}\int_{0}^{L}f(l)e^{-i{2{\pi}m\over L}l}dl $$
---------- Footnotes ----------
(1) We could have assumed $m<0$ and set the exponential to positive,
but this is more clear.
6.3.2.4 Fourier transform
.........................
In *note Fourier series::, we had to assume that the function is
periodic outside of the desired interval with a period of $L$.
Therefore, assuming that $L\rightarrow\infty$ will allow us to work with
any function. However, with this approximation, the fundamental
frequency ($\omega_0$) or the frequency resolution that we discussed in
*note Fourier series:: will tend to zero: $\omega_0\rightarrow0$. In
the equation to find $c_m$, every $m$ represented a frequency (multiple
of $\omega_0$) and the integration on $l$ removes the dependence of the
right side of the equation on $l$, making it only a function of $m$ or
frequency. Let's define the following two variables:
$$\omega{\equiv}m\omega_0={2{\pi}m\over L}$$
$$F(\omega){\equiv}Lc_m$$
The equation to find the coefficients of each frequency in *note Fourier
series:: thus becomes:
$$F(\omega)=\int_{-\infty}^{\infty}f(l)e^{-i{\omega}l}dl.$$
The function $F(\omega)$ is thus the _Fourier transform_ of $f(l)$ in
the frequency domain. So through this transformation, we can find
(analyze) the magnitudes of the constituting frequencies or the value in
the frequency space(1) of our spatial input function. The great thing
is that we can also do the reverse and later synthesize the input
function from its Fourier transform. Let's do it: with the
approximations above, multiply the right side of the definition of the
Fourier Series (*note Fourier series::) with
$1=L/L=({\omega_0}L)/(2\pi)$:
$$f(l)={1\over
2\pi}\displaystyle\sum_{n=-\infty}^{\infty}Lc_ne^{{2{\pi}in\over
L}l}\omega_0={1\over
2\pi}\displaystyle\sum_{n=-\infty}^{\infty}F(\omega)e^{i{\omega}l}\Delta\omega
$$
To find the right most side of this equation, we renamed $\omega_0$ as
$\Delta\omega$ because it was our resolution, $2{\pi}n/L$ was written as
$\omega$ and finally, $Lc_n$ was written as $F(\omega)$ as we defined
above. Now, as $L\rightarrow\infty$, $\Delta\omega\rightarrow0$ so we
can write:
$$f(l)={1\over
2\pi}\int_{-\infty}^{\infty}F(\omega)e^{i{\omega}l}d\omega $$
Together, these two equations provide us with a very powerful set of
tools that we can use to process (analyze) and recreate (synthesize) the
input signal. Through the first equation, we can break up our input
function into its constituent frequencies and analyze it, hence it is
also known as _analysis_. Using the second equation, we can synthesize
or make the input function from the known frequencies and their
magnitudes. Thus it is known as _synthesis_. Here, we symbolize the
Fourier transform (analysis) and its inverse (synthesis) of a function
$f(l)$ and its Fourier Transform $F(\omega)$ as ${\cal F}[f]$ and ${\cal
F}^{-1}[F]$.
---------- Footnotes ----------
(1) As we discussed before, this 'magnitude' can be interpreted as
the radius of the circle rotating at this frequency in the epicyclic
interpretation of the Fourier series, see *note Figure 6.1: epicycle.
and *note Figure 6.2: iandtime.
6.3.2.5 Dirac delta and comb
............................
The Dirac $\delta$ (delta) function (also known as an impulse) is the
way that we convert a continuous function into a discrete one. It is
defined to satisfy the following integral:
$$\int_{-\infty}^{\infty}\delta(l)dl=1$$
When integrated with another function, it gives that function's value at
$l=0$:
$$\int_{-\infty}^{\infty}f(l)\delta(l)dt=f(0)$$
An impulse positioned at another point (say $l_0$) is written as
$\delta(l-l_0)$:
$$\int_{-\infty}^{\infty}f(l)\delta(l-l_0)dt=f(l_0)$$
The Dirac $\delta$ function also operates similarly if we use summations
instead of integrals. The Fourier transform of the delta function is:
$${\cal F}[\delta(l)]=\int_{-\infty}^{\infty}\delta(l)e^{-i{\omega}l}dl=e^{-i{\omega}0}=1$$
$${\cal F}[\delta(l-l_0)]=\int_{-\infty}^{\infty}\delta(l-l_0)e^{-i{\omega}l}dl=e^{-i{\omega}l_0}$$
From the definition of the Dirac $\delta$ we can also define a Dirac
comb (${\rm III}_P$) or an impulse train with infinite impulses
separated by $P$:
$${\rm III}_P(l)\equiv\displaystyle\sum_{k=-\infty}^{\infty}\delta(l-kP) $$
$P$ is chosen to represent "pixel width" later in *note Sampling
theorem::. Therefore the Dirac comb is periodic with a period of $P$.
We have intentionally used a different name for the period of the Dirac
comb compared to the input signal's length of observation that we showed
with $L$ in *note Fourier series::. This difference is highlighted here
to avoid confusion later when these two periods are needed together in
*note Discrete Fourier transform::. The Fourier transform of the Dirac
comb will be necessary in *note Sampling theorem::, so let's derive it.
By its definition, it is periodic, with a period of $P$, so the Fourier
coefficients of its Fourier Series (*note Fourier series::) can be
calculated within one period:
$${\rm III}_P=\displaystyle\sum_{n=-\infty}^{\infty}c_ne^{i{2{\pi}n\over
P}l}$$
We can now find the $c_n$ from *note Fourier series:::
$$c_n={1\over P}\int_{-P/2}^{P/2}\delta(l)e^{-i{2{\pi}n\over P}l}
={1\over P}\quad\quad \rightarrow \quad\quad
{\rm III}_P={1\over P}\displaystyle\sum_{n=-\infty}^{\infty}e^{i{2{\pi}n\over P}l}
$$
So we can write the Fourier transform of the Dirac comb as:
$${\cal F}[{\rm III}_P]=\int_{-\infty}^{\infty}{\rm III}_Pe^{-i{\omega}l}dl
={1\over P}\displaystyle\sum_{n=-\infty}^{\infty}\int_{-\infty}^{\infty}e^{-i(\omega-{2{\pi}n\over P})l}dl={1\over P}\displaystyle\sum_{n=-\infty}^{\infty}\delta\left(\omega-{2{\pi}n\over P}\right)
$$
In the last step, we used the fact that the complex exponential is a
periodic function, that $n$ is an integer and that as we defined in
*note Fourier transform::, $\omega{\equiv}m\omega_0$, where $m$ was an
integer. The integral will be zero for any $\omega$ that is not equal
to $2{\pi}n/P$, a more complete explanation can be seen in *note Fourier
series::. Therefore, while in the spatial domain the impulses had
spacing of $P$ (meters for example), in the frequency space, the spacing
between the different impulses are $2\pi/P$ cycles per meters.
6.3.2.6 Convolution theorem
...........................
The convolution (shown with the $\ast$ operator) of the two functions
$f(l)$ and $h(l)$ is defined as:
$$c(l)\equiv[f{\ast}h](l)=\int_{-\infty}^{\infty}f(\tau)h(l-\tau)d\tau
$$
See *note Convolution process:: for a more detailed physical (pixel
based) interpretation of this definition. The Fourier transform of
convolution ($C(\omega)$) can be written as:
$$ C(\omega)=\int_{-\infty}^{\infty}[f{\ast}h](l)e^{-i{\omega}l}dl=
\int_{-\infty}^{\infty}f(\tau)\left[\int_{-\infty}^{\infty}h(l-\tau)e^{-i{\omega}l}dl\right]d\tau
$$
To solve the inner integral, let's define $s{\equiv}l-\tau$, so that
$ds=dl$ and $l=s+\tau$ then the inner integral becomes:
$$\int_{-\infty}^{\infty}h(l-\tau)e^{-i{\omega}l}dl=
\int_{-\infty}^{\infty}h(s)e^{-i{\omega}(s+\tau)}ds=e^{-i{\omega}\tau}\int_{-\infty}^{\infty}h(s)e^{-i{\omega}s}ds=H(\omega)e^{-i{\omega}\tau}
$$
where $H(\omega)$ is the Fourier transform of $h(l)$. Substituting this
result for the inner integral above, we get:
$$C(\omega)=H(\omega)\int_{-\infty}^{\infty}f(\tau)e^{-i{\omega}\tau}d\tau=H(\omega)F(\omega)=F(\omega)H(\omega)
$$
where $F(\omega)$ is the Fourier transform of $f(l)$. So multiplying
the Fourier transform of two functions individually, we get the Fourier
transform of their convolution. The convolution theorem also proves a
relation between the convolutions in the frequency space. Let's define:
$$D(\omega){\equiv}F(\omega){\ast}H(\omega)$$
Applying the inverse Fourier Transform or synthesis equation (*note
Fourier transform::) to both sides and following the same steps above,
we get:
$$d(l)=f(l)h(l)$$
Where $d(l)$ is the inverse Fourier transform of $D(\omega)$. We can
therefore re-write the two equations above formally as the convolution
theorem:
$$ {\cal F}[f{\ast}h]={\cal F}[f]{\cal F}[h]
$$
$$ {\cal F}[fh]={\cal F}[f]\ast{\cal F}[h]
$$
Besides its usefulness in blurring an image by convolving it with a
given kernel, the convolution theorem also enables us to do another very
useful operation in data analysis: to match the blur (or PSF) between
two images taken with different telescopes/cameras or under different
atmospheric conditions. This process is also known as deconvolution.
Let's take $f(l)$ as the image with a narrower PSF (less blurry) and
$c(l)$ as the image with a wider PSF which appears more blurred. Also
let's take $h(l)$ to represent the kernel that should be convolved with
the sharper image to create the more blurry image. Above, we proved the
relation between these three images through the convolution theorem.
But there, we assumed that $f(l)$ and $h(l)$ are known (given) and the
convolved image is desired.
In deconvolution, we have $f(l)$ -the sharper image- and $f*h(l)$
-the more blurry image- and we want to find the kernel $h(l)$. The
solution is a direct result of the convolution theorem:
$$ {\cal F}[h]={{\cal F}[f{\ast}h]\over {\cal F}[f]}
\quad\quad
{\rm or}
\quad\quad
h(l)={\cal F}^{-1}\left[{{\cal F}[f{\ast}h]\over {\cal F}[f]}\right]
$$
While this works really nice, it has two problems:
• If ${\cal F}[f]$ has any zero values, then the inverse Fourier
transform will not be a number!
• If there is significant noise in the image, then the high
frequencies of the noise are going to significantly reduce the
quality of the final result.
A standard solution to both these problems is the Weiner
deconvolution algorithm(1).
---------- Footnotes ----------
(1)
6.3.2.7 Sampling theorem
........................
Our mathematical functions are continuous, however, our data collecting
and measuring tools are discrete. Here we want to give a mathematical
formulation for digitizing the continuous mathematical functions so that
later, we can retrieve the continuous function from the digitized
recorded input. Assuming that we have a continuous function $f(l)$,
then we can define $f_s(l)$ as the 'sampled' $f(l)$ through the Dirac
comb (see *note Dirac delta and comb::):
$$f_s(l)=f(l){\rm III}_P=\displaystyle\sum_{n=-\infty}^{\infty}f(l)\delta(l-nP)
$$
The discrete data-element $f_k$ (for example, a pixel in an image),
where $k$ is an integer, can thus be represented as:
$$f_k=\int_{-\infty}^{\infty}f_s(l)dl=\int_{-\infty}^{\infty}f(l)\delta(l-kP)dt=f(kP)$$
Note that in practice, our discrete data points are not found in this
fashion. Each detector pixel (in an image for example) has an area and
averages the signal it receives over that area, not a mathematical point
as the Dirac $\delta$ function defines. However, as long as the
variation in the signal over one detector pixel is not significant, this
can be a good approximation. Having put this issue to the side, we can
now try to find the relation between the Fourier transforms of the
un-sampled $f(l)$ and the sampled $f_s(l)$. For a more clear notation,
let's define:
$$F_s(\omega)\equiv{\cal F}[f_s]$$
$$D(\omega)\equiv{\cal F}[{\rm III}_P]$$
Then using the Convolution theorem (see *note Convolution theorem::),
$F_s(\omega)$ can be written as:
$$F_s(\omega)={\cal F}[f(l){\rm III}_P]=F(\omega){\ast}D(\omega)$$
Finally, from the definition of convolution and the Fourier transform of
the Dirac comb (see *note Dirac delta and comb::), we get:
$$\eqalign{
F_s(\omega) &= \int_{-\infty}^{\infty}F(\omega)D(\omega-\mu)d\mu \cr
&= {1\over P}\displaystyle\sum_{n=-\infty}^{\infty}\int_{-\infty}^{\infty}F(\omega)\delta\left(\omega-\mu-{2{\pi}n\over P}\right)d\mu \cr
&= {1\over P}\displaystyle\sum_{n=-\infty}^{\infty}F\left(
\omega-{2{\pi}n\over P}\right).\cr }
$$
$F(\omega)$ was only a simple function, see *note Figure 6.3:
samplingfreq.(left). However, from the sampled Fourier transform
function we see that $F_s(\omega)$ is the superposition of infinite
copies of $F(\omega)$ that have been shifted, see *note Figure 6.3:
samplingfreq.(right). From the equation, it is clear that the shift in
each copy is $2\pi/P$.
../gnuastro-figures//samplingfreq.eps
Figure 6.3: Sampling causes infinite repetition in the frequency domain.
FT is an abbreviation for 'Fourier transform'. $\omega_m$ represents
the maximum frequency present in the input. $F(\omega)$ is only
symmetric on both sides of 0 when the input is real (not complex). In
general $F(\omega)$ is complex and thus cannot be simply plotted like
this. Here we have assumed a real Gaussian $f(t)$ which has produced a
Gaussian $F(\omega)$.
The input $f(l)$ can have any distribution of frequencies in it. In
the example of *note Figure 6.3: samplingfreq.(left), the input
consisted of a range of frequencies equal to $\Delta\omega=2\omega_m$.
Fortunately as *note Figure 6.3: samplingfreq.(right) shows, the assumed
pixel size ($P$) we used to sample this hypothetical function was such
that $2\pi/P>\Delta\omega$. The consequence is that each copy of
$F(\omega)$ has become completely separate from the surrounding copies.
Such a digitized (sampled) data set is thus called _over-sampled_. When
$2\pi/P=\Delta\omega$, $P$ is just small enough to finely separate even
the largest frequencies in the input signal and thus it is known as
_critically-sampled_. Finally if $2\pi/P<\Delta\omega$ we are dealing
with an _under-sampled_ data set. In an under-sampled data set, the
separate copies of $F(\omega)$ are going to overlap and this will
deprive us of recovering high constituent frequencies of $f(l)$. The
effects of under-sampling in an image with high rates of change (for
example, a brick wall imaged from a distance) can clearly be visually
seen and is known as _aliasing_.
When the input $f(l)$ is composed of a finite range of frequencies,
$f(l)$ is known as a _band-limited_ function. The example in *note
Figure 6.3: samplingfreq.(left) was a nice demonstration of such a case:
for all $\omega<-\omega_m$ or $\omega>\omega_m$, we have $F(\omega)=0$.
Therefore, when the input function is band-limited and our detector's
pixels are placed such that we have critically (or over-) sampled it,
then we can exactly reproduce the continuous $f(l)$ from the discrete or
digitized samples. To do that, we just have to isolate one copy of
$F(\omega)$ from the infinite copies and take its inverse Fourier
transform.
This ability to exactly reproduce the continuous input from the
sampled or digitized data leads us to the _sampling theorem_ which
connects the inherent property of the continuous signal (its maximum
frequency) to that of the detector (the spacing between its pixels).
The sampling theorem states that the full (continuous) signal can be
recovered when the pixel size ($P$) and the maximum constituent
frequency in the signal ($\omega_m$) have the following relation(1):
$${2\pi\over P}>2\omega_m$$
This relation was first formulated by Harry Nyquist (1889 - 1976 A.D.)
in 1928 and formally proved in 1949 by Claude E. Shannon (1916 - 2001
A.D.) in what is now known as the Nyquist-Shannon sampling theorem. In
signal processing, the signal is produced (synthesized) by a transmitter
and is received and de-coded (analyzed) by a receiver. Therefore
producing a band-limited signal is necessary.
In astronomy, we do not produce the shapes of our targets, we are
only observers. Galaxies can have any shape and size, therefore
ideally, our signal is not band-limited. However, since we are always
confined to observing through an aperture, the aperture will cause a
point source (for which $\omega_m=\infty$) to be spread over several
pixels. This spread is quantitatively known as the point spread
function or PSF. This spread does blur the image which is undesirable;
however, for this analysis it produces the positive outcome that there
will be a finite $\omega_m$. Though we should caution that any detector
will have noise which will add lots of very high frequency (ideally
infinite) changes between the pixels. However, the coefficients of
those noise frequencies are usually exceedingly small.
---------- Footnotes ----------
(1) This equation is also shown in some places without the $2\pi$.
Whether $2\pi$ is included or not depends on how you define the
frequency
6.3.2.8 Discrete Fourier transform
..................................
As we have stated several times so far, the input image is a digitized,
pixelated or discrete array of values ($f_s(l)$, see *note Sampling
theorem::). The input is not a continuous function. Also, all our
numerical calculations can only be done on a sampled, or discrete
Fourier transform. Note that $F_s(\omega)$ is not discrete, it is
continuous. One way would be to find the analytic $F_s(\omega)$, then
sample it at any desired "freq-pixel"(1) spacing. However, this process
would involve two steps of operations and computers in particular are
not too good at analytic operations for the first step. So here, we
will derive a method to directly find the 'freq-pixel'ated $F_s(\omega)$
from the pixelated $f_s(l)$. Let's start with the definition of the
Fourier transform (see *note Fourier transform::):
$$F_s(\omega)=\int_{-\infty}^{\infty}f_s(l)e^{-i{\omega}l}dl $$
From the definition of $f_s(\omega)$ (using $x$ instead of $n$) we get:
$$\eqalign{
F_s(\omega) &= \displaystyle\sum_{x=-\infty}^{\infty}
\int_{-\infty}^{\infty}f(l)\delta(l-xP)e^{-i{\omega}l}dl \cr
&= \displaystyle\sum_{x=-\infty}^{\infty}
f_xe^{-i{\omega}xP}
}
$$
Where $f_x$ is the value of $f(l)$ on the point $x$ or the value of the
$x$th pixel. As shown in *note Sampling theorem:: this function is
infinitely periodic with a period of $2\pi/P$. So all we need is the
values within one period: $0<\omega<2\pi/P$, see *note Figure 6.3:
samplingfreq. We want $X$ samples within this interval, so the
frequency difference between each frequency sample or freq-pixel is
$1/XP$. Hence we will evaluate the equation above on the points at:
$$\omega={u\over XP} \quad\quad u = 0, 1, 2, ..., X-1$$
Therefore the value of the freq-pixel $u$ in the frequency domain is:
$$F_u=\displaystyle\sum_{x=0}^{X-1} f_xe^{-i{ux\over X}} $$
Therefore, we see that for each freq-pixel in the frequency domain, we
are going to need all the pixels in the spatial domain(2). If the input
(spatial) pixel row is also $X$ pixels wide, then we can exactly recover
the $x$th pixel with the following summation:
$$f_x={1\over X}\displaystyle\sum_{u=0}^{X-1} F_ue^{i{ux\over X}} $$
When the input pixel row (we are still only working on 1D data) has
$X$ pixels, then it is $L=XP$ spatial units wide. $L$, or the length of
the input data was defined in *note Fourier series:: and $P$ or the
space between the pixels in the input was defined in *note Dirac delta
and comb::. As we saw in *note Sampling theorem::, the input (spatial)
pixel spacing ($P$) specifies the range of frequencies that can be
studied and in *note Fourier series:: we saw that the length of the
(spatial) input, ($L$) determines the resolution (or size of the
freq-pixels) in our discrete Fourier transformed image. Both result
from the fact that the frequency domain is the inverse of the spatial
domain.
---------- Footnotes ----------
(1) We are using the made-up word "freq-pixel" so they are not
confused with spatial domain "pixels".
(2) So even if one pixel is a blank pixel (see *note Blank pixels::),
all the pixels in the frequency domain will also be blank.
6.3.2.9 Fourier operations in two dimensions
............................................
Once all the relations in the previous sections have been clearly
understood in one dimension, it is very easy to generalize them to two
or even more dimensions since each dimension is by definition
independent. Previously we defined $l$ as the continuous variable in 1D
and the inverse of the period in its direction to be $\omega$. Let's
show the second spatial direction with $m$ the inverse of the period in
the second dimension with $\nu$. The Fourier transform in 2D (see *note
Fourier transform::) can be written as:
$$F(\omega, \nu)=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}
f(l, m)e^{-i({\omega}l+{\nu}m)}dl$$
$$f(l, m)=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}
F(\omega, \nu)e^{i({\omega}l+{\nu}m)}dl$$
The 2D Dirac $\delta(l,m)$ is non-zero only when $l=m=0$. The 2D
Dirac comb (or Dirac brush! See *note Dirac delta and comb::) can be
written in units of the 2D Dirac $\delta$. For most image detectors,
the sides of a pixel are equal in both dimensions. So $P$ remains
unchanged, if a specific device is used which has non-square pixels,
then for each dimension a different value should be used.
$${\rm III}_P(l, m)\equiv\displaystyle\sum_{j=-\infty}^{\infty}
\displaystyle\sum_{k=-\infty}^{\infty}
\delta(l-jP, m-kP) $$
The Two dimensional Sampling theorem (see *note Sampling theorem::)
is thus very easily derived as before since the frequencies in each
dimension are independent. Let's take $\nu_m$ as the maximum frequency
along the second dimension. Therefore the two dimensional sampling
theorem says that a 2D band-limited function can be recovered when the
following conditions hold(1):
$${2\pi\over P} > 2\omega_m \quad\quad\quad {\rm and}
\quad\quad\quad {2\pi\over P} > 2\nu_m$$
Finally, let's represent the pixel counter on the second dimension in
the spatial and frequency domains with $y$ and $v$ respectively. Also
let's assume that the input image has $Y$ pixels on the second
dimension. Then the two dimensional discrete Fourier transform and its
inverse (see *note Discrete Fourier transform::) can be written as:
$$F_{u,v}=\displaystyle\sum_{x=0}^{X-1}\displaystyle\sum_{y=0}^{Y-1}
f_{x,y}e^{-i({ux\over X}+{vy\over Y})} $$
$$f_{x,y}={1\over XY}\displaystyle\sum_{u=0}^{X-1}\displaystyle\sum_{v=0}^{Y-1}
F_{u,v}e^{i({ux\over X}+{vy\over Y})} $$
---------- Footnotes ----------
(1) If the pixels are not a square, then each dimension has to use
the respective pixel size, but since most detectors have square pixels,
we assume so here too
6.3.2.10 Edges in the frequency domain
......................................
With a good grasp of the frequency domain, we can revisit the problem of
convolution on the image edges, see *note Edges in the spatial domain::.
When we apply the convolution theorem (see *note Convolution theorem::)
to convolve an image, we first take the discrete Fourier transforms
(DFT, *note Discrete Fourier transform::) of both the input image and
the kernel, then we multiply them with each other and then take the
inverse DFT to construct the convolved image. Of course, in order to
multiply them with each other in the frequency domain, the two images
have to be the same size, so let's assume that we pad the kernel (it is
usually smaller than the input image) with zero valued pixels in both
dimensions so it becomes the same size as the input image before the
DFT.
Having multiplied the two DFTs, we now apply the inverse DFT which is
where the problem is usually created. If the DFT of the kernel only had
values of 1 (unrealistic condition!) then there would be no problem and
the inverse DFT of the multiplication would be identical with the input.
However in real situations, the kernel's DFT has a maximum of 1 (because
the sum of the kernel has to be one, see *note Convolution process::)
and decreases something like the hypothetical profile of *note Figure
6.3: samplingfreq. So when multiplied with the input image's DFT, the
coefficients or magnitudes (see *note Circles and the complex plane::)
of the smallest frequency (or the sum of the input image pixels) remains
unchanged, while the magnitudes of the higher frequencies are
significantly reduced.
As we saw in *note Sampling theorem::, the Fourier transform of a
discrete input will be infinitely repeated. In the final inverse DFT
step, the input is in the frequency domain (the multiplied DFT of the
input image and the kernel DFT). So the result (our output convolved
image) will be infinitely repeated in the spatial domain. In order to
accurately reconstruct the input image, we need all the frequencies with
the correct magnitudes. However, when the magnitudes of higher
frequencies are decreased, longer periods (shorter frequencies) will
dominate in the reconstructed pixel values. Therefore, when
constructing a pixel on the edge of the image, the newly empowered
longer periods will look beyond the input image edges and will find the
repeated input image there. So if you convolve an image in this fashion
using the convolution theorem, when a bright object exists on one edge
of the image, its blurred wings will be present on the other side of the
convolved image. This is often termed as circular convolution or cyclic
convolution.
So, as long as we are dealing with convolution in the frequency
domain, there is nothing we can do about the image edges. The least we
can do is to eliminate the ghosts of the other side of the image. So,
we add zero valued pixels to both the input image and the kernel in both
dimensions so the image that will be convolved has a size equal to the
sum of both images in each dimension. Of course, the effect of this
zero-padding is that the sides of the output convolved image will become
dark. To put it another way, the edges are going to drain the flux from
nearby objects. But at least it is consistent across all the edges of
the image and is predictable. In Convolve, you can see the padded
images when inspecting the frequency domain convolution steps with the
‘--viewfreqsteps’ option.
6.3.3 Spatial vs. Frequency domain
----------------------------------
With the discussions above it might not be clear when to choose the
spatial domain and when to choose the frequency domain. Here we will
try to list the benefits of each.
The spatial domain,
• Can correct for the edge effects of convolution, see *note Edges in
the spatial domain::.
• Can operate on blank pixels.
• Can be faster than frequency domain when the kernel is small (in
terms of the number of pixels on the sides).
The frequency domain,
• Will be much faster when the image and kernel are both large.
As a general rule of thumb, when working on an image of modeled profiles
use the frequency domain and when working on an image of real (observed)
objects use the spatial domain (corrected for the edges). The reason is
that if you apply a frequency domain convolution to a real image, you
are going to loose information on the edges and generally you do not
want large kernels. But when you have made the profiles in the image
yourself, you can just make a larger input image and crop the central
parts to completely remove the edge effect, see *note If convolving
afterwards::. Also due to oversampling, both the kernels and the images
can become very large and the speed boost of frequency domain
convolution will significantly improve the processing time, see *note
Oversampling::.
6.3.4 Convolution kernel
------------------------
All the programs that need convolution will need to be given a
convolution kernel file and extension. In most cases (other than
Convolve, see *note Convolve::) the kernel file name is optional.
However, the extension is necessary and must be specified either on the
command-line or at least one of the configuration files (see *note
Configuration files::). Within Gnuastro, there are two ways to create a
kernel image:
• MakeProfiles: You can use MakeProfiles to create a parametric
(based on a radial function) kernel, see *note MakeProfiles::. By
default MakeProfiles will make the Gaussian and Moffat profiles in
a separate file so you can feed it into any of the programs.
• ConvertType: You can write your own desired kernel into a text file
table and convert it to a FITS file with ConvertType, see *note
ConvertType::. Just be careful that the kernel has to have an odd
number of pixels along its two axes, see *note Convolution
process::. All the programs that do convolution will normalize the
kernel internally, so if you choose this option, you do not have to
worry about normalizing the kernel. Only within Convolve, there is
an option to disable normalization, see *note Invoking
astconvolve::.
The two options to specify a kernel file name and its extension are
shown below. These are common between all the programs that will do
convolution.
‘-k FITS’
‘--kernel=FITS’
The convolution kernel file name. The ‘BITPIX’ (data type) value
of this file can be any standard type and it does not necessarily
have to be normalized. Several operations will be done on the
kernel image prior to the program's processing:
• It will be converted to floating point type.
• All blank pixels (see *note Blank pixels::) will be set to
zero.
• It will be normalized so the sum of its pixels equal unity.
• It will be flipped so the convolved image has the same
orientation. This is only relevant if the kernel is not
circular. See *note Convolution process::.
‘-U STR’
‘--khdu=STR’
The convolution kernel HDU. Although the kernel file name is
optional, before running any of the programs, they need to have a
value for ‘--khdu’ even if the default kernel is to be used. So be
sure to keep its value in at least one of the configuration files
(see *note Configuration files::). By default, the system
configuration file has a value.
6.3.5 Invoking Convolve
-----------------------
Convolve an input dataset (2D image or 1D spectrum for example) with a
known kernel, or make the kernel necessary to match two PSFs. The
general template for Convolve is:
$ astconvolve [OPTION...] ASTRdata
One line examples:
## Convolve mockimg.fits with psf.fits:
$ astconvolve --kernel=psf.fits mockimg.fits
## Convolve in the spatial domain:
$ astconvolve observedimg.fits --kernel=psf.fits --domain=spatial
## Convolve a 3D cube (only spatial domain is supported in 3D).
## It is also necessary to define 3D tiles and channels for
## parallelization (see the Tessellation section for more).
$ astconvolve cube.fits --kernel=kernel3d.fits --domain=spatial \
--tilesize=30,30,30 --numchannels=1,1,1
## Find the kernel to match sharper and blurry PSF images (they both
## have to have the same pixel size).
$ astconvolve --kernel=sharperimage.fits --makekernel=10 \
blurryimage.fits
## Convolve a Spectrum (column 14 in the FITS table below) with a
## custom kernel (the kernel will be normalized internally, so only
## the ratios are important). Sed is used to replace the spaces with
## new line characters so Convolve sees them as values in one column.
$ echo "1 3 10 3 1" | sed 's/ /\n/g' | astconvolve spectra.fits -c14
The only argument accepted by Convolve is an input image file. Some
of the options are the same between Convolve and some other Gnuastro
programs. Therefore, to avoid repetition, they will not be repeated
here. For the full list of options shared by all Gnuastro programs,
please see *note Common options::. In particular, in the spatial
domain, on a multi-dimensional datasets, convolve uses Gnuastro's
tessellation to speed up the run, see *note Tessellation::. Common
options related to tessellation are described in *note Processing
options::.
1-dimensional datasets (for example, spectra) are only read as
columns within a table (see *note Tables:: for more on how Gnuastro
programs read tables). Note that currently 1D convolution is only
implemented in the spatial domain and thus kernel-matching is also not
supported.
Here we will only explain the options particular to Convolve. Run
Convolve with ‘--help’ in order to see the full list of options Convolve
accepts, irrespective of where they are explained in this book.
‘--kernelcolumn’
Column containing the 1D kernel. When the input dataset is a
1-dimensional column, and the host table has more than one column,
use this option to specify which column should be used.
‘--nokernelflip’
Do not flip the kernel after reading; only for spatial domain
convolution. This can be useful if the flipping has already been
applied to the kernel. By default, the input kernel is flipped to
avoid the output getting flipped; see *note Convolution process::.
‘--nokernelnorm’
Do not normalize the kernel after reading it, such that the sum of
its pixels is unity. As described in *note Convolution process::,
the kernel is normalized by default.
‘--conv-on-blank’
Do not ignore blank pixels in the convolution. The output pixels
that were originally non-blank are not affected by this option
(they will have the same value if this option is called or not).
This option just expands/dilates the non-blank regions of your
dataset into the blank regions and only works in spatial domain
convolution. Therefore, with this option convolution can be used
as a proxy for interpolation or dilation.
By default, blank pixels are ignored during spatial domain
convolution; so the input and output have exactly the same number
of blank pixels. With this option, the blank pixels that are
sufficiently close to non-blank pixels (based on the kernel) will
be given a value based on the non-blank elements that overlap with
the kernel for that blank pixel (see *note Edges in the spatial
domain::).
‘-d STR’
‘--domain=STR’
The domain to use for the convolution. The acceptable values are
'‘spatial’' and '‘frequency’', corresponding to the respective
domain.
For large images, the frequency domain process will be more
efficient than convolving in the spatial domain. However, the
edges of the image will loose some flux (see *note Edges in the
spatial domain::) and the image must not contain any blank pixels,
see *note Spatial vs. Frequency domain::.
‘--checkfreqsteps’
With this option a file with the initial name of the output file
will be created that is suffixed with ‘_freqsteps.fits’, all the
steps done to arrive at the final convolved image are saved as
extensions in this file. The extensions in order are:
1. The padded input image. In frequency domain convolution the
two images (input and convolved) have to be the same size and
both should be padded by zeros.
2. The padded kernel, similar to the above.
3. The Fourier spectrum of the forward Fourier transform of the
input image. Note that the Fourier transform is a complex
operation (and not view able in one image!) So we either have
to show the 'Fourier spectrum' or the 'Phase angle'. For the
complex number $a+ib$, the Fourier spectrum is defined as
$\sqrt{a^2+b^2}$ while the phase angle is defined as
$\arctan(b/a)$.
4. The Fourier spectrum of the forward Fourier transform of the
kernel image.
5. The Fourier spectrum of the multiplied (through complex
arithmetic) transformed images.
6. The inverse Fourier transform of the multiplied image. If you
open it, you will see that the convolved image is now in the
center, not on one side of the image as it started with (in
the padded image of the first extension). If you are working
on a mock image which originally had pixels of precisely 0.0,
you will notice that in those parts that your convolved
profile(s) did not convert, the values are now $\sim10^{-18}$,
this is due to floating-point round off errors. Therefore in
the final step (when cropping the central parts of the image),
we also remove any pixel with a value less than $10^{-17}$.
‘--noedgecorrection’
Do not correct the edge effect in spatial domain convolution (this
correction is done in spatial domain convolution by default). For
a full discussion, please see *note Edges in the spatial domain::.
‘-m INT’
‘--makekernel=INT’
If this option is called, Convolve will do PSF-matching: the output
will be the kernel that you should convolve with the sharper image
to obtain the blurry one (see *note Convolution theorem::). The
two images must have the same size (number of pixels). This option
is not yet supported in 1-dimensional datasets. In effect, it is
only necessary to give the two PSFs of your two datasets, find the
matching kernel based on that, then apply that kernel to the
higher-resolution (sharper image).
The image given to the ‘--kernel’ option is assumed to be the
sharper (less blurry) image and the input image (with no option) is
assumed to be the more blurry image. The value given to this
option will be used as the maximum radius of the kernel. Any pixel
in the final kernel that is larger than this distance from the
center will be set to zero.
Noise has large frequencies which can make the result less reliable
for the higher frequencies of the final result. So all the
frequencies which have a spectrum smaller than the value given to
the ‘minsharpspec’ option in the sharper input image are set to
zero and not divided. This will cause the wings of the final
kernel to be flatter than they would ideally be which will make the
convolved image result unreliable if it is too high.
Some notes to on how to prepare your two input PSFs. Note that
these (and several other issues that relate to an accurate
estimation of the PSF) are practically described in the following
tutorial: *note Building the extended PSF::.
• Choose a bright (unsaturated) star and use a region box (with
Crop for example, see *note Crop::) that is sufficiently above
the noise.
• Mask all background sources that may be nearby (you can use
Segment's clumps, see *note Segment::).
• Use Warp (see *note Warp::) to warp the pixel grid so the
star's center is exactly on the center of the central pixel in
the cropped image. This will certainly slightly degrade the
result, however, it is necessary. If there are multiple good
stars, you can shift all of them, then normalize them (so the
sum of each star's pixels is one) and then take their average
to decrease this effect.
• The shifting might move the center of the star by one pixel in
any direction, so crop the central pixel of the warped image
to have a clean image for the deconvolution.
‘-c’
‘--minsharpspec’
(‘=FLT’) The minimum frequency spectrum (or coefficient, or pixel
value in the frequency domain image) to use in deconvolution, see
the explanations under the ‘--makekernel’ option for more
information.
6.4 Warp
========
Image warping is the process of mapping the pixels of one image onto a
new pixel grid. This process is sometimes known as transformation,
however following the discussion of Heckbert 1989(1) we will not be
using that term because it can be confused with only pixel value or flux
transformations. Here we specifically mean the pixel grid
transformation which is better conveyed with 'warp'.
Image warping is a very important step in astronomy, both in
observational data analysis and in simulating modeled images. In
modeling, warping an image is necessary when we want to apply grid
transformations to the initial models, for example, in simulating
gravitational lensing. Observational reasons for warping an image are
listed below:
• *Noise:* Most scientifically interesting targets are inherently
faint (have a very low Signal to noise ratio). Therefore one short
exposure is not enough to detect such objects that are drowned
deeply in the noise. We need multiple exposures so we can add them
together and increase the objects' signal to noise ratio. Keeping
the telescope fixed on one field of the sky is practically
impossible. Therefore very deep observations have to put into the
same grid before adding them.
• *Resolution:* If we have multiple images of one patch of the sky
(hopefully at multiple orientations) we can warp them to the same
grid. The multiple orientations will allow us to 'guess' the
values of pixels on an output pixel grid that has smaller pixel
sizes and thus increase the resolution of the output. This process
of merging multiple observations is known as Mosaicing.
• *Cosmic rays:* Cosmic rays can randomly fall on any part of an
image. If they collide vertically with the camera, they are going
to create a very sharp and bright spot that in most cases can be
separated easily(2). However, depending on the depth of the camera
pixels, and the angle that a cosmic rays collides with it, it can
cover a line-like larger area on the CCD which makes the detection
using their sharp edges very hard and error prone. One of the best
methods to remove cosmic rays is to compare multiple images of the
same field. To do that, we need all the images to be on the same
pixel grid.
• *Optical distortion:* In wide field images, the optical distortion
that occurs on the outer parts of the focal plane will make
accurate comparison of the objects at various locations impossible.
It is therefore necessary to warp the image and correct for those
distortions prior to the analysis.
• *Detector not on focal plane:* In some cases (like the Hubble Space
Telescope ACS and WFC3 cameras), the CCD might be tilted compared
to the focal plane, therefore the recorded CCD pixels have to be
projected onto the focal plane before further analysis.
---------- Footnotes ----------
(1) Paul S. Heckbert. 1989. _Fundamentals of Texture mapping and
Image Warping_, Master's thesis at University of California, Berkeley.
(2) All astronomical targets are blurred with the PSF, see *note
PSF::, however a cosmic ray is not and so it is very sharp (it suddenly
stops at one pixel).
6.4.1 Linear warping basics
---------------------------
Let's take $\left[\matrix{u&v}\right]$ as the coordinates of a point in
the input image and $\left[\matrix{x&y}\right]$ as the coordinates of
that same point in the output image(1). The simplest form of coordinate
transformation (or warping) is the scaling of the coordinates, let's
assume we want to scale the first axis by $M$ and the second by $N$, the
output coordinates of that point can be calculated by
$$\left[\matrix{x\cr y}\right]=
\left[\matrix{Mu\cr Nv}\right]=
\left[\matrix{M&0\cr0&N}\right]\left[\matrix{u\cr v}\right]$$
Note that these are matrix multiplications. We thus see that we can
represent any such grid warping as a matrix. Another thing we can do
with this $2\times2$ matrix is to rotate the output coordinate around
the common center of both coordinates. If the output is rotated
anticlockwise by $\theta$ degrees from the positive (to the right)
horizontal axis, then the warping matrix should become:
$$\left[\matrix{x\cr y}\right]=
\left[\matrix{ucos\theta-vsin\theta\cr usin\theta+vcos\theta}\right]=
\left[\matrix{cos\theta&-sin\theta\cr sin\theta&cos\theta}\right]
\left[\matrix{u\cr v}\right]
$$
We can also flip the coordinates around the first axis, the second axis
and the coordinate center with the following three matrices
respectively:
$$\left[\matrix{1&0\cr0&-1}\right]\quad\quad
\left[\matrix{-1&0\cr0&1}\right]\quad\quad
\left[\matrix{-1&0\cr0&-1}\right]$$
The final thing we can do with this definition of a $2\times2$ warping
matrix is shear. If we want the output to be sheared along the first
axis with $A$ and along the second with $B$, then we can use the matrix:
$$\left[\matrix{1&A\cr B&1}\right]$$
To have one matrix representing any combination of these steps, you use
matrix multiplication, see *note Merging multiple warpings::. So any
combinations of these transformations can be displayed with one
$2\times2$ matrix:
$$\left[\matrix{a&b\cr c&d}\right]$$
The transformations above can cover a lot of the needs of most
coordinate transformations. However they are limited to mapping the
point $[\matrix{0&0}]$ to $[\matrix{0&0}]$. Therefore they are useless
if you want one coordinate to be shifted compared to the other one.
They are also space invariant, meaning that all the coordinates in the
image will receive the same transformation. In other words, all the
pixels in the output image will have the same area if placed over the
input image. So transformations which require varying output pixel
sizes like projections cannot be applied through this $2\times2$ matrix
either (for example, for the tilted ACS and WFC3 camera detectors on
board the Hubble space telescope).
To add these further capabilities, namely translation and projection,
we use the homogeneous coordinates. They were defined about 200 years
ago by August Ferdinand Möbius (1790 - 1868). For simplicity, we will
only discuss points on a 2D plane and avoid the complexities of higher
dimensions. We cannot provide a deep mathematical introduction here,
interested readers can get a more detailed explanation from Wikipedia(2)
and the references therein.
By adding an extra coordinate to a point we can add the flexibility
we need. The point $[\matrix{x&y}]$ can be represented as
$[\matrix{xZ&yZ&Z}]$ in homogeneous coordinates. Therefore multiplying
all the coordinates of a point in the homogeneous coordinates with a
constant will give the same point. Put another way, the point
$[\matrix{x&y&Z}]$ corresponds to the point $[\matrix{x/Z&y/Z}]$ on the
constant $Z$ plane. Setting $Z=1$, we get the input image plane, so
$[\matrix{u&v&1}]$ corresponds to $[\matrix{u&v}]$. With this
definition, the transformations above can be generally written as:
$$\left[\matrix{x\cr y\cr 1}\right]=
\left[\matrix{a&b&0\cr c&d&0\cr 0&0&1}\right]
\left[\matrix{u\cr v\cr 1}\right]$$
We thus acquired 4 extra degrees of freedom. By giving non-zero values
to the zero valued elements of the last column we can have translation
(try the matrix multiplication!). In general, any coordinate
transformation that is represented by the matrix below is known as an
affine transformation(3):
$$\left[\matrix{a&b&c\cr d&e&f\cr 0&0&1}\right]$$
We can now consider translation, but the affine transform is still
spatially invariant. Giving non-zero values to the other two elements
in the matrix above gives us the projective transformation or
Homography(4) which is the most general type of transformation with the
$3\times3$ matrix:
$$\left[\matrix{x'\cr y'\cr w}\right]=
\left[\matrix{a&b&c\cr d&e&f\cr g&h&1}\right]
\left[\matrix{u\cr v\cr 1}\right]$$
So the output coordinates can be calculated from:
$$x={x' \over w}={au+bv+c \over gu+hv+1}\quad\quad\quad\quad
y={y' \over w}={du+ev+f \over gu+hv+1}$$
Thus with Homography we can change the sizes of the output pixels on
the input plane, giving a 'perspective'-like visual impression. This
can be quantitatively seen in the two equations above. When $g=h=0$,
the denominator is independent of $u$ or $v$ and thus we have spatial
invariance. Homography preserves lines at all orientations. A very
useful fact about Homography is that its inverse is also a Homography.
These two properties play a very important role in the implementation of
this transformation. A short but instructive and illustrated review of
affine, projective and also bi-linear mappings is provided in Heckbert
1989(5).
---------- Footnotes ----------
(1) These can be any real number, we are not necessarily talking
about integer pixels here.
(2)
(3)
(4)
(5) Paul S. Heckbert. 1989. _Fundamentals of Texture mapping and
Image Warping_, Master's thesis at University of California, Berkeley.
Note that since points are defined as row vectors there, the matrix is
the transpose of the one discussed here.
6.4.2 Merging multiple warpings
-------------------------------
In *note Linear warping basics:: we saw how a basic warp/transformation
can be represented with a matrix. To make more complex warpings (for
example, to define a translation, rotation and scale as one warp) the
individual matrices have to be multiplied through matrix multiplication.
However matrix multiplication is not commutative, so the order of the
set of matrices you use for the multiplication is going to be very
important.
The first warping should be placed as the left-most matrix. The
second warping to the right of that and so on. The second
transformation is going to occur on the warped coordinates of the first.
As an example for merging a few transforms into one matrix, the
multiplication below represents the rotation of an image about a point
$[\matrix{U&V}]$ anticlockwise from the horizontal axis by an angle of
$\theta$. To do this, first we take the origin to $[\matrix{U&V}]$
through translation. Then we rotate the image, then we translate it
back to where it was initially. These three operations can be merged in
one operation by calculating the matrix multiplication below:
$$\left[\matrix{1&0&U\cr0&1&V\cr{}0&0&1}\right]
\left[\matrix{cos\theta&-sin\theta&0\cr sin\theta&cos\theta&0\cr 0&0&1}\right]
\left[\matrix{1&0&-U\cr0&1&-V\cr{}0&0&1}\right]$$
6.4.3 Resampling
----------------
A digital image is composed of discrete 'picture elements' or 'pixels'.
When a real image is created from a camera or detector, each pixel's
area is used to store the number of photo-electrons that were created
when incident photons collided with that pixel's surface area. This
process is called the 'sampling' of a continuous or analog data into
digital data.
When we change the pixel grid of an image, or "warp" it, we have to
calculate the flux value of each pixel on the new grid based on the old
grid, or resample it. Because of the calculation (as opposed to
observation), any form of warping on the data is going to degrade the
image and mix the original pixel values with each other. So if an
analysis can be done on an unwarped data image, it is best to leave the
image untouched and pursue the analysis. However as discussed in *note
Warp:: this is not possible in some scenarios and re-sampling is
necessary.
When the FWHM of the PSF of the camera is much larger than the pixel
scale (see *note Sampling theorem::) we are sampling the signal in a
much higher resolution than the camera can offer. This is usually the
case in many applications of image processing (nonastronomical imaging).
In such cases, we can consider each pixel to be a point and not an area:
the PSF doesn't vary much over a single pixel.
Approximating a pixel's area to a point can significantly speed up
the resampling and also the simplicity of the code. Because resampling
becomes a problem of interpolation: points of the input grid need to be
interpolated at certain other points (over the output grid). To
increase the accuracy, you might also sample more than one point from
within a pixel giving you more points for a more accurate interpolation
in the output grid.
However, interpolation has several problems. The first one is that
it will depend on the type of function you want to assume for the
interpolation. For example, you can choose a bi-linear or bi-cubic (the
'bi's are for the 2 dimensional nature of the data) interpolation
method. For the latter there are various ways to set the constants(1).
Such parametric interpolation functions can fail seriously on the edges
of an image, or when there is a sharp change in value (for example, the
bleeding saturation of bright stars in astronomical CCDs). They will
also need normalization so that the flux of the objects before and after
the warping is comparable.
The parametric nature of these methods adds a level of subjectivity
to the data (it makes more assumptions through the functions than the
data can handle). For most applications this is fine (as discussed
above: when the PSF is over-sampled), but in scientific applications
where we push our instruments to the limit and the aim is the detection
of the faintest possible galaxies or fainter parts of bright galaxies,
we cannot afford this loss. Because of these reasons Warp will not use
parametric interpolation techniques.
Warp will do interpolation based on "pixel mixing"(2) or "area
resampling". This is also similar to what the Hubble Space Telescope
pipeline calls "Drizzling"(3). This technique requires no functions, it
is thus non-parametric. It is also the closest we can get (make least
assumptions) to what actually happens on the detector pixels.
In pixel mixing, the basic idea is that you reverse-transform each
output pixel to find which pixels of the input image it covers, and what
fraction of the area of the input pixels are covered by that output
pixel. We then multiply each input pixel's value by the fraction of its
area that overlaps with the output pixel (between 0 to 1). The output's
pixel value is derived by summing all these multiplications for the
input pixels that it covers.
Through this process, pixels are treated as an area not as a point
(which is how detectors create the image), also the brightness (see
*note Brightness flux magnitude::) of an object will be fully preserved.
Since it involves the mixing of the input's pixel values, this pixel
mixing method is a form of *note Spatial domain convolution::.
Therefore, after comparing the input and output, you will notice that
the output is slightly smoothed, thus boosting the more diffuse signal,
but creating correlated noise. In astronomical imaging the correlated
noise will be decreased later when you stack many exposures(4).
If there are very high spatial-frequency signals in the image (for
example, fringes) which vary on a scale _smaller than_ your output image
pixel size (this is rarely the case in astronomical imaging), pixel
mixing can cause ailiasing(5). Therefore, in case such fringes are
present, they have to be calculated and removed separately (which would
naturally be done in any astronomical reduction pipeline). Because of
the PSF, no astronomical target has a sharp change in their signal.
Thus this issue is less important for astronomical applications, see
*note PSF::.
To find the overlap area of the output pixel over the input pixels,
we need to define polygons and clip them (find the overlap). Usually,
it is sufficient to define a pixel with a four-vertice polygon.
However, when a non-linear distortion (for example, ‘SIP’ or ‘TPV’) is
present and the distortion is significant over an output pixel's size
(usually far from the reference point), the shadow of the output pixel
on the input grid can be curved. To account for such cases (which can
only happen when correcting for non-linear distortions), Warp has the
‘--edgesampling’ option to sample the output pixel over more vertices.
For more, see the description of this option in *note Align pixels with
WCS considering distortions::.
---------- Footnotes ----------
(1) see for a nice
introduction.
(2) For a graphic demonstration see
.
(3)
(4) If you are working on a single exposure image and see pronounced
Moiré patterns after Warping, check *note Moire pattern in stacking and
its correction:: for a possible way to reduce them
(5)
6.4.4 Invoking Warp
-------------------
Warp will warp an input image into a new pixel grid by pixel mixing (see
*note Resampling::). Without any options, Warp will remove any
non-linear distortions from the image and align the output pixel
coordinates to its WCS coordinates. Any homographic warp (for example,
scaling, rotation, translation, projection, see *note Linear warping
basics::) can also be done by calling the relevant option explicitly.
The general template for invoking Warp is:
$ astwarp [OPTIONS...] InputImage
One line examples:
## Align image with celestial coordinates and remove any distortion
$ astwarp image.fits
## Align four exposures to same pixel grid and stack them with
## Arithmetic program's sigma-clipped mean operator (out of many
## stacking operators, see Arithmetic's documentation).
$ grid="--center=1.234,5.678 --width=1001,1001 --widthinpix --cdelt=0.2/3600"
$ astwarp a.fits $grid --output=A.fits
$ astwarp b.fits $grid --output=B.fits
$ astwarp c.fits $grid --output=C.fits
$ astwarp d.fits $grid --output=D.fits
$ astarithmetic A.fits B.fits C.fits D.fits 4 5 0.2 sigclip-mean \
-g1 --output=stack.fits
## Warp a previously created mock image to the same pixel grid as the
## real image (including any distortions).
$ astwarp mock.fits --gridfile=real.fits
## Rotate and then scale input image:
$ astwarp --rotate=37.92 --scale=0.8 image.fits
## Scale, then translate the input image:
$ astwarp --scale 8/3 --translate 2.1 image.fits
## Directly input a custom warping matrix (using fraction):
$ astwarp --matrix=1/5,0,4/10,0,1/5,4/10,0,0,1 image.fits
## Directly input a custom warping matrix, with final numbers:
$ astwarp --matrix="0.7071,-0.7071, 0.7071,0.7071" image.fits
If any processing is to be done, Warp needs to be given a 2D FITS
image. As in all Gnuastro programs, when an output is not explicitly
set with the ‘--output’ option, the output filename will be set
automatically based on the operation, see *note Automatic output::. For
the full list of general options to all Gnuastro programs (including
Warp), please see *note Common options::.
Warp uses pixel mixing to derive the pixel values of the output
image, see *note Resampling::. To be the most accurate, the input image
will be read as a 64-bit double precision floating point dataset and all
internal processing is done in this format. Upon writing, by default it
will be converted to 32-bit single precision floating point type (actual
observational data rarely have such precision!). In case you want a
different output type, you can use the ‘--type’ option that is common to
several Gnuastro programs. For example, if your input is a mock image
without noise, and you want to preserve the 64-bit precision, use (with
‘--type=float64’. Just note that the file size will also be double!
For more on the precision of various types, see *note Numeric data
types::.
By default (if no linear operation is requested), Warp will align the
pixel grid of the input image to the WCS coordinates it contains. This
operation and the the options that govern it are described in *note
Align pixels with WCS considering distortions::. You can Warp an input
image to the same pixel grid as a reference FITS file using the
‘--wcsfile’ option. In this case, the output image will take all the
information needed from the reference WCS file and HDU/extension
specified with ‘--wcshdu’, thus it will discard any other resampling
options given.
If you need any custom linear warping (independent of the WCS, see
*note Linear warping basics::), you need to call the respective
operation manually. These are described in *note Linear warps to be
called explicitly::. Please note that you may not use both linear and
non-linear modes simultaneously. For example, you cannot scale or
rotate the image while removing its non-linear distortions at the same
time.
The following options are shared between both modes:
‘--hstartwcs=INT’
Specify the first header keyword number (line) that should be used
to read the WCS information, see the full explanation in *note
Invoking astcrop::.
‘--hendwcs=INT’
Specify the last header keyword number (line) that should be used
to read the WCS information, see the full explanation in *note
Invoking astcrop::.
‘-C FLT’
‘--coveredfrac=FLT’
Depending on the warp, the output pixels that cover pixels on the
edge of the input image, or blank pixels in the input image, are
not going to be fully covered by input data. With this option, you
can specify the acceptable covered fraction of such pixels (any
value between 0 and 1). If you only want output pixels that are
fully covered by the input image area (and are not blank), then you
can set ‘--coveredfrac=1’ (which is the default!). Alternatively,
a value of ‘0’ will keep output pixels that are even
infinitesimally covered by the input. As a result, with
‘--coveredfrac=0’, the sum of the pixels in the input and output
images will be exactly the same.
6.4.4.1 Align pixels with WCS considering distortions
.....................................................
When none of the linear warps(1) are requested, Warp will align the
input's pixel axes with it's WCS axes. In the process, any possibly
existing distortion is also removed (such as ‘TPV’ and ‘SIP’). Usually,
the WCS axes are the Right Ascension and Declination in equatorial
coordinates. The output image's pixel grid is highly customizable
through the options in this section. To learn about Warp's strategy to
build the new pixel grid, see *note Resampling::. For strong
distortions (that produce strong curvatures), you can fine-tune the
area-based resampling with ‘--edgesampling’, as described below.
On the other hand, sometimes you need to Warp an input image to the
exact same grid of an already available reference FITS image with an
existing WCS. If that image is already aligned, finding its center,
number of pixels and pixel scale can be annoying (and just increase the
complexity of your script). On the other hand, if that image is not
aligned (for example, has a certain rotation in the sky, and has a
different distortion), there are too many WCS parameters to set (some
are not yet available explicitly in the options here)! For such
scenarios, Warp has the ‘--gridfile’ option. When ‘--gridfile’ is
called, the options below that are used to define the output's WCS will
be ignored (these options: ‘--center’, ‘--widthinpix’, ‘--cdelt’,
‘--ctype’). In this case, the output's WCS and pixel grid will exactly
match the image given to ‘--gridfile’ (including any rotation, pixel
scale, or distortion or projection).
*Set ‘--cdelt’ explicitly when you plan to stack many warped images:* To
align some images and later stack them, it is necessary to be sure the
pixel sizes of all the images are the same exactly. Most of the time
the measured (during astrometry) pixel scale of the separate exposures,
will be different in the second or third digit number after the decimal
point. It is a normal/statistical error in measuring the astrometry.
On a large image, these slight differences can cause different output
sizes (of one or two pixels on a very large image).
You can fix this by explicitly setting the pixel scale of each warped
exposure with Warp's ‘--cdelt’ option that is described below. For good
strategies of setting the pixel scale, see *note Moire pattern in
stacking and its correction::.
Another problem that may arise when aligning images to new pixel
grids is the aliasing or visible Moiré patterns on the output image.
This artifact should be removed if you are stacking several exposures,
especially with a pointing pattern. If not see *note Moire pattern in
stacking and its correction:: for ways to mitigate the visible patterns.
See the description of ‘--gridfile’ below for more.
*Known issue:* Warp's WCS-based aligning works best with WCSLIB version
7.12 (released in September 2022) and above. If you have an older
version of WCSLIB, you might get a ‘wcss2p’ error otherwise.
‘-c FLT,FLT’
‘--center=FLT,FLT’
WCS coordinates of the center of the central pixel of the output
image. Since a central pixel is only defined with an odd number of
pixels along both dimensions, the output will always have an odd
number of pixels. When ‘--center’ or ‘--gridfile’ aren't given,
the output will have the same central WCS coordinate as the input.
Usually, the WCS coordinates are Right Ascension and Declination
(when the first three characters of ‘CTYPE1’ and ‘CTYPE2’ are
respectively ‘RA-’ and ‘DEC’). For more on the ‘CTYPEi’ keyword
values, see ‘--ctype’ below.
‘-w INT[,INT]’
‘--width=INT[,INT]’
Width and height of the output image in units of WCS (usually
degrees). If you want the values to be read as pixels, also call
the ‘--widthinpix’ option with ‘--width’. If a single value is
given, Warp will use the same value for the second dimension
(creating a square output). When ‘--width’ or ‘--gridfile’ aren't
given, Warp will calculate the necessary size of the output pixel
grid to fully contain the input image.
Usually the WCS coordinates are in units of degrees (defined by the
‘CUNITi’ keywords of the FITS standard). But entering a certain
number of arcseconds or arcminutes for the width can be annoying
(you will usually need to go to the calculator!). To simplify such
situations, this option also accepts division. For example
‘--width=1/60,2/60’ will make an aligned warp that is 1 arcmin
along Right Ascension and 2 arcminutes along the Declination.
With the ‘--widthinpix’ option the values will be interpreted as
numbers of pixels. In this scenario, this option should be given
_odd_ integer(s) that are greater than 1. This ensures that the
output image can have a _central_ pixel. Recall that through the
‘--center’ option, you specify the WCS coordinate of the center of
the central pixel. The central coordinate of an image with an even
number of pixels will be on the edge of two pixels, so a "central"
pixel is not well defined. If any of the given values are even,
Warp will automatically add a single pixel (to make it an odd
integer) and print a warning message.
‘--widthinpix’
When called, the values given to the ‘--width’ option will be
interpreted as the number of pixels along each dimension(s). See
the description of ‘--width’ for more.
‘-x FLT[,FLT]’
‘--cdelt=FLT[,FLT]’
Coordinate deltas or increments (‘CDELTi’ in the FITS standard), or
the pixel scale in both dimensions. If a single value is given, it
will be used for both axes. In this way, the output's pixels will
be squares on the sky at the reference point (as is usually
expected!). When ‘--cdelt’ or ‘--gridfile’ aren't given, Warp will
read the input's pixel scale and choose the larger of ‘CDELT1’ or
‘CDELT2’ so the output pixels are square.
Usually (when dealing with RA and Dec, and the ‘CUNITi’s have a
value of ‘deg’), the units of the given values are degrees/pixel.
Warp allows you to easily convert from _arcsec_ to _degrees_ by
simply appending a ‘/3600’ to the value. For example, for an
output image of pixel scale ‘0.27’ arcsec/pixel, you can use
‘--cdelt=0.27/3600’.
‘--ctype=STR,STR’
The coordinate types of the output (‘CTYPE1’ and ‘CTYPE2’ keywords
in the FITS standard), separated by a comma. By default the value
to this option is '‘RA---TAN,DEC--TAN’'. However, if ‘--gridfile’
is given, this option is ignored.
If you don't call ‘--ctype’ or ‘--gridfile’, the output WCS
coordinates will be Right Ascension and Declination, while the
output's projection will be Gnomonic
(https://en.wikipedia.org/wiki/Gnomonic_projection), also known as
Tangential (TAN). This combination is the most common in
extra-galactic imaging surveys. For other coordinates and
projections in your output use other values, as described below.
According to the FITS standard version 4.0(2): ‘CTYPEi’ is the
"type for the Intermediate-coordinate Axis $i$. Any coordinate
type that is not covered by this Standard or an officially
recognized FITS convention shall be taken to be linear. All
non-linear coordinate system names must be expressed in '4–3' form:
the first four characters specify the coordinate type, the fifth
character is a hyphen (‘-’), and the remaining three characters
specify an algorithm code for computing the world coordinate value.
Coordinate types with names of fewer than four characters are
padded on the right with hyphens, and algorithm codes with fewer
than three characters are padded on the right with SPACE. Algorithm
codes should be three characters" (see list of algorithm codes
below).
You can use any of the projection algorithms (last three characters
of each coordinate's type) provided by your host WCSLIB (a
mandatory dependency of Gnuastro; see *note WCSLIB::). For a very
elaborate and complete description of projection algorithms in the
FITS WCS standard, see Calabretta and Greisen 2002
(https://doi.org/10.1051/0004-6361:20021327). Wikipedia also has a
nice article on Map projections
(https://en.wikipedia.org/wiki/Map_projection). As an example,
WCSLIB 7.12 (released in September 2022) has the following
projection algorithms:
‘AZP’
Zenithal/azimuthal perspective
‘SZP’
Slant zenithal perspective
‘TAN’
Gnomonic (tangential)
‘STG’
Stereographic
‘SIN’
Orthographic/synthesis
‘ARC’
Zenithal/azimuthal equidistant
‘ZPN’
Zenithal/azimuthal polynomial
‘ZEA’
Zenithal/azimuthal equal area
‘AIR’
Airy
‘CYP’
Cylindrical perspective
‘CEA’
Cylindrical equal area
‘CAR’
Plate carree
‘MER’
Mercator
‘SFL’
Sanson-Flamsteed
‘PAR’
Parabolic
‘MOL’
Mollweide
‘AIT’
Hammer-Aitoff
‘COP’
Conic perspective
‘COE’
Conic equal area
‘COD’
Conic equidistant
‘COO’
Conic orthomorphic
‘BON’
Bonne
‘PCO’
Polyconic
‘TSC’
Tangential spherical cube
‘CSC’
COBE spherical cube
‘QSC’
Quadrilateralized spherical cube
‘HPX’
HEALPix
‘XPH’
HEALPix polar, aka "butterfly"
‘-G’
‘--gridfile’
FITS filename containing the final pixel grid and WCS for the
output image. The HDU/extension containing should be specified
with ‘--gridhdu’ or its short option ‘-H’. The HDU should contain
a WCS, otherwise, Warp will abort with a crash. When this option
is used, Warp will read the respective WCS and the size of the
image to resample the input. Since this WCS of this HDU contains
everything needed to construct the WCS the options above will be
ignored when ‘--gridfile’ is called: ‘--cdelt’, ‘--center’, and
‘--widthinpix’.
In the example below, let's use this option to put the image of M51
in one survey (J-PLUS) into the pixel grid of another survey (SDSS)
containing M51. The J-PLUS field of view is very large (almost
$1.5\times1.5$ deg$^2$, in $9500\times9500$ pixels), while the
field of view of SDSS in each filter is small (almost
$0.3\times0.25$ deg$^2$ in $2048\times1489$ pixels). With the
first two commands, we'll first download the two images, then we'll
extract the portion of the J-PLUS image that overlaps with the SDSS
image and align it exactly to SDSS's pixel grid. Note that these
are the two images that were used in two of Gnuastro's tutorials:
*note Building the extended PSF:: and *note Detecting large
extended targets::.
## Download the J-PLUS DR2 image of M51 in the r filter.
$ jplusbase="http://archive.cefca.es/catalogues/vo/siap"
$ wget $jplusbase/jplus-dr2/get_fits?id=67510 \
-O jplus.fits.fz
## Download the SDSS image in r filter and decompress it
## (Bzip2 is not a standard FITS compression algorithm).
$ sdssbase=https://dr12.sdss.org/sas/dr12/boss/photoObj/frames
$ wget $sdssbase/301/3716/6/frame-r-003716-6-0117.fits.bz2 \
-O sdss.fits.bz2
$ bunzip2 sdss.fits.bz2
## Warp and crop the J-PLUS image so the output exactly
## matches the SDSS pixel gid.
$ astwarp jplus.fits.fz --gridfile=sdss.fits --gridhdu=0 \
--output=jplus-on-sdss.fits
## View the two images side-by-side:
$ astscript-fits-view sdss.fits jplus-on-sdss.fits
As the example above shows, this option can therefore be very
useful when comparing images from multiple surveys. But there are
other very interesting use cases also. For example, when you are
making a mock dataset and need to add distortion to the image so it
matches the distortion of your camera. Through ‘--gridhdu’, you
can easily insert that distortion over the mock image and put the
mock image in the pixel grid of an exposure.
‘-H’
‘--gridhdu’
The HDU/extension of the reference WCS file specified with option
‘--wcsfile’ or its short version ‘-H’ (see the description of
‘--wcsfile’ for more).
‘--edgesampling=INT’
Number of extra samplings along the edge of a pixel. By default
the value is ‘0’ (the output pixel's polygon over the input will be
a quadrilateral (a polygon with four edges/vertices).
Warp uses pixel mixing to derive the output pixel values. For a
complete introduction, see *note Resampling::, and in particular
its later part on distortions. To account for this possible
curvature due to distortion, you can use this option. For example,
‘--edgesampling=1’ will add one extra vertice in the middle of each
edge of the output pixel, producing an 8-vertice polygon.
Similarly, ‘--edgesampling=5’ will put 5 extra vertices along each
edge, thus sampling the shape (and possible curvature) of the
output pixel over an input pixel with $4+5\times4=24$ vertice
polygon. Since the polygon clipping will happen for every output
pixel, a higher value to this option can significantly reduce the
running speed and increase the RAM usage of Warp; so use it with
caution: in most cases the default ‘--edgesampling=0’ is
sufficient.
To visually inspect the curvature effect on pixel area of the input
image, see option ‘--pixelareaonwcs’ in *note Pixel information
images::.
‘--checkmaxfrac’
Check each output pixel's maximum coverage on the input data and
append as the '‘MAX-FRAC’' HDU/extension to the output aligned
image. This option provides an easy visual inspection for possible
recurring patterns or fringes caused by aligning to a new pixel
grid. For more detail about the origin of these patterns and how
to mitigate them see *note Moire pattern in stacking and its
correction::.
Note that the '‘MAX-FRAC’' HDU/extension is not showing the
patterns themselves; It represents the largest area coverage on the
input data for that particular pixel. The values can be in the
range between 0 to 1, where 1 means the pixel is covering at least
one complete pixel of the input data. On the other hand, 0 means
that the pixel is not covering any pixels of the input at all.
---------- Footnotes ----------
(1) For linear warps, see *note Linear warps to be called
explicitly::.
(2) FITS standard version 4.0:
6.4.4.2 Linear warps to be called explicitly
............................................
Linear warps include operations like rotation, scaling, sheer, etc. For
an introduction, see *note Linear warping basics::. These are warps
that don't depend on the WCS of the image and should be explicitly
requested. To align the input pixel coordinates with the WCS
coordinates, see *note Align pixels with WCS considering distortions::.
While they will correct any existing WCS based on the warp, they can
also operate on images without any WCS. For example, you have a mock
image that doesn't (yet!) have its mock WCS, and it has been created on
an over-sampled grid and convolved with an over-sampled PSF. In this
scenario, you can use the ‘--scale’ option to under-sample it to your
desired resolution. This is similar to the *note Sufi simulates a
detection:: tutorial.
Linear warps must be specified as command-line options, either as
(possibly multiple) modular warpings (for example, ‘--rotate’, or
‘--scale’), or directly as a single raw matrix (with ‘--matrix’). If
specified together, the latter (direct matrix) will take precedence and
all the modular warpings will be ignored. Any number of modular
warpings can be specified on the command-line and configuration files.
If more than one modular warping is given, all will be merged to create
one warping matrix. As described in *note Merging multiple warpings::,
matrix multiplication is not commutative, so the order of specifying the
modular warpings on the command-line, and/or configuration files makes a
difference (see *note Configuration file precedence::). The full list
of modular warpings and the other options particular to Warp are
described below.
The values to the warping options (modular warpings as well as
‘--matrix’), are a sequence of at least one number. Each number in this
sequence is separated from the next by a comma (<,>). Each number can
also be written as a single fraction (with a forward-slash between
the numerator and denominator). Space and Tab characters are permitted
between any two numbers, just do not forget to quote the whole value.
Otherwise, the value will not be fully passed onto the option. See the
examples above as a demonstration.
Based on the FITS standard, integer values are assigned to the center
of a pixel and the coordinate [1.0, 1.0] is the center of the first
pixel (bottom left of the image when viewed in SAO DS9). So the
coordinate center [0.0, 0.0] is half a pixel away (in each axis) from
the bottom left vertex of the first pixel. The resampling that is done
in Warp (see *note Resampling::) is done on the coordinate axes and thus
directly depends on the coordinate center. In some situations this if
fine, for example, when rotating/aligning a real image, all the edge
pixels will be similarly affected. But in other situations (for
example, when scaling an over-sampled mock image to its intended
resolution, this is not desired: you want the center of the coordinates
to be on the corner of the pixel. In such cases, you can use the
‘--centeroncorner’ option which will shift the center by $0.5$ before
the main warp, then shift it back by $-0.5$ after the main warp.
‘-r FLT’
‘--rotate=FLT’
Rotate the input image by the given angle in degrees: $\theta$ in
*note Linear warping basics::. Note that commonly, the WCS
structure of the image is set such that the RA is the inverse of
the image horizontal axis which increases towards the right in the
FITS standard and as viewed by SAO DS9. So the default center for
rotation is on the right of the image. If you want to rotate about
other points, you have to translate the warping center first (with
‘--translate’) then apply your rotation and then return the center
back to the original position (with another call to ‘--translate’,
see *note Merging multiple warpings::.
‘-s FLT[,FLT]’
‘--scale=FLT[,FLT]’
Scale the input image by the given factor(s): $M$ and $N$ in *note
Linear warping basics::. If only one value is given, then both
image axes will be scaled with the given value. When two values
are given (separated by a comma), the first will be used to scale
the first axis and the second will be used for the second axis. If
you only need to scale one axis, use ‘1’ for the axis you do not
need to scale. The value(s) can also be written (on the
command-line or in configuration files) as a fraction.
‘-f FLT[,FLT]’
‘--flip=FLT[,FLT]’
Flip the input image around the given axis(s). If only one value
is given, then both image axes are flipped. When two values are
given (separated by acomma), you can choose which axis to flip
over. ‘--flip’ only takes values ‘0’ (for no flip), or ‘1’ (for a
flip). Hence, if you want to flip by the second axis only, use
‘--flip=0,1’.
‘-e FLT[,FLT]’
‘--shear=FLT[,FLT]’
Shear the input image by the given value(s): $A$ and $B$ in *note
Linear warping basics::. If only one value is given, then both
image axes will be sheared with the given value. When two values
are given (separated by a comma), the first will be used to shear
the first axis and the second will be used for the second axis. If
you only need to shear along one axis, use ‘0’ for the axis that
must be untouched. The value(s) can also be written (on the
command-line or in configuration files) as a fraction.
‘-t FLT[,FLT]’
‘--translate=FLT[,FLT]’
Translate (move the center of coordinates) the input image by the
given value(s): $c$ and $f$ in *note Linear warping basics::. If
only one value is given, then both image axes will be translated by
the given value. When two values are given (separated by a comma),
the first will be used to translate the first axis and the second
will be used for the second axis. If you only need to translate
along one axis, use ‘0’ for the axis that must be untouched. The
value(s) can also be written (on the command-line or in
configuration files) as a fraction.
‘-p FLT[,FLT]’
‘--project=FLT[,FLT]’
Apply a projection to the input image by the given values(s): $g$
and $h$ in *note Linear warping basics::. If only one value is
given, then projection will apply to both axes with the given
value. When two values are given (separated by a comma), the first
will be used to project the first axis and the second will be used
for the second axis. If you only need to project along one axis,
use ‘0’ for the axis that must be untouched. The value(s) can also
be written (on the command-line or in configuration files) as a
fraction.
‘-m STR’
‘--matrix=STR’
The warp/transformation matrix. All the elements in this matrix
must be separated by commas(<,>) characters and as described above,
you can also use fractions (a forward-slash between two numbers).
The transformation matrix can be either a 2 by 2 (4 numbers), or a
3 by 3 (9 numbers) array. In the former case (if a 2 by 2 matrix
is given), then it is put into a 3 by 3 matrix (see *note Linear
warping basics::).
The determinant of the matrix has to be non-zero and it must not
contain any non-number values (for example, infinities or NaNs).
The elements of the matrix have to be written row by row. So for
the general Homography matrix of *note Linear warping basics::, it
should be called with ‘--matrix=a,b,c,d,e,f,g,h,1’.
The raw matrix takes precedence over all the modular warping
options listed above, so if it is called with any number of modular
warps, the latter are ignored.
‘--centeroncorner’
Put the center of coordinates on the corner of the first
(bottom-left when viewed in SAO DS9) pixel. This option is applied
after the final warping matrix has been finalized: either through
modular warpings or the raw matrix. See the explanation above for
coordinates in the FITS standard to better understand this option
and when it should be used.
‘-k’
‘--keepwcs’
Do not correct the WCS information of the input image and save it
untouched to the output image. By default the WCS (World
Coordinate System) information of the input image is going to be
corrected in the output image so the objects in the image are at
the same WCS coordinates. But in some cases it might be useful to
keep it unchanged (for example, to correct alignments).
7 Data analysis
***************
Astronomical datasets (images or tables) contain very valuable
information, the tools in this section can help in analyzing,
extracting, and quantifying that information. For example, getting
general or specific statistics of the dataset (with *note Statistics::),
detecting signal within a noisy dataset (with *note NoiseChisel::), or
creating a catalog from an input dataset (with *note MakeCatalog::).
7.1 Statistics
==============
The distribution of values in a dataset can provide valuable information
about it. For example, in an image, if it is a positively skewed
distribution, we can see that there is significant data in the image.
If the distribution is roughly symmetric, we can tell that there is no
significant data in the image. In a table, when we need to select a
sample of objects, it is important to first get a general view of the
whole sample.
On the other hand, you might need to know certain statistical
parameters of the dataset. For example, if we have run a detection
algorithm on an image, and we want to see how accurate it was, one
method is to calculate the average of the undetected pixels and see how
reasonable it is (if detection is done correctly, the average of
undetected pixels should be approximately equal to the background value,
see *note Sky value::). In a table, you might have calculated the
magnitudes of a certain class of objects and want to get some general
characteristics of the distribution immediately on the command-line
(very fast!), to possibly change some parameters. The Statistics
program is designed for such situations.
7.1.1 Histogram and Cumulative Frequency Plot
---------------------------------------------
Histograms and the cumulative frequency plots are both used to visually
study the distribution of a dataset. A histogram shows the number of
data points which lie within pre-defined intervals (bins). So on the
horizontal axis we have the bin centers and on the vertical, the number
of points that are in that bin. You can use it to get a general view of
the distribution: which values have been repeated the most? how
close/far are the most significant bins? Are there more values in the
larger part of the range of the dataset, or in the lower part?
Similarly, many very important properties about the dataset can be
deduced from a visual inspection of the histogram. In the Statistics
program, the histogram can be either output to a table to plot with your
favorite plotting program(1), or it can be shown with ASCII characters
on the command-line, which is very crude, but good enough for a fast and
on-the-go analysis, see the example in *note Invoking aststatistics::.
The width of the bins is only necessary parameter for a histogram.
In the limiting case that the bin-widths tend to zero (while assuming
the number of points in the dataset tend to infinity), then the
histogram will tend to the probability density function
(https://en.wikipedia.org/wiki/Probability_density_function) of the
distribution. When the absolute number of points in each bin is not
relevant to the study (only the shape of the histogram is important),
you can _normalize_ a histogram so like the probability density
function, the sum of all its bins will be one.
In the cumulative frequency plot of a distribution, the horizontal
axis is the sorted data values and the y axis is the index of each data
in the sorted distribution. Unlike a histogram, a cumulative frequency
plot does not involve intervals or bins. This makes it less prone to
any sort of bias or error that a given bin-width would have on the
analysis. When a larger number of the data points have roughly the same
value, then the cumulative frequency plot will become steep in that
vicinity. This occurs because on the horizontal axis, there is little
change while on the vertical axis, the indexes constantly increase.
Normalizing a cumulative frequency plot means to divide each index (y
axis) by the total number of data points (or the last value).
Unlike the histogram which has a limited number of bins, ideally the
cumulative frequency plot should have one point for every data element.
Even in small datasets (for example, a $200\times200$ image) this will
result in an unreasonably large number of points to plot (40000)! As a
result, for practical reasons, it is common to only store its value on a
certain number of points (intervals) in the input range rather than the
whole dataset, so you should determine the number of bins you want when
asking for a cumulative frequency plot. In Gnuastro (and thus the
Statistics program), the number reported for each bin is the total
number of data points until the larger interval value for that bin. You
can see an example histogram and cumulative frequency plot of a single
dataset under the ‘--asciihist’ and ‘--asciicfp’ options of *note
Invoking aststatistics::.
So as a summary, both the histogram and cumulative frequency plot in
Statistics will work with bins. Within each bin/interval, the lower
value is considered to be within then bin (it is inclusive), but its
larger value is not (it is exclusive). Formally, an interval/bin
between a and b is represented by [a, b). When the over-all range of
the dataset is specified (with the ‘--greaterequal’, ‘--lessthan’, or
‘--qrange’ options), the acceptable values of the dataset are also
defined with a similar inclusive-exclusive manner. But when the range
is determined from the actual dataset (none of these options is called),
the last element in the dataset is included in the last bin's count.
---------- Footnotes ----------
(1) We recommend PGFPlots (http://pgfplots.sourceforge.net/) which
generates your plots directly within TeX (the same tool that generates
your document).
7.1.2 2D Histograms
-------------------
In *note Histogram and Cumulative Frequency Plot:: the concept of
histograms were introduced on a single dataset. But they are only
useful for viewing the distribution of a single variable (column in a
table). In many contexts, the distribution of two variables in relation
to each other may be of interest. For example, the color-magnitude
diagrams in astronomy, where the horizontal axis is the luminosity or
magnitude of an object, and the vertical axis is the color. Scatter
plots are useful to see these relations between the objects of interest
when the number of the objects is small.
As the density of points in the scatter plot increases, the points
will fall over each other and just make a large connected region hide
potentially interesting behaviors/correlations in the densest regions.
This is where 2D histograms can become very useful. A 2D histogram is
composed of 2D bins (boxes or pixels), just as a 1D histogram consists
of 1D bins (lines). The number of points falling within each box/pixel
will then be the value of that box. Added with a color-bar, you can now
clearly see the distribution independent of the density of points (for
example, you can even visualize it in log-scale if you want).
Gnuastro's Statistics program has the ‘--histogram2d’ option for this
task. It takes a single argument (either ‘table’ or ‘image’) that
specifies the format of the output 2D histogram. The two formats will
be reviewed separately in the sub-sections below. But let's start with
the generalities that are common to both (related to the input, not the
output).
You can specify the two columns to be shown using the ‘--column’ (or
‘-c’) option. So if you want to plot the color-magnitude diagram from a
table with the ‘MAG-R’ column on the horizontal and ‘COLOR-G-R’ on the
vertical column, you can use ‘--column=MAG-r,COLOR-G-r’. The number of
bins along each dimension can be set with ‘--numbins’ (for first input
column) and ‘--numbins2’ (for second input column).
Without specifying any range, the full range of values will be used
in each dimension. If you only want to focus on a certain interval of
the values in the columns in any dimension you can use the
‘--greaterequal’ and ‘--lessthan’ options to limit the values along the
first/horizontal dimension and ‘--greaterequal2’ and ‘--lessthan2’
options for the second/vertical dimension.
7.1.2.1 2D histogram as a table for plotting
............................................
When called with the ‘--histogram=table’ option, Statistics will output
a table file with three columns that have the information of every box
as a column. If you asked for ‘--numbins=N’ and ‘--numbins2=M’, all
three columns will have $M\times N$ rows (one row for every box/pixel of
the 2D histogram). The first and second columns are the position of the
box along the first and second dimensions. The third column has the
number of input points that fall within that box/pixel.
For example, you can make high-quality plots within your paper (using
the same LaTeX engine, thus blending very nicely with your text) using
PGFPlots (https://ctan.org/pkg/pgfplots). Below you can see one such
minimal example, using your favorite text editor, save it into a file,
make the two small corrections in it, then run the commands shown at the
top. This assumes that you have LaTeX installed, if not the steps to
install a minimally sufficient LaTeX package on your system, see the
respective section in *note Bootstrapping dependencies::.
The two parts that need to be corrected are marked with '‘%% <--’':
the first one (‘XXXXXXXXX’) should be replaced by the value to the
‘--numbins’ option which is the number of bins along the first
dimension. The second one (‘FILE.txt’) should be replaced with the name
of the file generated by Statistics.
%% Replace 'XXXXXXXXX' with your selected number of bins in the first
%% dimension.
%%
%% Then run these commands to build the plot in a LaTeX command.
%% mkdir tikz
%% pdflatex --shell-escape --halt-on-error report.tex
\documentclass{article}
%% Load PGFPlots and set it to build the figure separately in a 'tikz'
%% directory (which has to exist before LaTeX is run). This
%% "externalization" is very useful to include the commands of multiple
%% plots in the middle of your paper/report, but also have the plots
%% separately to use in slides or other scenarios.
\usepackage{pgfplots}
\usetikzlibrary{external}
\tikzexternalize
\tikzsetexternalprefix{tikz/}
%% Define colormap for the PGFPlots 2D histogram
\pgfplotsset{
/pgfplots/colormap={hsvwhitestart}{
rgb255(0cm)=(255,255,255)
rgb255(0.10cm)=(128,0,128)
rgb255(0.5cm)=(0,0,230)
rgb255(1.cm)=(0,255,255)
rgb255(2.5cm)=(0,255,0)
rgb255(3.5cm)=(255,255,0)
rgb255(6cm)=(255,0,0)
}
}
%% Start the prinable document
\begin{document}
You can write a full paper here and include many figures!
Describe what the two axes are, and how you measured them.
Also, do not forget to explain what it shows and how to interpret it.
You also have separate PDFs for every figure in the `tikz' directory.
Feel free to change this text.
%% Draw the plot.
\begin{tikzpicture}
\small
\begin{axis}[
width=\linewidth,
view={0}{90},
colorbar horizontal,
xlabel=X axis,
ylabel=Y axis,
ylabel shift=-0.1cm,
colorbar style={at={(0,1.01)}, anchor=south west,
xticklabel pos=upper},
]
\addplot3[
surf,
shader=flat corner,
mesh/ordering=rowwise,
mesh/cols=XXXXXXXXX, %% <-- Number of bins in 1st column.
] file {FILE.txt}; %% <-- Name of aststatistics output.
\end{axis}
\end{tikzpicture}
%% End the printable document.
\end{document}
Let's assume you have put the LaTeX source above, into a plain-text
file called ‘report.tex’. The PGFPlots call above is configured to
build the plots as separate PDF files in a ‘tikz/’ directory(1). This
allows you to directly load those PDFs in your slides or other reports.
Therefore, before building the PDF report, you should first make a
‘tikz/’ directory:
$ mkdir tikz
To build the final PDF, you should run ‘pdflatex’ with the
‘--shell-escape’ option, so it can build the separate PDF(s) separately.
We are also adding the ‘--halt-on-error’ so it immediately aborts in the
case of an error (in the case of an error, by default LaTeX will not
abort, it will stop and ask for your input to temporarily change things
and try fixing the error, but it has a special interface which can be
hard to master).
$ pdflatex --shell-escape --halt-on-error report.tex
You can now open ‘report.pdf’ to see your very high quality 2D histogram
within your text. And if you need the plots separately (for example,
for slides), you can take the PDF inside the ‘tikz/’ directory.
---------- Footnotes ----------
(1) TiKZ (https://www.ctan.org/pkg/pgf) is the name of the
lower-level engine behind PGPlots.
7.1.2.2 2D histogram as an image
................................
When called with the ‘--histogram=image’ option, Statistics will output
a FITS file with an image/array extension. If you asked for
‘--numbins=N’ and ‘--numbins2=M’ the image will have a size of $N\times
M$ pixels (one pixel per 2D bin). Also, the FITS image will have a
linear WCS that is scaled to the 2D bin size along each dimension. So
when you hover your mouse over any part of the image with a FITS viewer
(for example, SAO DS9), besides the number of points in each pixel, you
can directly also see "coordinates" of the pixels along the two axes.
You can also use the optimized and fast FITS viewer features for many
aspects of visually inspecting the distributions (which we will not go
into further).
For example, let's assume you want to derive the color-magnitude
diagram (CMD) of the UVUDF survey (http://uvudf.ipac.caltech.edu). You
can run the first command below to download the table with magnitudes of
objects in many filters and run the second command to see general column
metadata after it is downloaded.
$ wget http://asd.gsfc.nasa.gov/UVUDF/uvudf_rafelski_2015.fits.gz
$ asttable uvudf_rafelski_2015.fits.gz -i
Let's assume you want to find the color to be between the ‘F606W’ and
‘F775W’ filters (roughly corresponding to the g and r filters in
ground-based imaging). However, the original table does not have color
columns (there would be too many combinations!). Therefore you can use
the *note Column arithmetic:: feature of Gnuastro's Table program for
deriving a new table with the ‘F775W’ magnitude in one column and the
difference between the ‘F606W’ and ‘F775W’ on the other column. With
the second command, you can see the actual values if you like.
$ asttable uvudf_rafelski_2015.fits.gz -cMAG_F775W \
-c'arith MAG_F606W MAG_F775W -' \
--colmetadata=ARITH_1,F606W-F775W,"AB mag" -ocmd.fits
$ asttable cmd.fits
You can now construct your 2D histogram as a $100\times100$ pixel FITS
image with this command (assuming you want ‘F775W’ magnitudes between 22
and 30, colors between -1 and 3 and 100 bins in each dimension). Note
that without the ‘--manualbinrange’ option the range of each axis will
be determined by the values within the columns (which may be larger or
smaller than your desired large).
aststatistics cmd.fits -cMAG_F775W,F606W-F775W --histogram2d=image \
--numbins=100 --greaterequal=22 --lessthan=30 \
--numbins2=100 --greaterequal2=-1 --lessthan2=3 \
--manualbinrange --output=cmd-2d-hist.fits
If you have SAO DS9, you can now open this FITS file as a normal FITS
image, for example, with the command below. Try hovering/zooming over
the pixels: not only will you see the number of objects in the UVUDF
catalog that fall in each bin, but you also see the ‘F775W’ magnitude
and color of that pixel also.
$ ds9 cmd-2d-hist.fits -cmap sls -zoom to fit
With the first command below, you can activate the grid feature of DS9
to actually see the coordinate grid, as well as values on each line.
With the second command, DS9 will even read the labels of the axes and
use them to generate an almost publication-ready plot.
$ ds9 cmd-2d-hist.fits -cmap sls -zoom to fit -grid yes
$ ds9 cmd-2d-hist.fits -cmap sls -zoom to fit -grid yes \
-grid type publication
If you are happy with the grid, coloring and the rest, you can also
use ds9 to save this as a JPEG image to directly use in your
documents/slides with these extra DS9 options (DS9 will write the image
to ‘cmd-2d.jpeg’ and quit immediately afterwards):
$ ds9 cmd-2d-hist.fits -cmap sls -zoom 4 -grid yes \
-grid type publication -saveimage cmd-2d.jpeg -quit
This is good for a fast progress update. But for your paper or more
official report, you want to show something with higher quality. For
that, you can use the PGFPlots package in LaTeX to add axes in the same
font as your text, sharp grids and many other elegant/powerful features
(like over-plotting interesting points and lines). But to load the 2D
histogram into PGFPlots first you need to convert the FITS image into a
more standard format, for example, PDF. We will use Gnuastro's *note
ConvertType:: for this, and use the ‘sls-inverse’ color map (which will
map the pixels with a value of zero to white):
$ astconvertt cmd-2d-hist.fits --colormap=sls-inverse \
--borderwidth=0 -ocmd-2d-hist.pdf
Below you can see a minimally working example of how to add axis
numbers, labels and a grid to the PDF generated above. Copy and paste
the LaTeX code below into a plain-text file called ‘cmd-report.tex’
Notice the ‘xmin’, ‘xmax’, ‘ymin’, ‘ymax’ values and how they are the
same as the range specified above.
\documentclass{article}
\usepackage{pgfplots}
\dimendef\prevdepth=0
\begin{document}
You can write all you want here...
\begin{tikzpicture}
\begin{axis}[
enlargelimits=false,
grid,
axis on top,
width=\linewidth,
height=\linewidth,
xlabel={Magnitude (F775W)},
ylabel={Color (F606W-F775W)}]
\addplot graphics[xmin=22, xmax=30, ymin=-1, ymax=3]
{cmd-2d-hist.pdf};
\end{axis}
\end{tikzpicture}
\end{document}
Run this command to build your PDF (assuming you have LaTeX and
PGFPlots).
$ pdflatex cmd-report.tex
The improved quality, blending in with the text, vector-graphics
resolution and other features make this plot pleasing to the eye, and
let your readers focus on the main point of your scientific argument.
PGFPlots can also built the PDF of the plot separately from the rest of
the paper/report, see *note 2D histogram as a table for plotting:: for
the necessary changes in the preamble.
7.1.3 Least squares fitting
---------------------------
After completing a good observation, doing robust data reduction and
finalizing the measurements, it is commonly necessary to parameterize
the derived correlations. For example, you have derived the radial
profile of the PSF of your image (see *note Building the extended
PSF::). You now want to parameterize the radial profile to estimate the
slope. Alternatively, you may have found the star formation rate and
stellar mass of your sample of galaxies. Now, you want to derive the
star formation main sequence as a parametric relation between the two.
The fitting functions below can be used for such purposes.
Gnuastro's least squares fitting features are just wrappers over the
least squares fitting methods of the linear
(https://www.gnu.org/software/gsl/doc/html/lls.html) and nonlinear
(https://www.gnu.org/software/gsl/doc/html/nls.html) least-squares
fitting functions of the GNU Scientific Library (GSL). For the low-level
details and equations of the methods, please see the GSL documentation.
The names have been preserved here in Gnuastro to simplify the
connection with GSL and follow the details in the detailed documentation
there.
GSL is a very low-level library, designed for maximal portability to
many scenarios, and power. Therefore calling GSL's functions directly
for a fast operation requires a good knowledge of the C programming
language and many lines of code. As a low-level library, GSL is
designed to be the back-end of higher-level programs (like Gnuastro).
Through the Statistics program, in Gnuastro we provide a high-level
interface to access to GSL's very powerful least squares fitting engine
to read/write from/to standard data formats in astronomy. A fully
working example is shown below.
To activate fitting in Statistics, simply give your desired fitting
method to the ‘--fit’ option (for the full list of acceptable methods,
see *note Fitting options::). For example, with the command below,
we'll build a fake measurement table (including noise) from the
polynomial $y=1.23-4.56x+7.89x^2$. To understand how this equation
translates to the command below (part before ‘set-y’), see *note Reverse
polish notation:: and *note Column arithmetic::. We will set the X axis
to have values from 0.1 to 2, with steps of 0.01 and let's assume a
random Gaussian noise to each $y$ measurement: $\sigma_y=0.1y$. To make
the random number generation exactly reproducible, we are also setting
the seed (see *note Generating random numbers::, which also uses GSL as
a backend). To learn more about the ‘mknoise-sigma’ operator, see the
Arithmetic program's *note Random number generators::.
$ export GSL_RNG_SEED=1664015492
$ seq 0.1 0.01 2 \
| asttable --output=noisy.fits --envseed -c1 \
-c'arith 1.23 -4.56 $1 x + 7.89 $1 x $1 x + set-y \
0.1 y x set-yerr \
y yerr mknoise-sigma yerr' \
--colmetadata=1,X --colmetadata=2,Y \
--colmetadata=3,Yerr
Let's have a look at the output plot with TOPCAT using the command
below.
$ astscript-fits-view noisy.fits
To see the error-bars, after opening the scatter plot, go into the
"Form" tab for that plot. Click on the button with a green "+" sign
followed by "Forms" and select "XYError". On the side-menu, in front of
"Y Positive Error", select the ‘Yerr’ column of the input table.
As you see, the error bars do indeed increase for higher X axis
values. Since we have error bars in this example (as in any
measurement), we can use weighted fitting. Also, this isn't a linear
relation, so we'll use a polynomial to second order (a maximum power of
2 in the form of $Y=c_0+c_1X+c_2X^2$):
$ aststatistics noisy.fits -cX,Y,Yerr --fit=polynomial-weighted \
--fitmaxpower=2
Statistics (GNU Astronomy Utilities) 0.23
-------
Fitting results (remove extra info with '--quiet' or '-q)
Input file: noisy.fits (hdu: 1) with 191 non-blank rows.
X column: X
Y column: Y
Weight column: Yerr [Standard deviation of Y in each row]
Fit function: Y = c0 + (c1 * X^1) + (c2 * X^2) + ... (cN * X^N)
N: 2
c0: +1.2286211608
c1: -4.5127796636
c2: +7.8435883943
Covariance matrix:
+0.0010496001 -0.0039928488 +0.0028367390
-0.0039928488 +0.0175244127 -0.0138030778
+0.0028367390 -0.0138030778 +0.0128129806
Reduced chi^2 of fit:
+0.9740670090
As you see from the elaborate message, the weighted polynomial
fitting has found return the $c_0$, $c_1$ and $c_2$ of
$Y=c_0+c_1X+c_2X^2$ that best represents the data we inserted. Our
input values were $c_0=1.23$, $c_1=-4.56$ and $c_2=7.89$, and the fitted
values are $c_0\approx1.2286$, $c_1\approx-4.5128$ and
$c_2\approx7.8436$ (which is statistically a very good fit! given that
we knew the original values a-priori!). The covariance matrix is also
calculated, it is necessary to calculate error bars on the estimations
and contains a lot of information (e.g., possible correlations between
parameters). Finally, the reduced $\chi^2$ (or $\chi_{red}^2$) of the
fit is also printed (which was the measure to minimize). A
$\chi_{red}^2\approx1$ shows a good fit. This is good for real-world
scenarios when you don't know the original values a-priori. For more on
interpreting $\chi_{red}^2\approx1$, see Andrae et al. 2010
(https://arxiv.org/abs/1012.3754).
The comparison of fitted and input values look pretty good, but
nothing beats visual inspection! To see how this looks compared to the
data, let's open the table again:
$ astscript-fits-view noisy.fits
Repeat the steps below to show the scatter plot and error-bars.
Then, go to the "Layers" menu and select "Add Function Control". Use
the results above to fill the box in front of "Function Expression":
‘1.2286+(-4.5128*x)+(7.8436*x*x)’. You will see that the second order
polynomial falls very nicely over the points(1). But this fit is not
perfect: it also has errors (inherited from the measurement errors). We
need the covariance matrix to estimate the errors on each point, and
that can be complex to do by hand.
Fortunately GSL has the tools to easily estimate the function at any
point and also calculate its corresponding error. To access this
feature within Gnuastro's Statistics program, you should use the
‘--fitestimate’ option. You can either give an independent table file
name (with ‘--fitestimatehdu’ and ‘--fitestimatecol’ to specify the HDU
and column in that file), or just ‘self’ so it uses the same X axis
column that was used in this fit. Let's use the easier case:
$ aststatistics noisy.fits -cX,Y,Yerr --fit=polynomial-weighted \
--fitmaxpower=2 --fitestimate=self --output=est.fits
...[[truncated; same as above]]...
Requested estimation:
Written to: est.fits
The first lines of the printed text are the same as before.
Afterwards, you will see a new line printed in the output, saying that
the estimation was written in ‘est.fits’. You can now inspect the two
tables with TOPCAT again with the command below. After TOPCAT opens,
plot both scatter plots:
$ astscript-fits-view noisy.fits est.fits
It is clear that they fall nicely on top of each other. The
‘est.fits’ table also has a third column with error bars. You can
follow the same steps before and draw the error bars to see how they
compare with the scatter of the measured data. They are much smaller
than the error in each point because we had a very good sampling of the
function in our noisy data.
Another useful point with the estimated output file is that it
contains all the fitting outputs as keywords in the header:
$ astfits est.fits -h1
...[[truncated]]...
/ Fit results
FITTYPE = 'polynomial-weighted' / Functional form of the fitting.
FITMAXP = 2 / Maximum power of polynomial.
FITIN = 'noisy.fits' / Name of file with input columns.
FITINHDU= '1 ' / Name or Number of HDU with input cols.
FITXCOL = 'X ' / Name or Number of independent (X) col.
FITYCOL = 'Y ' / Name or Number of measured (Y) column.
FITWCOL = 'Yerr ' / Name or Number of weight column.
FITWNAT = 'Standard deviation' / Nature of weight column.
FRDCHISQ= 0.974067008958516 / Reduced chi^2 of fit.
FITC0 = 1.22862116084727 / C0: multiple of x^0 in polynomial
FITC1 = -4.51277966356177 / C1: multiple of x^1 in polynomial
FITC2 = 7.84358839431161 / C2: multiple of x^2 in polynomial
FCOV11 = 0.00104960011629718 / Covariance matrix element (1,1).
FCOV12 = -0.00399284880859776 / Covariance matrix element (1,2).
FCOV13 = 0.00283673901863388 / Covariance matrix element (1,3).
FCOV21 = -0.00399284880859776 / Covariance matrix element (2,1).
FCOV22 = 0.0175244126670659 / Covariance matrix element (2,2).
FCOV23 = -0.0138030778380786 / Covariance matrix element (2,3).
FCOV31 = 0.00283673901863388 / Covariance matrix element (3,1).
FCOV32 = -0.0138030778380786 / Covariance matrix element (3,2).
FCOV33 = 0.0128129806394559 / Covariance matrix element (3,3).
...[[truncated]]...
In scenarios were you don't want the estimation, but only the fitted
parameters, all that verbose, human-friendly text or FITS keywords can
be an annoying extra step. For such cases, you should use the ‘--quiet’
option like below. It will print the parameters, rows of the covariance
matrix and $\chi_{red}^2$ on separate lines with nothing extra. This
allows you to parse the values in any way that you would like.
$ aststatistics noisy.fits -cX,Y,Yerr --fit=polynomial-weighted \
--fitmaxpower=2 --quiet
+1.2286211608 -4.5127796636 +7.8435883943
+0.0010496001 -0.0039928488 +0.0028367390
-0.0039928488 +0.0175244127 -0.0138030778
+0.0028367390 -0.0138030778 +0.0128129806
+0.9740670090
As a final example, because real data usually have outliers, let's
look at the "robust" polynomial fit which has special features to remove
outliers. First, we need to add some outliers to the table. To do
this, we'll make a plain-text table with ‘echo’, and use Table's
‘--catrowfile’ to concatenate (or append) those two rows to the original
table. Finally, we'll run the same fitting step above:
$ echo "0.6 20 0.01" > outliers.txt
$ echo "0.8 20 0.01" >> outliers.txt
$ asttable noisy.fits --catrowfile=outliers.txt \
--output=with-outlier.fits
$ aststatistics with-outlier.fits -cX,Y,Yerr --fit=polynomial-weighted \
--fitmaxpower=2 --fitestimate=self \
--output=est-out.fits
Statistics (GNU Astronomy Utilities) 0.23
-------
Fitting results (remove extra info with '--quiet' or '-q)
Input file: with-outlier.fits (hdu: 1) with 193 non-blank rows.
X column: X
Y column: Y
Weight column: Yerr [Standard deviation of Y in each row]
Fit function: Y = c0 + (c1 * X^1) + (c2 * X^2) + ... (cN * X^N)
N: 2
c0: -13.6446036899
c1: +66.8463258547
c2: -30.8746303591
Covariance matrix:
+0.0007889160 -0.0027706310 +0.0022208939
-0.0027706310 +0.0113922468 -0.0100306732
+0.0022208939 -0.0100306732 +0.0094087226
Reduced chi^2 of fit:
+4501.8356719150
Requested estimation:
Written to: est-out.fit
We see that the coefficient values have changed significantly and
that $\chi_{red}^2$ has increased to $4501$! Recall that a good fit
should have $\chi_{red}^2\approx1$. These numbers clearly show that the
fit was bad, but again, nothing beats a visual inspection. To visually
see the effect of those outliers, let's plot them with the command
below. You see that those two points have clearly caused a turn in the
fitted result which is terrible.
$ astscript-fits-view with-outlier.fits est-out.fits
For such cases, GSL has Robust linear regression
(https://www.gnu.org/software/gsl/doc/html/lls.html#robust-linear-regression).
In Gnuastro's Statistics, you can access it with
‘--fit=polynomial-robust’, like the example below. Just note that the
robust method doesn't take an error column (because it estimates the
errors internally while rejecting outliers, based on the method).
$ aststatistics with-outlier.fits -cX,Y --fit=polynomial-robust \
--fitmaxpower=2 --fitestimate=self \
--output=est-out.fits --quiet
$ astfits est-out.fits -h1 | grep ^FITC
FITC0 = 1.20422691185238 / C0: multiple of x^0 in polynomial
FITC1 = -4.4779253576348 / C1: multiple of x^1 in polynomial
FITC2 = 7.84986153686548 / C2: multiple of x^2 in polynomial
$ astscript-fits-view with-outlier.fits est-out.fits
It is clear that the coefficients are very similar to the no-outlier
scenario above and if you run the second command to view the scatter
plots on TOPCAT, you also see that the fit nicely follows the curve and
is not affected by those two points. GSL provides many methods to
reject outliers. For their full list, see the description of
‘--fitrobust’ in *note Fitting options::. For a description of the
outlier rejection methods, see the GSL manual
(https://www.gnu.org/software/gsl/doc/html/lls.html#c.gsl_multifit_robust_workspace).
You may have noticed that unlike the cases before the last Statistics
command above didn't print anything on the standard output. This is
becasue ‘--quiet’ and ‘--fitestimate’ were called together. In this
case, because all the fitting parameters are written as FITS keywords,
because of the ‘--quiet’ option, they are no longer printed on standard
output.
---------- Footnotes ----------
(1) After plotting, you will notice that the legend made the plot too
thin. Fortunately you have a lot of empty space within the plot. To
bring the legend in, click on the "Legend" item on the bottom-left menu,
in the "Location" tab, click on "Internal" and hold and move it to the
top-left in the box below. To make the functional fit more clear, you
can click on the "Function" item of the bottom-left menu. In the
"Style" tab, change the color and thickness.
7.1.4 Sky value
---------------
One of the most important aspects of a dataset is its reference value:
the value of the dataset where there is no signal. Without knowing, and
thus removing the effect of, this value it is impossible to compare the
derived results of many high-level analyses over the dataset with other
datasets (in the attempt to associate our results with the "real"
world).
In astronomy, this reference value is known as the "Sky" value: the
value that noise fluctuates around: where there is no signal from
detectable objects or artifacts (for example, galaxies, stars, planets
or comets, star spikes or internal optical ghost). Depending on the
dataset, the Sky value maybe a fixed value over the whole dataset, or it
may vary based on location. For an example of the latter case, see
Figure 11 in Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664).
Because of the significance of the Sky value in astronomical data
analysis, we have devoted this subsection to it for a thorough review.
We start with a thorough discussion on its definition (*note Sky value
definition::). In the astronomical literature, researchers use a
variety of methods to estimate the Sky value, so in *note Sky value
misconceptions::) we review those and discuss their biases. From the
definition of the Sky value, the most accurate way to estimate the Sky
value is to run a detection algorithm (for example, *note NoiseChisel::)
over the dataset and use the undetected pixels. However, there is also
a more crude method that maybe useful when good direct detection is not
initially possible (for example, due to too many cosmic rays in a
shallow image). A more crude (but simpler method) that is usable in
such situations is discussed in *note Quantifying signal in a tile::.
7.1.4.1 Sky value definition
............................
This analysis is taken from Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664). Let's assume that all instrument
defects - bias, dark and flat - have been corrected and the magnitude
(see *note Brightness flux magnitude::) of a detected object, $O$, is
desired. The sources of flux on pixel(1) $i$ of the image can be
written as follows:
• Contribution from the target object ($O_i$).
• Contribution from other detected objects ($D_i$).
• Undetected objects or the fainter undetected regions of bright
objects ($U_i$).
• A cosmic ray ($C_i$).
• The background flux, which is defined to be the count if none of
the others exists on that pixel ($B_i$).
The total flux in this pixel ($T_i$) can thus be written as:
$$T_i=B_i+D_i+U_i+C_i+O_i.$$
By definition, $D_i$ is detected and it can be assumed that it is
correctly estimated (deblended) and subtracted, we can thus set $D_i=0$.
There are also methods to detect and remove cosmic rays, for example,
the method described in van Dokkum (2001)(2), or by comparing multiple
exposures. This allows us to set $C_i=0$. Note that in practice, $D_i$
and $U_i$ are correlated, because they both directly depend on the
detection algorithm and its input parameters. Also note that no
detection or cosmic ray removal algorithm is perfect. With these
limitations in mind, the observed Sky value for this pixel ($S_i$) can
be defined as
$$S_i\equiv{}B_i+U_i.$$
Therefore, as the detection process (algorithm and input parameters)
becomes more accurate, or $U_i\to0$, the Sky value will tend to the
background value or $S_i\to B_i$. Hence, we see that while $B_i$ is an
inherent property of the data (pixel in an image), $S_i$ depends on the
detection process. Over a group of pixels, for example, in an image or
part of an image, this equation translates to the average of undetected
pixels (Sky$=\sum{S_i}$). With this definition of Sky, the object flux
in the data can be calculated, per pixel, with
$$T_{i}=S_{i}+O_{i} \quad\rightarrow\quad
O_{i}=T_{i}-S_{i}.$$
In the fainter outskirts of an object, a very small fraction of the
photo-electrons in a pixel actually belongs to objects, the rest is
caused by random factors (noise), see Figure 1b in Akhlaghi and Ichikawa
2015 (https://arxiv.org/abs/1505.01664). Therefore even a small over
estimation of the Sky value will result in the loss of a very large
portion of most galaxies. Besides the lost area/brightness, this will
also cause an over-estimation of the Sky value and thus even more
under-estimation of the object's magnitude. It is thus very important
to detect the diffuse flux of a target, even if they are not your
primary target.
In summary, the more accurately the Sky is measured, the more
accurately the magnitude (calculated from the sum of pixel values) of
the target object can be measured (photometry). Any
under/over-estimation in the Sky will directly translate to an
over/under-estimation of the measured object's magnitude.
The *Sky value* is only correctly found when all the detected objects
($D_i$ and $C_i$) have been removed from the data.
---------- Footnotes ----------
(1) For this analysis the dimension of the data (image) is
irrelevant. So if the data is an image (2D) with width of $w$ pixels,
then a pixel located on column $x$ and row $y$ (where all counting
starts from zero and (0, 0) is located on the bottom left corner of the
image), would have an index: $i=x+y\times{}w$.
(2) van Dokkum, P. G. (2001). Publications of the Astronomical
Society of the Pacific. 113, 1420.
7.1.4.2 Sky value misconceptions
................................
As defined in *note Sky value::, the sky value is only accurately
defined when the detection algorithm is not significantly reliant on the
sky value. In particular its detection threshold. However, most
signal-based detection tools(1) use the sky value as a reference to
define the detection threshold. These older techniques therefore had to
rely on approximations based on other assumptions about the data. A
review of those other techniques can be seen in Appendix A of Akhlaghi
and Ichikawa 2015 (https://arxiv.org/abs/1505.01664).
These methods were extensively used in astronomical data analysis for
several decades, therefore they have given rise to a lot of
misconceptions, ambiguities and disagreements about the sky value and
how to measure it. As a summary, the major methods used until now were
an approximation of the mode of the image pixel distribution and
$\sigma$-clipping.
• To find the mode of a distribution those methods would either have
to assume (or find) a certain probability density function (PDF) or
use the histogram. But astronomical datasets can have any
distribution, making it almost impossible to define a generic
function. Also, histogram-based results are very inaccurate (there
is a large dispersion) and it depends on the histogram bin-widths.
Generally, the mode of a distribution also shifts as signal is
added. Therefore, even if it is accurately measured, the mode is a
biased measure for the Sky value.
• Another approach was to iteratively clip the brightest pixels in
the image (which is known as $\sigma$-clipping). See *note Sigma
clipping:: for a complete explanation. $\sigma$-clipping is useful
when there are clear outliers (an object with a sharp edge in an
image for example). However, real astronomical objects have
diffuse and faint wings that penetrate deeply into the noise, see
Figure 1 in Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664).
As discussed in *note Sky value::, the sky value can only be
correctly defined as the average of undetected pixels. Therefore all
such approaches that try to approximate the sky value prior to detection
are ultimately poor approximations.
---------- Footnotes ----------
(1) According to Akhlaghi and Ichikawa (2015), signal-based detection
is a detection process that relies heavily on assumptions about the
to-be-detected objects. This method was the most heavily used technique
prior to the introduction of NoiseChisel in that paper.
7.1.4.3 Quantifying signal in a tile
....................................
In order to define detection thresholds on the image, or calibrate it
for measurements (subtract the signal of the background sky and define
errors), we need some basic measurements. For example, the quantile
threshold in NoiseChisel (‘--qthresh’ option), or the mean of the
undetected regions (Sky) and the Sky standard deviation (Sky STD) which
are the output of NoiseChisel and Statistics. But astronomical images
will contain a lot of stars and galaxies that will bias those
measurements if not properly accounted for. Quantifying where signal is
present is thus a very important step in the usage of a dataset; for
example, if the Sky level is over-estimated, your target object's
magnitude will be under-estimated.
Let's start by clarifying some definitions: _Signal_ is defined as
the non-random source of flux in each pixel (you can think of this as
the mean in a Gaussian or Poisson distribution). In astronomical
images, signal is mostly photons coming of a star or galaxy, and counted
in each pixel. _Noise_ is defined as the random source of flux in each
pixel (or the standard deviation of a Gaussian or Poisson distribution).
Noise is mainly due to counting errors in the detector electronics upon
data collection. _Data_ is defined as the combination of signal and
noise (so a noisy image of a galaxy is one _data_set).
When a dataset does not have any signal (for example, you take an
image with a closed shutter, producing an image that only contains
noise), the mean, median and mode of the distribution are equal within
statistical errors. Signal from emitting objects, like astronomical
targets, always has a positive value and will never become negative, see
Figure 1 in Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664). Therefore, when signal is added to
the data (you take an image with an open shutter pointing to a galaxy
for example), the mean, median and mode of the dataset shift to the
positive, creating a positively skewed distribution. The shift of the
mean is the largest. The median shifts less, since it is defined after
ordering all the elements/pixels (the median is the value at a quantile
of 0.5), thus it is not affected by outliers. Finally, the mode's shift
to the positive is the least.
Inverting the argument above gives us a robust method to quantify the
significance of signal in a dataset: when the mean and median of a
distribution are approximately equal we can argue that there is no
significant signal. In other words: when the quantile of the mean
($q_{mean}$) is around 0.5. This definition of skewness through the
quantile of the mean is further introduced with a real image the
tutorials, see *note Skewness caused by signal and its measurement::.
However, in an astronomical image, some of the pixels will contain
more signal than the rest, so we cannot simply check $q_{mean}$ on the
whole dataset. For example, if we only look at the patch of pixels that
are placed under the central parts of the brightest stars in the field
of view, $q_{mean}$ will be very high. The signal in other parts of the
image will be weaker, and in some parts it will be much smaller than the
noise (for example, 1/100-th of the noise level). When the
signal-to-noise ratio is very small, we can generally assume no signal
(because its effectively impossible to measure it) and $q_{mean}$ will
be approximately 0.5.
To address this problem, we break the image into a grid of tiles(1)
(see *note Tessellation::). For example, a tile can be a square box of
size $30\times30$ pixels. By measuring $q_{mean}$ on each tile, we can
find which tiles that contain significant signal and ignore them.
Technically, if a tile's $|q_{mean}-0.5|$ is larger than the value given
to the ‘--meanmedqdiff’ option, that tile will be ignored for the next
steps. You can read this option as "mean-median-quantile-difference".
The raw dataset's pixel distribution (in each tile) is noisy, to
decrease the noise/error in estimating $q_{mean}$, we convolve the image
before tessellation (see *note Convolution process::. Convolution
decreases the range of the dataset and enhances its skewness, See
Section 3.1.1 and Figure 4 in Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664). This enhanced skewness can be
interpreted as an increase in the Signal to noise ratio of the objects
buried in the noise. Therefore, to obtain an even better measure of the
presence of signal in a tile, the mean and median discussed above are
measured on the convolved image.
There is one final hurdle: raw astronomical datasets are commonly
peppered with Cosmic rays. Images of Cosmic rays are not smoothed by
the atmosphere or telescope aperture, so they have sharp boundaries.
Also, since they do not occupy too many pixels, they do not affect the
mode and median calculation. But their very high values can greatly
bias the calculation of the mean (recall how the mean shifts the fastest
in the presence of outliers), for example, see Figure 15 in Akhlaghi and
Ichikawa 2015 (https://arxiv.org/abs/1505.01664). The effect of
outliers like cosmic rays on the mean and standard deviation can be
removed through $\sigma$-clipping, see *note Sigma clipping:: for a
complete explanation.
Therefore, after asserting that the mean and median are approximately
equal in a tile (see *note Tessellation::), the Sky and its STD are
measured on each tile after $\sigma$-clipping with the ‘--sigmaclip’
option (see *note Sigma clipping::). In the end, some of the tiles will
pass the test and will be given a value. Others (that had signal in
them) will just be assigned a NaN (not-a-number) value. But we need a
measurement over each tile (and thus pixel). We will therefore use
interpolation to assign a value to the NaN tiles.
However, prior to interpolating over the failed tiles, another point
should be considered: large and extended galaxies, or bright stars, have
wings which sink into the noise very gradually. In some cases, the
gradient over these wings can be on scales that is larger than the tiles
(for example, the pixel value changes by $0.1\sigma$ after 100 pixels,
but the tile has a width of 30 pixels).
In such cases, the $q_{mean}$ test will be successful, even though
there is signal. Recall that $q_{mean}$ is a measure of skewness. If
we do not identify (and thus set to NaN) such outlier tiles before the
interpolation, the photons of the outskirts of the objects will leak
into the detection thresholds or Sky and Sky STD measurements and bias
our result, see *note Detecting large extended targets::. Therefore,
the final step of "quantifying signal in a tile" is to look at this
distribution of successful tiles and remove the outliers.
$\sigma$-clipping is a good solution for removing a few outliers, but
the problem with outliers of this kind is that there may be many such
tiles (depending on the large/bright stars/galaxies in the image). We
therefore apply the following local outlier rejection strategy.
For each tile, we find the nearest $N_{ngb}$ tiles that had a usable
value ($N_{ngb}$ is the value given to ‘--outliernumngb’). We then sort
them and find the difference between the largest and second-to-smallest
elements (The minimum is not used because the scatter can be large).
Let's call this the tile's _slope_ (measured from its neighbors). All
the tiles that are on a region of flat noise will have similar slope
values, but if a few tiles fall on the wings of a bright star or large
galaxy, their slope will be significantly larger than the tiles with no
signal. We just have to find the smallest tile slope value that is an
outlier compared to the rest, and reject all tiles with a slope larger
than that.
To identify the smallest outlier, we will use the distribution of
distances between sorted elements. Let's assume the total number of
tiles with a good mean-median quantile difference is $N$. They are
first sorted and searching for the outlier starts on element $N/3$
(integer division). Let's take $v_i$ to be the $i$-th element of the
sorted input (with no blank values) and $m$ and $\sigma$ as the
$\sigma$-clipped median and standard deviation from the distances of the
previous $N/3-1$ elements (not including $v_i$). If the value given to
‘--outliersigma’ is displayed with $s$, the $i$-th element is considered
as an outlier when the condition below is true.
$${(v_i-v_{i-1})-m\over \sigma}>s$$
Since $i$ begins from the $N/3$-th element in the sorted array (a
quantile of $1/3=0.33$), the outlier has to be larger than the $0.33$
quantile value of the dataset (this is usually the case; otherwise, it
is hard to define it as an "outlier"!).
Once the outlying tiles have been successfully identified and set to
NaN, we use nearest-neighbor interpolation to give a value to all tiles
in the image. We do not use parametric interpolation methods (like
bicubic), because they will effectively extrapolate on the edges,
creating strong artifacts. Nearest-neighbor interpolation is very
simple: for each tile, we find the $N_{ngb}$ nearest tiles that had a
good value, the tile's value is found by estimating the median. You can
set $N_{ngb}$ through the ‘--interpnumngb’ option. Once all the tiles
are given a value, a smoothing step is implemented to remove the sharp
value contrast that can happen on the edges of tiles. The size of the
smoothing box is set with the ‘--smoothwidth’ option.
As mentioned above, the process above is used for any of the basic
measurements (for example, identifying the quantile-based thresholds in
NoiseChisel, or the Sky value in Statistics). You can use the
check-image feature of NoiseChisel or Statistics to inspect the steps
and visually see each step (all the options that start with ‘--check’).
For example, as mentioned in the *note NoiseChisel optimization::
tutorial, when given a dataset from a new instrument (with differing
noise properties), we highly recommend to use ‘--checkqthresh’ in your
first call and visually inspect how the parameters above affect the
final quantile threshold (e.g., have the wings of bright sources leaked
into the threshold?). The same goes for the ‘--checksky’ option of
Statistics or NoiseChisel.
---------- Footnotes ----------
(1) The options to customize the tessellation are discussed in *note
Processing options::.
7.1.5 Invoking Statistics
-------------------------
Statistics will print statistical measures of an input dataset (table
column or image). The executable name is ‘aststatistics’ with the
following general template
$ aststatistics [OPTION ...] InputImage.fits
One line examples:
## Print some general statistics of input image:
$ aststatistics image.fits
## Print some general statistics of column named MAG_F160W:
$ aststatistics catalog.fits -h1 --column=MAG_F160W
## Make the histogram of the column named MAG_F160W:
$ aststatistics table.fits -cMAG_F160W --histogram
## Find the Sky value on image with a given kernel:
$ aststatistics image.fits --sky --kernel=kernel.fits
## Print Sigma-clipped results of records with a MAG_F160W
## column value between 26 and 27:
$ aststatistics cat.fits -cMAG_F160W -g26 -l27 --sigmaclip=3,0.2
## Find the polynomial (to third order) that best fits the X and Y
## columns of 'table.fits'. Robust fitting will be used to reject
## outliers. Also, estimate the fitted polynomial on the same input
## column (with errors).
$ aststatistics table.fits --fit=polynomial-robust --fitmaxpower=3 \
-cX,Y --fitestimate=self --output=estimated.fits
## Print the median value of all records in column MAG_F160W that
## have a value larger than 3 in column PHOTO_Z:
$ aststatistics tab.txt -rPHOTO_Z -g3 -cMAG_F160W --median
## Calculate the median of the third column in the input table, but only
## for rows where the mean of the first and second columns is >5.
$ awk '($1+$2)/2 > 5 {print $3}' table.txt | aststatistics --median
Statistics can take its input dataset either from a file (image or
table) or the Standard input (see *note Standard input::). If any
output file is to be created, the value to the ‘--output’ option, is
used as the base name for the generated files. Without ‘--output’, the
input name will be used to generate an output name, see *note Automatic
output::. The options described below are particular to Statistics, but
for general operations, it shares a large collection of options with the
other Gnuastro programs, see *note Common options:: for the full list.
For more on reading from standard input, please see the description of
‘--stdintimeout’ option in *note Input output options::. Options can
also be given in configuration files, for more, please see *note
Configuration files::.
The input dataset may have blank values (see *note Blank pixels::),
in this case, all blank pixels are ignored during the calculation.
Initially, the full dataset will be read, but it is possible to select a
specific range of data elements to use in the analysis of each run. You
can either directly specify a minimum and maximum value for the range of
data elements to use (with ‘--greaterequal’ or ‘--lessthan’), or specify
the range using quantiles (with ‘--qrange’). If a range is specified,
all pixels outside of it are ignored before any processing.
When no operation is requested, Statistics will print some general
basic properties of the input dataset on the command-line like the
example below (ran on one of the output images of ‘make check’(1)).
This default behavior is designed to help give you a general feeling of
how the data are distributed and help in narrowing down your analysis.
$ aststatistics convolve_spatial_scaled_noised.fits \
--greaterequal=9500 --lessthan=11000
Statistics (GNU Astronomy Utilities) X.X
-------
Input: convolve_spatial_scaled_noised.fits (hdu: 0)
Range: from (inclusive) 9500, upto (exclusive) 11000.
Unit: counts
-------
Number of elements: 9074
Minimum: 9622.35
Maximum: 10999.7
Mode: 10055.45996
Mode quantile: 0.4001983908
Median: 10093.7
Mean: 10143.98257
Standard deviation: 221.80834
-------
Histogram:
| **
| ******
| *******
| *********
| *************
| **************
| ******************
| ********************
| *************************** *
| ***************************************** ***
|* **************************************************************
|-----------------------------------------------------------------
Gnuastro's Statistics is a very general purpose program, so to be
able to easily understand this diversity in its operations (and how to
possibly run them together), we will divided the operations into two
types: those that do not respect the position of the elements and those
that do (by tessellating the input on a tile grid, see *note
Tessellation::). The former treat the whole dataset as one and can
re-arrange all the elements (for example, sort them), but the former do
their processing on each tile independently. First, we will review the
operations that work on the whole dataset.
The group of options below can be used to get single value
measurement(s) of the whole dataset. They will print only the requested
value as one field in a line/row, like the ‘--mean’, ‘--median’ options.
These options can be called any number of times and in any order. The
outputs of all such options will be printed on one line following each
other (with a space character between them). This feature makes these
options very useful in scripts, or to redirect into programs like GNU
AWK for higher-level processing. These are some of the most basic
measures, Gnuastro is still under heavy development and this list will
grow. If you want another statistical parameter, please contact us and
we will do out best to add it to this list, see *note Suggest new
feature::.
---------- Footnotes ----------
(1) You can try it by running the command in the ‘tests’ directory,
open the image with a FITS viewer and have a look at it to get a sense
of how these statistics relate to the input image/dataset.
7.1.5.1 Input to Statistics
...........................
The following set of options are for specifying the input/outputs of
Statistics. There are many other input/output options that are common
to all Gnuastro programs including Statistics, see *note Input output
options:: for those.
‘-c STR/INT’
‘--column=STR/INT’
The column to use when the input file is a table with more than one
column. See *note Selecting table columns:: for a full description
of how to use this option. For more on how tables are read in
Gnuastro, please see *note Tables::.
‘-g FLT’
‘--greaterequal=FLT’
Limit the range of inputs into those with values greater and equal
to what is given to this option. None of the values below this
value will be used in any of the processing steps below.
‘-l FLT’
‘--lessthan=FLT’
Limit the range of inputs into those with values less-than what is
given to this option. None of the values greater or equal to this
value will be used in any of the processing steps below.
‘-Q FLT[,FLT]’
‘--qrange=FLT[,FLT]’
Specify the range of usable inputs using the quantile. This option
can take one or two quantiles to specify the range. When only one
number is input (let's call it $Q$), the range will be those values
in the quantile range $Q$ to $1-Q$. So when only one value is
given, it must be less than 0.5. When two values are given, the
first is used as the lower quantile range and the second is used as
the larger quantile range.
The quantile of a given element in a dataset is defined by the
fraction of its index to the total number of values in the sorted
input array. So the smallest and largest values in the dataset
have a quantile of 0.0 and 1.0. The quantile is a very useful
non-parametric (making no assumptions about the input) relative
measure to specify a range. It can best be understood in terms of
the cumulative frequency plot, see *note Histogram and Cumulative
Frequency Plot::. The quantile of each horizontal axis value in
the cumulative frequency plot is the vertical axis value associate
with it.
7.1.5.2 Single value measurements
.................................
‘-n’
‘--number’
Print the number of all used (non-blank and in range) elements.
‘--minimum’
Print the minimum value of all used elements.
‘--maximum’
Print the maximum value of all used elements.
‘--sum’
Print the sum of all used elements.
‘-m’
‘--mean’
Print the mean (average) of all used elements.
‘-t’
‘--std’
Print the standard deviation of all used elements.
‘--mad’
Print the median absolute deviation (MAD) of all used elements.
‘-E’
‘--median’
Print the median of all used elements.
‘-u FLT[,FLT[,...]]’
‘--quantile=FLT[,FLT[,...]]’
Print the values at the given quantiles of the input dataset. Any
number of quantiles may be given and one number will be printed for
each. Values can either be written as a single number or as
fractions, but must be between zero and one (inclusive). Hence, in
effect ‘--quantile=0.25 --quantile=0.75’ is equivalent to
‘--quantile=0.25,3/4’, or ‘-u1/4,3/4’.
The returned value is one of the elements from the dataset. Taking
$q$ to be your desired quantile, and $N$ to be the total number of
used (non-blank and within the given range) elements, the returned
value is at the following position in the sorted array:
$round(q\times{}N$).
‘--quantfunc=FLT[,FLT[,...]]’
Print the quantiles of the given values in the dataset. This
option is the inverse of the ‘--quantile’ and operates similarly
except that the acceptable values are within the range of the
dataset, not between 0 and 1. Formally it is known as the
"Quantile function".
Since the dataset is not continuous this function will find the
nearest element of the dataset and use its position to estimate the
quantile function.
‘--quantofmean’
Print the quantile of the mean in the dataset. This is a very good
measure of detecting skewness or outliers. The concept is used by
programs like NoiseChisel to identify the presence of signal in a
tile of the image (because signal in noise causes skewness).
For example, take this simple array: ‘1 2 20 4 5 6 3’. The mean is
‘5.85’. The nearest element to this mean is ‘6’ and the quantile
of ‘6’ in this distribution is 0.8333. Here is how we got to this:
in the sorted dataset (‘1 2 3 4 5 6 20’), ‘6’ is the 5-th element
(counting from zero, since a quantile of zero corresponds to the
minimum, by definition) and the maximum is the 6-th element (again,
counting from zero). So the quantile of the mean in this case is
$5/6=0.8333$.
In the example above, if we had ‘7’ instead of ‘20’ (which was an
outlier), then the mean would be ‘4’ and the quantile of the mean
would be 0.5 (which by definition, is the quantile of the median),
showing no outliers. As the number of elements increases, the mean
itself is less affected by a small number of outliers, but skewness
can be nicely identified by the quantile of the mean.
‘-O’
‘--mode’
Print the mode of all used elements. The mode is found through the
mirror distribution which is fully described in Appendix C of
Akhlaghi and Ichikawa 2015 (https://arxiv.org/abs/1505.01664). See
that section for a full description.
This mode calculation algorithm is non-parametric, so when the
dataset is not large enough (larger than about 1000 elements
usually), or does not have a clear mode it can fail. In such
cases, this option will return a value of ‘nan’ (for the floating
point NaN value).
As described in that paper, the easiest way to assess the quality
of this mode calculation method is to use it's symmetricity (see
‘--modesym’ below). A better way would be to use the ‘--mirror’
option to generate the histogram and cumulative frequency tables
for any given mirror value (the mode in this case) as a table. If
you generate plots like those shown in Figure 21 of that paper,
then your mode is accurate.
‘--modequant’
Print the quantile of the mode. You can get the actual mode value
from the ‘--mode’ described above. In many cases, the absolute
value of the mode is irrelevant, but its position within the
distribution is important. In such cases, this option will become
handy.
‘--modesym’
Print the symmetricity of the calculated mode. See the description
of ‘--mode’ for more. This mode algorithm finds the mode based on
how symmetric it is, so if the symmetricity returned by this option
is too low, the mode is not too accurate. See Appendix C of
Akhlaghi and Ichikawa 2015 (https://arxiv.org/abs/1505.01664) for a
full description. In practice, symmetricity values larger than 0.2
are mostly good.
‘--modesymvalue’
Print the value in the distribution where the mirror and input
distributions are no longer symmetric, see ‘--mode’ and Appendix C
of Akhlaghi and Ichikawa 2015 (https://arxiv.org/abs/1505.01664)
for more.
‘--sigclip-std’
‘--sigclip-mad’
‘--sigclip-mean’
‘--sigclip-number’
‘--sigclip-median’
Calculate the desired statistic after applying $\sigma$-clipping
(see *note Sigma clipping::, part of the tutorial *note Clipping
outliers::). $\sigma$-clipping configuration is done with the
‘--sclipparams’ option.
Here is one scenario where this can be useful: assume you have a
table and you would like to remove the rows that are outliers (not
within the $\sigma$-clipping range). Let's assume your table is
called ‘table.fits’ and you only want to keep the rows that have a
value in ‘COLUMN’ within the $\sigma$-clipped range (to $3\sigma$,
with a tolerance of 0.1). This command will return the
$\sigma$-clipped median and standard deviation (used to define the
range later).
$ aststatistics table.fits -cCOLUMN --sclipparams=3,0.1 \
--sigclip-median --sigclip-std
You can then use the ‘--range’ option of Table (see *note Table::)
to select the proper rows. But for that, you need the actual
starting and ending values of the range ($m\pm s\sigma$; where $m$
is the median and $s$ is the multiple of sigma to define an
outlier). Therefore, the raw outputs of Statistics in the command
above are not enough.
To get the starting and ending values of the non-outlier range (and
put a '<,>' between them, ready to be used in ‘--range’), pipe the
result into AWK. But in AWK, we will also need the multiple of
$\sigma$, so we will define it as a shell variable (‘s’) before
calling Statistics (note how ‘$s’ is used two times now):
$ s=3
$ aststatistics table.fits -cCOLUMN --sclipparams=$s,0.1 \
--sigclip-median --sigclip-std \
| awk '{s='$s'; printf("%f,%f\n", $1-s*$2, $1+s*$2)}'
To pass it onto Table, we will need to keep the printed output from
the command above in another shell variable (‘r’), not print it.
In Bash, can do this by putting the whole statement within a ‘$()’:
$ s=3
$ r=$(aststatistics table.fits -cCOLUMN --sclipparams=$s,0.1 \
--sigclip-median --sigclip-std \
| awk '{s='$s'; printf("%f,%f\n", $1-s*$2, $1+s*$2)}')
$ echo $r # Just to confirm.
Now you can use Table with the ‘--range’ option to only print the
rows that have a value in ‘COLUMN’ within the desired range:
$ asttable table.fits --range=COLUMN,$r
To save the resulting table (that is clean of outliers) in another
file (for example, named ‘cleaned.fits’, it can also have a ‘.txt’
suffix), just add ‘--output=cleaned.fits’ to the command above.
‘--madclip-std’
‘--madclip-mad’
‘--madclip-mean’
‘--madclip-number’
‘--madclip-median’
Calculate the desired statistic after applying median absolute
deviation (MAD) clipping (see *note MAD clipping::, part of the
tutorial *note Clipping outliers::). MAD-clipping configuration is
done with the ‘--mclipparams’ option.
This option behaves similarly to ‘--sigclip-*’ options, read their
description for usage examples.
‘--concentration=FLT[,FLT[,...]]’
Return the "concentration" around the median (see rest of this
description for the definition); the input value(s) are the
quantile width(s) where it is measured.
For a uniform distribution, the output of this operation will be
approximately $1.0$. With a higher density of values around the
median, the value will be larger for a Gaussian distribution, and
even larger for more concentrated distributions (than a Gaussian).
This is the algorithm used to measure this value:
1. Sort the input dataset and remove all blank values. If there
is one non-blank value or less, then return NaN.
2. The minimum and maximum are respectively selected to be the
second and second-to-last elements in the sorted array. The
first and last elements are not selected as minimum and
maximum because they are affected too strongly by scatter.
3. Subtract each element from the minimum, and divide it by the
difference between the minimum and maximum. After this
operation, the input's values(1) will be between 0 and 1.
This scaling does not change the order of the input elements;
instead, each element's value now shows its relation to the
range of the whole distribution's values (the minimum and
maximum values above).
4. Calculate the scaled values corresponding to quantiles that
are defined by the width above. For example, if the given
width (value to this option) is 0.2, the quantiles of
$0.5-(0.2/2)=0.4$ and $0.5+(0.2/2)=0.5$ will be measured.
5. The width is divided by the difference between the quantiles
and returned as the concentration.
In a uniform distribution, the scaling step will convert each input
into its quantile: the spacing between scaled values will be
uniform. As a result, the difference between the quantiles
measured around the median will be equal to the input width and the
result will be approximately $1.0$. However, if the distribution
is concentrated around the median, the spacing between the scaled
values will be much less around the median and the quantile
difference will be less than the width. Therefore, when we divide
the width by the quantile difference, the value will be larger than
one.
The example commands below create two randomly distributed "noisy"
images, one with a Gaussian distribution and one with a uniform
distribution. We will then run this option on both to see the
different concentrations(2). See *note Generating histograms and
cumulative frequency plots:: on how you can generate the histogram
of these two images on the command-line to visualize the
distribution.
$ astarithmetic 1000 1000 2 makenew 10 mknoise-sigma \
--output=gaussian.fits
$ astarithmetic 1000 1000 2 makenew 10 mknoise-uniform \
--output=uniform.fits
$ aststatistics gaussian.fits --concentration=0.25
3.71347573489440e+00
$ aststatistics uniform.fits --concentration=0.25
9.99988794452348e-01
Note that this option is primarily designed for symmetric
distributions, not skewed ones (where the mode and median will be
distant). Here, we define the "center" in "concentration" as the
median, not the mode. To check if the distribution is symmetric
(that the mode and median are similar), you can use the
‘--quantofmean’ option described above. Recall that you can call
all the options in this section in one call to the Statistics
program like below:
$ aststatistics gaussian.fits \
--quantofmean --concentration=0.25
5.00260500260500e-01 3.71347573489440e+00
From the quantile-of-mean value of approximately 0.5, we see that
the distribution is symmetric and from the concentration, we see
that it is not a uniform one.
---------- Footnotes ----------
(1) Technically, the second sorted value will be 0 and the
second-to-last value will be 1.
(2) The values you get will be slightly different because of the
different random seeds. To get a reproducible result, see *note
Generating random numbers::.
7.1.5.3 Generating histograms and cumulative freq.
..................................................
The list of options below are for those statistical operations that
output more than one value. So while they can be called together in one
run, their outputs will be distinct (each one's output will usually be
printed in more than one line).
‘-A’
‘--asciihist’
Print an ASCII histogram of the usable values within the input
dataset along with some basic information like the example below
(from the UVUDF catalog(1)). The width and height of the histogram
(in units of character widths and heights on your command-line
terminal) can be set with the ‘--numasciibins’ (for the width) and
‘--asciiheight’ options.
For a full description of the histogram, please see *note Histogram
and Cumulative Frequency Plot::. An ASCII plot is certainly very
crude and cannot be used in any publication, but it is very useful
for getting a general feeling of the input dataset very fast and
easily on the command-line without having to take your hands off
the keyboard (which is a major distraction!). If you want to try
it out, you can write it all in one line and ignore the <\> and
extra spaces.
$ aststatistics uvudf_rafelski_2015.fits.gz --hdu=1 \
--column=MAG_F160W --lessthan=40 \
--asciihist --numasciibins=55
ASCII Histogram:
Number: 8593
Y: (linear: 0 to 660)
X: (linear: 17.7735 -- 31.4679, in 55 bins)
| ****
| *****
| ******
| ********
| *********
| ***********
| **************
| *****************
| ***********************
| ********************************
|*** ***************************************************
|-------------------------------------------------------
‘--asciicfp’
Print the cumulative frequency plot of the usable elements in the
input dataset. Please see descriptions under ‘--asciihist’ for
more, the example below is from the same input table as that
example. To better understand the cumulative frequency plot,
please see *note Histogram and Cumulative Frequency Plot::.
$ aststatistics uvudf_rafelski_2015.fits.gz --hdu=1 \
--column=MAG_F160W --lessthan=40 \
--asciicfp --numasciibins=55
ASCII Cumulative frequency plot:
Y: (linear: 0 to 8593)
X: (linear: 17.7735 -- 31.4679, in 55 bins)
| *******
| **********
| ***********
| *************
| **************
| ***************
| *****************
| *******************
| ***********************
| ******************************
|*******************************************************
|-------------------------------------------------------
‘-H’
‘--histogram’
Save the histogram of the usable values in the input dataset into a
table. The first column is the value at the center of the bin and
the second is the number of points in that bin. If the
‘--cumulative’ option is also called with this option in a run,
then the table will have three columns (the third is the cumulative
frequency plot). Through the ‘--numbins’, ‘--onebinstart’, or
‘--manualbinrange’, you can modify the first column values and with
‘--normalize’ and ‘--maxbinone’ you can modify the second columns.
See below for the description of each.
By default (when no ‘--output’ is specified) a plain text table
will be created, see *note Gnuastro text table format::. If a FITS
name is specified, you can use the common option ‘--tableformat’ to
have it as a FITS ASCII or FITS binary format, see *note Common
options::. This table can then be fed into your favorite plotting
tool and get a much more clean and nice histogram than what the raw
command-line can offer you (with the ‘--asciihist’ option).
‘--histogram2d’
Save the 2D histogram of two input columns into an output file, see
*note 2D Histograms::. The output will have three columns: the
first two are the coordinates of each box's center in the first and
second dimensions/columns. The third will be number of input
points that fall within that box.
‘-C’
‘--cumulative’
Save the cumulative frequency plot of the usable values in the
input dataset into a table, similar to ‘--histogram’.
‘--madclip’
Do median absolute deviation (MAD) clipping on the usable pixels of
the input dataset. See *note MAD clipping:: for a description on
MAD-clipping and *note Clipping outliers:: for a complete tutorial
on clipping of outliers. The MAD-clipping parameters can be set
through the ‘--mclipparams’ option (see below).
‘-s’
‘--sigmaclip’
Do $\sigma$-clipping on the usable pixels of the input dataset.
See *note Sigma clipping:: for a full description on
$\sigma$-clipping and *note Clipping outliers:: for a complete
tutorial on clipping of outliers. The $\sigma$-clipping parameters
can be set through the ‘--sclipparams’ option (see below).
‘--mirror=FLT’
Make a histogram and cumulative frequency plot of the mirror
distribution for the given dataset when the mirror is located at
the value to this option. The mirror distribution is fully
described in Appendix C of Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664) and currently it is only used to
calculate the mode (see ‘--mode’).
Just note that the mirror distribution is a discrete distribution
like the input, so while you may give any number as the value to
this option, the actual mirror value is the closest number in the
input dataset to this value. If the two numbers are different,
Statistics will warn you of the actual mirror value used.
This option will make a table as output. Depending on your
selected name for the output, it will be either a FITS table or a
plain text table (which is the default). It contains three
columns: the first is the center of the bins, the second is the
histogram (with the largest value set to 1) and the third is the
normalized cumulative frequency plot of the mirror distribution.
The bins will be positioned such that the mode is on the starting
interval of one of the bins to make it symmetric around the mirror.
With this output file and the input histogram (that you can
generate in another run of Statistics, using the ‘--onebinvalue’),
it is possible to make plots like Figure 21 of Akhlaghi and
Ichikawa 2015 (https://arxiv.org/abs/1505.01664).
The list of options below allow customization of the histogram and
cumulative frequency plots (for the ‘--histogram’, ‘--cumulative’,
‘--asciihist’, and ‘--asciicfp’ options).
‘--numbins’
The number of bins (rows) to use in the histogram and the
cumulative frequency plot tables (outputs of ‘--histogram’ and
‘--cumulative’).
‘--numasciibins’
The number of bins (characters) to use in the ASCII plots when
printing the histogram and the cumulative frequency plot (outputs
of ‘--asciihist’ and ‘--asciicfp’).
‘--asciiheight’
The number of lines to use when printing the ASCII histogram and
cumulative frequency plot on the command-line (outputs of
‘--asciihist’ and ‘--asciicfp’).
‘-n’
‘--normalize’
Normalize the histogram or cumulative frequency plot tables
(outputs of ‘--histogram’ and ‘--cumulative’). For a histogram,
the sum of all bins will become one and for a cumulative frequency
plot the last bin value will be one.
‘--maxbinone’
Divide all the histogram values by the maximum bin value so it
becomes one and the rest are similarly scaled. In some situations
(for example, if you want to plot the histogram and cumulative
frequency plot in one plot) this can be very useful.
‘--onebinstart=FLT’
Make sure that one bin starts with the value to this option. In
practice, this will shift the bins used to find the histogram and
cumulative frequency plot such that one bin's lower interval
becomes this value.
For example, when a histogram range includes negative and positive
values and zero has a special significance in your analysis, then
zero might fall somewhere in one bin. As a result that bin will
have counts of positive and negative. By setting
‘--onebinstart=0’, you can make sure that one bin will only count
negative values in the vicinity of zero and the next bin will only
count positive ones in that vicinity.
Note that by default, the first row of the histogram and cumulative
frequency plot show the central values of each bin. So in the
example above you will not see the 0.000 in the first column, you
will see two symmetric values.
If the value is not within the usable input range, this option will
be ignored. When it is, this option is the last operation before
the bins are finalized, therefore it has a higher priority than
options like ‘--manualbinrange’.
‘--manualbinrange’
Use the values given to the ‘--greaterequal’ and ‘--lessthan’ to
define the range of all bin-based calculations like the histogram.
This option itself does not take any value, but just tells the
program to use the values of those two options instead of the
minimum and maximum values of a plot. If any of the two options
are not given, then the minimum or maximum will be used
respectively. Therefore, if none of them are called calling this
option is redundant.
The ‘--onebinstart’ option has a higher priority than this option.
In other words, ‘--onebinstart’ takes effect after the range has
been finalized and the initial bins have been defined, therefore it
has the power to (possibly) shift the bins. If you want to
manually set the range of the bins _and_ have one bin on a special
value, it is thus better to avoid ‘--onebinstart’.
‘--numbins2=INT’
Similar to ‘--numbins’, but for the second column when a 2D
histogram is requested, see ‘--histogram2d’.
‘--greaterequal2=FLT’
Similar to ‘--greaterequal’, but for the second column when a 2D
histogram is requested, see ‘--histogram2d’.
‘--lessthan2=FLT’
Similar to ‘--lessthan’, but for the second column when a 2D
histogram is requested, see ‘--histogram2d’.
‘--onebinstart2=FLT’
Similar to ‘--onebinstart’, but for the second column when a 2D
histogram is requested, see ‘--histogram2d’.
---------- Footnotes ----------
(1)
7.1.5.4 Fitting options
.......................
With the options below, you can customize the least squares fitting
features of Statistics. For a tutorial of the usage of least squares
fitting in Statistics, please see *note Least squares fitting::. Here,
we will just review the details of each option.
To activate least squares fitting in Statistics, it is necessary to
use the ‘--fit’ option to specify the type of fit you want to do. See
the description of ‘--fit’ for the various available fitting models.
The fitting models that account for weights require three input columns,
while the non-weighted ones only take two input columns. Here is a
summary of the input columns:
1. The first input column is assumed to be the independent variable
(on the horizontal axis of a plot, or $X$ in the equations of each
fit).
2. The second input column is assumed to be the measured value (on the
vertical axis of a plot, or $Y$ in the equation above).
3. The third input column is only for fittings with a weight. It is
assumed to be the "weight" of the measurement column. The nature
of the "weight" can be set with the ‘--fitweight’ option, for
example, if you have the standard deviation of the error in $Y$,
you can use ‘--fitweight=std’ (which is the default, so unless the
default value has been changed, you will not need to set this).
If three columns are given to a model without weight, or two columns
are given to a model that requires weights, Statistics will abort and
inform you. Below you can see an example of fitting with the same
linear model, once weighted and once without weights.
$ aststatistics table.fits --column=X,Y --fit=linear
$ aststatistics table.fits --column=X,Y,Yerr --fit=linear-weighted
The output of the fitting can be in three modes listed below. For a
complete example, see the tutorial in *note Least squares fitting::).
Human friendly format
By default (for example, the commands above) the output is an
elaborate description of the model parameters. For example, $c_0$
and $c_1$ in the linear model ($Y=c_0+c_1X$). Their covariance
matrix and the reduced $\chi^2$ of the fit are also printed on the
output.
Raw numbers
If you don't need the human friendly components of the output
(which are annoying when you want to parse the outputs in some
scenarios), you can use ‘--quiet’ option. Only the raw output
numbers will be printed.
Estimate on a custom X column
Through the ‘--fitestimate’ option, you can specify an independent
table column to estimate the fit (it can also take a single value).
See the description of this option for more.
‘-f STR’
‘--fit=STR’
The name of the fitting method to use. They are based on the
linear (https://www.gnu.org/software/gsl/doc/html/lls.html) and
nonlinear (https://www.gnu.org/software/gsl/doc/html/nls.html)
least-squares fitting functions of the GNU Scientific Library
(GSL).
‘linear’
$Y=c_0+c_1X$
‘linear-weighted’
$Y=c_0+c_1X$; accounting for "weights" in $Y$.
‘linear-no-constant’
$Y=c_1X$.
‘linear-no-constant-weighted’
$Y=c_1X$; accounting for "weights" in $Y$.
‘polynomial’
$Y=c_0+c_1X+c_2X^2+\cdots+c_nX^n$; the maximum required power
($n$) is specified by ‘--fitmaxpower’.
‘polynomial-weighted’
$Y=c_0+c_1X+c_2X^2+\cdots+c_nX^n$; accounting for "weights" in
$Y$. The maximum required power ($n$) is specified by
‘--fitmaxpower’.
‘polynomial-robust’
$Y=c_0+c_1X+c_2X^2+\cdots+c_nX^n$; rejects outliers. The
function to use for outlier removal can be specified with the
‘--fitrobust’ option described below. This model doesn't take
weights since they are calculated internally based on the
outlier removal function (requires two input columns). The
maximum required power ($n$) is specified by ‘--fitmaxpower’.
For a comprehensive review of "robust" fitting and the
available functions, please see the Robust linear regression
(https://www.gnu.org/software/gsl/doc/html/lls.html#robust-linear-regression)
section of the GNU Scientific Library.
‘--fitweight=STR’
The nature of the "weight" column (when a weight is necessary for
the model). It can take one of the following values:
‘std’
Standard deviation of each $Y$ axis measurement: this is the
usual "error" associated with a measurement (for example, in
*note MakeCatalog::) and is the default value to this option.
‘var’
Variance of each $Y$ axis measurement. Assuming a Gaussian
distribution with standard deviation $\sigma$, the variance is
$\sigma^2$.
‘inv-var’
Inverse variance of each $Y$ axis measurement. Assuming a
Gaussian distribution with standard deviation $\sigma$, the
variance is $1/\sigma^2$.
‘--fitmaxpower=INT’
The maximum power (an integer) in a polynomial ($n$ in
$Y=c_0+c_1X+c_2X^2+\cdots+c_nX^n$). This is only relevant when one
of the polynomial models is given to ‘--fit’. The fit will return
$n+1$ coefficients.
‘--fitrobust=STR’
The function for rejecting outliers in the ‘polynomial-robust’
fitting model. For a comprehensive review of "robust" fitting and
the available functions, please see the Robust linear regression
(https://www.gnu.org/software/gsl/doc/html/lls.html#robust-linear-regression)
section of the GNU Scientific Library. This function can take the
following values:
‘bisquare’
Tukey’s biweight (bisquare) function, this is the default
function. According to the GSL manual, this is a good general
purpose weight function.
‘cauchy’
Cauchy’s function (also known as the Lorentzian function). It
doesn't guarantee a unique solution, so it should be used with
care.
‘fair’
The fair function. It guarantees a unique solution and has
continuous derivatives to three orders.
‘huber’
Huber's $\rho$ function. This is also a good general purpose
weight function for rejecting outliers, but can cause
difficulty in some special scenarios.
‘ols’
Ordinary Least Squares (OLS) solution with a constant weight
of unity.
‘welsch’
Welsch function which is useful when the residuals follow an
exponential distribution.
‘--fitestimate=STR/FLT’
Estimate the fitted function at a single point or a complete column
of points. The input $X$ axis positions to estimate the function
can be specified in the following ways:
• A real number: the fitted function will be estimated at that
$X$ position and the corresponding $Y$ and its error will be
printed to standard output.
• ‘self’: in this mode, the same X axis column that was used in
the fit will be used for estimating the fitted function. This
can be useful to visually/easily check the fit, see *note
Least squares fitting::.
• A file name: If the value is none of the above, Statistics
expects it to be a file name containing a table. If the file
is a FITS file, the HDU containing the table should be
specified with the ‘--fitestimatehdu’ option. The column of
the table to use for the $X$ axis points should be specified
with the ‘--fitestimatecol’ option.
The output in this mode can be customized in the following ways:
• If a single floating point value is given ‘--fitestimate’, the
fitted function will be estimated on that point and printed to
standard output.
• When nothing is given to ‘--output’, the independent column
and the estimated values and errors are printed on the
standard output.
• If a file name is given to ‘--output’, the estimated table
above is saved in that file. It can have any of the formats
in *note Recognized table formats::. As a FITS file, all the
fit outputs (coefficients, covariance matrix and reduced
$\chi^2$) are kept as FITS keywords in the same HDU of the
estimated table. For a complete example, see *note Least
squares fitting::.
When the covariance matrix (and thus the $\chi^2$) cannot be
calculated (for example if you only have two rows!), the
printed values on the terminal will be NaN. However, the FITS
standard does not allow NaN values in keyword values!
Therefore, when writing the $\chi^2$ and covariance matrix
elements into the output FITS keywords, the largest value of
the 64-bit floating point type will be written:
$1.79769313486232\times10^{308}$; see *note Numeric data
types::.
• When ‘--quiet’ is given with ‘--fitestimate’, the fitted
parameters are no longer printed on the standard output; they
are available as FITS keywords in the file given to
‘--output’.
‘--fitestimatehdu=STR/INT’
HDU name or counter (counting from zero) that contains the table to
be used for the estimating the fitted function over many points
through ‘--fitestimate’. For more on selecting a HDU, see the
description of ‘--hdu’ in *note Input output options::.
‘--fitestimatecol=STR/INT’
Column name or counter (counting from one) that contains the table
to be used for the estimating the fitted function over many points
through ‘--fitestimate’. See *note Selecting table columns::.
7.1.5.5 Contour options
.......................
Contours are useful to highlight the 2D shape of a certain flux level
over an image. To derive contours in Statistics, you can use the option
below:
‘-R FLT[,FLT[,FLT...]]’
‘--contour=FLT[,FLT[,FLT...]]’
Write the contours for the requested levels in a file ending with
‘_contour.txt’. It will have three columns: the first two are the
coordinates of each point and the third is the level it belongs to
(one of the input values). Each disconnected contour region will
be separated by a blank line. This is the requested format for
adding contours with PGFPlots in LaTeX. If any other format can be
useful for your work please let us know so we can add it. If the
image has World Coordinate System information, the written
coordinates will be in RA and Dec, otherwise, they will be in pixel
coordinates.
Note that currently, this is a very crude/simple implementation,
please let us know if you find problematic situations so we can fix
it.
7.1.5.6 Statistics on tiles
...........................
All the options described until now were from the first class of
operations discussed above: those that treat the whole dataset as one.
However, it often happens that the relative position of the dataset
elements over the dataset is significant. For example, you do not want
one median value for the whole input image, you want to know how the
median changes over the image. For such operations, the input has to be
tessellated (see *note Tessellation::). Thus this class of options
cannot currently be called along with the options above in one run of
Statistics.
‘-t’
‘--ontile’
Do the respective single-valued calculation over one tile of the
input dataset, not the whole dataset. This option must be called
with at least one of the single valued options discussed above (for
example, ‘--mean’ or ‘--quantile’). The output will be a file in
the same format as the input. If the ‘--oneelempertile’ option is
called, then one element/pixel will be used for each tile (see
*note Processing options::). Otherwise, the output will have the
same size as the input, but each element will have the value
corresponding to that tile's value. If multiple single valued
operations are called, then for each operation there will be one
extension in the output FITS file.
‘-y’
‘--sky’
Estimate the Sky value on each tile as fully described in *note
Quantifying signal in a tile::. As described in that section,
several options are necessary to configure the Sky estimation which
are listed below. The output file will have two extensions: the
first is the Sky value and the second is the Sky standard deviation
on each tile. Similar to ‘--ontile’, if the ‘--oneelempertile’
option is called, then one element/pixel will be used for each tile
(see *note Processing options::).
The parameters for estimating the sky value can be set with the
following options, except for the ‘--sclipparams’ option (which is also
used by the ‘--sigmaclip’), the rest are only used for the Sky value
estimation.
‘-k=FITS’
‘--kernel=FITS’
File name of kernel to help in estimating the significance of
signal in a tile, see *note Quantifying signal in a tile::.
‘--khdu=STR’
Kernel HDU to help in estimating the significance of signal in a
tile, see *note Quantifying signal in a tile::.
‘--meanmedqdiff=FLT’
The maximum acceptable distance between the quantiles of the mean
and median, see *note Quantifying signal in a tile::. The initial
Sky and its standard deviation estimates are measured on tiles
where the quantiles of their mean and median are less distant than
the value given to this option. For example, ‘--meanmedqdiff=0.01’
means that only tiles where the mean's quantile is between 0.49 and
0.51 (recall that the median's quantile is 0.5) will be used.
‘--sclipparams=FLT,FLT’
The $\sigma$-clipping parameters, see *note Sigma clipping::. This
option takes two values which are separated by a comma (<,>). Each
value can either be written as a single number or as a fraction of
two numbers (for example, ‘3,1/10’). The first value to this
option is the multiple of $\sigma$ that will be clipped ($\alpha$
in that section). The second value is the exit criteria. If it is
less than 1, then it is interpreted as tolerance and if it is
larger than one it is a specific number. Hence, in the latter case
the value must be an integer.
‘--mclipparams=FLT,FLT’
The MAD-clipping parameters. This is very similar to
‘--sclipparams’ above, see there for more.
‘--outliersclip=FLT,FLT’
$\sigma$-clipping parameters for the outlier rejection of the Sky
value (similar to ‘--sclipparams’).
Outlier rejection is useful when the dataset contains a large and
diffuse (almost flat within each tile) signal. The flatness of the
profile will cause it to successfully pass the mean-median quantile
difference test, so we will need to use the distribution of
successful tiles for removing these false positive. For more, see
the latter half of *note Quantifying signal in a tile::.
‘--outliernumngb=INT’
Number of neighboring tiles to use for outlier rejection (mostly
the wings of bright stars or galaxies). If this option is given a
value of zero, no outlier rejection will take place. For more see
the latter half of *note Quantifying signal in a tile::.
‘--outliersigma=FLT’
Multiple of sigma to define an outlier in the Sky value estimation.
If this option is given a value of zero, no outlier rejection will
take place. For more see ‘--outliersclip’ and the latter half of
*note Quantifying signal in a tile::.
‘--smoothwidth=INT’
Width of a flat kernel to convolve the interpolated tile values.
Tile interpolation is done using the median of the ‘--interpnumngb’
neighbors of each tile (see *note Processing options::). If this
option is given a value of zero or one, no smoothing will be done.
Without smoothing, strong boundaries will probably be created
between the values estimated for each tile. It is thus good to
smooth the interpolated image so strong discontinuities do not show
up in the final Sky values. The smoothing is done through
convolution (see *note Convolution process::) with a flat kernel,
so the value to this option must be an odd number.
‘--ignoreblankintiles’
Do not set the input's blank pixels to blank in the tiled outputs
(for example, Sky and Sky standard deviation extensions of the
output). This is only applicable when the tiled output has the
same size as the input, in other words, when ‘--oneelempertile’ is
not called.
By default, blank values in the input (commonly on the edges which
are outside the survey/field area) will be set to blank in the
tiled outputs also. But in other scenarios this default behavior
is not desired; for example, if you have masked something in the
input, but want the tiled output under that also.
‘--checksky’
Create a multi-extension FITS file showing the steps that were used
to estimate the Sky value over the input, see *note Quantifying
signal in a tile::. The file will have two extensions for each
step (one for the Sky and one for the Sky standard deviation).
‘--checkskynointerp’
Similar to ‘--checksky’, but it will stop as soon as the outlier
tiles have been identified and before it interpolates the values to
cover the whole image.
This is useful when you want the good tile values before
interpolation, and don't want to slow down your pipeline with the
extra computing that interpolation and smoothing require.
7.2 NoiseChisel
===============
Once instrumental signatures are removed from the raw data (image) in
the initial reduction process (see *note Data manipulation::). You are
naturally eager to start answering the scientific questions that
motivated the data collection in the first place. However, the raw
dataset/image is just an array of values/pixels, that is all! These raw
values cannot directly be used to answer your scientific questions; for
example, "how many galaxies are there in the image?" and "What is their
magnitude?".
The first high-level step your analysis will therefore be to
classify, or label, the dataset elements (pixels) into two classes: 1)
Noise, where random effects are the major contributor to the value, and
2) Signal, where non-random factors (for example, light from a distant
galaxy) are present. This classification of the elements in a dataset
is formally known as _detection_.
In an observational/experimental dataset, signal is always buried in
noise: only mock/simulated datasets are free of noise. Therefore
detection, or the process of separating signal from noise, determines
the number of objects you study and the accuracy of any higher-level
measurement you do on them. Detection is thus the most important step
of any analysis and is not trivial. In particular, the most
scientifically interesting astronomical targets are faint, can have a
large variety of morphologies, along with a large distribution in
magnitude and size. Therefore when noise is significant, proper
detection of your targets is a uniquely decisive step in your final
scientific analysis/result.
NoiseChisel is Gnuastro's program for detection of targets that do
not have a sharp border (almost all astronomical objects). When the
targets have sharp edges/borders (for example, cells in biological
imaging), a simple threshold is enough to separate them from noise and
each other (if they are not touching). To detect such sharp-edged
targets, you can use Gnuastro's Arithmetic program in a command like
below (assuming the threshold is ‘100’, see *note Arithmetic::):
$ astarithmetic in.fits 100 gt 2 connected-components
Since almost no astronomical target has such sharp edges, we need a
more advanced detection methodology. NoiseChisel uses a new noise-based
paradigm for detection of very extended and diffuse targets that are
drowned deeply in the ocean of noise. It was initially introduced in
Akhlaghi and Ichikawa 2015 (https://arxiv.org/abs/1505.01664) and
improvements after the first four were published in Akhlaghi 2019
(https://arxiv.org/abs/1909.11230). Please take the time to go through
these papers to most effectively understand the need of NoiseChisel and
how best to use it.
The name of NoiseChisel is derived from the first thing it does after
thresholding the dataset: to erode it. In mathematical morphology,
erosion on pixels can be pictured as carving-off boundary pixels.
Hence, what NoiseChisel does is similar to what a wood chisel or stone
chisel do. It is just not a hardware, but a software. In fact, looking
at it as a chisel and your dataset as a solid cube of rock will greatly
help in effectively understanding and optimally using it: with
NoiseChisel you literally carve your targets out of the noise. Try
running it with the ‘--checkdetection’ option, and open the temporary
output as a multi-extension cube, to see each step of the carving
process on your input dataset (see *note Viewing FITS file contents with
DS9 or TOPCAT::).
NoiseChisel's primary output is a binary detection map with the same
size as the input but its pixels only have two values: 0 (background)
and 1 (foreground). Pixels that do not harbor any detected signal
(noise) are given a label (or value) of zero and those with a value of 1
have been identified as hosting signal.
Segmentation is the process of classifying the signal into
higher-level constructs. For example, if you have two separate galaxies
in one image, NoiseChisel will give a value of 1 to the pixels of both
(each forming an "island" of touching foreground pixels). After
segmentation, the connected foreground pixels will get separate labels,
enabling you to study them individually. NoiseChisel is only focused on
detection (separating signal from noise), to _segment_ the signal (into
separate galaxies for example), Gnuastro has a separate specialized
program *note Segment::. NoiseChisel's output can be directly/readily
fed into Segment.
For more on NoiseChisel's output format and its benefits (especially
in conjunction with *note Segment:: and later *note MakeCatalog::),
please see Akhlaghi 2016 (https://arxiv.org/abs/1611.06387). Just note
that when that paper was published, Segment was not yet spun-off into a
separate program, and NoiseChisel done both detection and segmentation.
NoiseChisel's output is designed to be generic enough to be easily
used in any higher-level analysis. If your targets are not touching
after running NoiseChisel and you are not interested in their
sub-structure, you do not need the Segment program at all. You can ask
NoiseChisel to find the connected pixels in the output with the
‘--label’ option. In this case, the output will not be a binary image
any more, the signal will have counters/labels starting from 1 for each
connected group of pixels. You can then directly feed NoiseChisel's
output into MakeCatalog for measurements over the detections and the
production of a catalog (see *note MakeCatalog::).
Thanks to the published papers mentioned above, there is no need to
provide a more complete introduction to NoiseChisel in this book.
However, published papers cannot be updated any more, but the software
has evolved/changed. The changes since publication are documented in
*note NoiseChisel changes after publication::. In *note Invoking
astnoisechisel::, the details of running NoiseChisel and its options are
discussed.
As discussed above, detection is one of the most important steps for
your scientific result. It is therefore very important to obtain a good
understanding of NoiseChisel (and afterwards *note Segment:: and *note
MakeCatalog::). We strongly recommend reviewing two tutorials of *note
General program usage tutorial:: and *note Detecting large extended
targets::. They are designed to show how to most effectively use
NoiseChisel for the detection of small faint objects and large extended
objects. In the meantime, they also show the modular principle behind
Gnuastro's programs and how they are built to complement, and build
upon, each other.
*note General program usage tutorial:: culminates in using
NoiseChisel to detect galaxies and use its outputs to find the galaxy
colors. Defining colors is a very common process in most science-cases.
Therefore it is also recommended to (patiently) complete that tutorial
for optimal usage of NoiseChisel in conjunction with all the other
Gnuastro programs. *note Detecting large extended targets:: shows you
can optimize NoiseChisel's settings for very extended objects to
successfully carve out to signal-to-noise ratio levels of below 1/10.
After going through those tutorials, play a little with the settings (in
the order presented in the paper and *note Invoking astnoisechisel::) on
a dataset you are familiar with and inspect all the check images
(options starting with ‘--check’) to see the effect of each parameter.
Below, in *note Invoking astnoisechisel::, we will review
NoiseChisel's input, detection, and output options in *note NoiseChisel
input::, *note Detection options::, and *note NoiseChisel output::. If
you have used NoiseChisel within your research, please run it with
‘--cite’ to list the papers you should cite and how to acknowledge its
funding sources.
7.2.1 NoiseChisel changes after publication
-------------------------------------------
NoiseChisel was initially introduced in Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664) and updates after the first four
years were published in Akhlaghi 2019
(https://arxiv.org/abs/1909.11230). To help in understanding how it
works, those papers have many figures showing every step on multiple
mock and real examples. We recommended to read these papers for a good
understanding of what it does and how each parameter influences the
output.
However, the papers cannot be updated anymore, but NoiseChisel has
evolved (and will continue to do so): better algorithms or steps have
been found and implemented and some options have been added, removed or
changed behavior. This book is thus the final and definitive guide to
NoiseChisel. The aim of this section is to make the transition from the
papers above to the installed version on your system, as smooth as
possible with the list below. For a more detailed list of changes in
each Gnuastro version, please see the ‘NEWS’ file(1).
• An improved outlier rejection for identifying tiles without any
signal has been implemented in the quantile-threshold phase: Prior
to version 0.14, outliers were defined globally: the distribution
of all tiles with an acceptable ‘--meanmedqdiff’ was inspected and
outliers were found and rejected. However, this caused problems
when there are strong gradients over the image (for example, an
image prior to flat-fielding, or in the presence of a large
foreground galaxy). In these cases, the faint wings of
galaxies/stars could be mistakenly identified as Sky (leaving a
footprint of the object on the Sky output) and wrongly subtracted.
It was possible to play with the parameters to correct this for
that particular dataset, but that was frustrating. Therefore from
version 0.14, instead of finding outliers from the full tile
distribution, we now measure the _slope_ of the tile's nearby tiles
and find outliers locally. Three options have been added to
configure this part of NoiseChisel: ‘--outliernumngb’,
‘--outliersclip’ and ‘--outliersigma’. For more on the local
outlier-by-distance algorithm and the definition of _slope_
mentioned above, see *note Quantifying signal in a tile::. In our
tests, this gave a much improved estimate of the quantile
thresholds and final Sky values with default values.
---------- Footnotes ----------
(1) The ‘NEWS’ file is present in the released Gnuastro tarball, see
*note Release tarball::.
7.2.2 Invoking NoiseChisel
--------------------------
NoiseChisel will detect signal in noise producing a multi-extension
dataset containing a binary detection map which is the same size as the
input. Its output can be readily used for input into *note Segment::,
for higher-level segmentation, or *note MakeCatalog:: to do measurements
and generate a catalog. The executable name is ‘astnoisechisel’ with
the following general template
$ astnoisechisel [OPTION ...] InputImage.fits
One line examples:
## Detect signal in input.fits.
$ astnoisechisel input.fits
## Inspect all the detection steps after changing a parameter.
$ astnoisechisel input.fits --qthresh=0.4 --checkdetection
## Detect signal assuming input has 4 amplifier channels along first
## dimension and 1 along the second. Also set the regular tile size
## to 100 along both dimensions:
$ astnoisechisel --numchannels=4,1 --tilesize=100,100 input.fits
If NoiseChisel is to do processing (for example, you do not want to get
help, or see the values to each input parameter), an input image should
be provided with the recognized extensions (see *note Arguments::).
NoiseChisel shares a large set of common operations with other Gnuastro
programs, mainly regarding input/output, general processing steps, and
general operating modes. To help in a unified experience between all of
Gnuastro's programs, these operations have the same command-line
options, see *note Common options:: for a full list/description (they
are not repeated here).
As in all Gnuastro programs, options can also be given to NoiseChisel
in configuration files. For a thorough description on Gnuastro's
configuration file parsing, please see *note Configuration files::. All
of NoiseChisel's options with a short description are also always
available on the command-line with the ‘--help’ option, see *note
Getting help::. To inspect the option values without actually running
NoiseChisel, append your command with ‘--printparams’ (or ‘-P’).
NoiseChisel's input image may contain blank elements (see *note Blank
pixels::). Blank elements will be ignored in all steps of NoiseChisel.
Hence if your dataset has bad pixels which should be masked with a mask
image, please use Gnuastro's *note Arithmetic:: program (in particular
its ‘where’ operator) to convert those pixels to blank pixels before
running NoiseChisel. Gnuastro's Arithmetic program has bitwise
operators helping you select specific kinds of bad-pixels when
necessary.
A convolution kernel can also be optionally given. If a value (file
name) is given to ‘--kernel’ on the command-line or in a configuration
file (see *note Configuration files::), then that file will be used to
convolve the image prior to thresholding. Otherwise a default kernel
will be used. For a 2D image, the default kernel is a 2D Gaussian with
a FWHM of 2 pixels truncated at 5 times the FWHM. This choice of the
default kernel is discussed in Section 3.1.1 of Akhlaghi and Ichikawa
2015 (https://arxiv.org/abs/1505.01664). For a 3D cube, it is a
Gaussian with FWHM of 1.5 pixels in the first two dimensions and 0.75
pixels in the third dimension. See *note Convolution kernel:: for
kernel related options. Passing ‘none’ to ‘--kernel’ will disable
convolution. On the other hand, through the ‘--convolved’ option, you
may provide an already convolved image, see descriptions below for more.
NoiseChisel defines two tessellations over the input (see *note
Tessellation::). This enables it to deal with possible gradients in the
input dataset and also significantly improve speed by processing each
tile on different threads simultaneously. Tessellation related options
are discussed in *note Processing options::. In particular, NoiseChisel
uses two tessellations (with everything between them identical except
the tile sizes): a fine-grained one with smaller tiles (used in
thresholding and Sky value estimations) and another with larger tiles
which is used for pseudo-detections over non-detected regions of the
image. The common Tessellation options described in *note Processing
options:: define all parameters of both tessellations. The large tile
size for the latter tessellation is set through the ‘--largetilesize’
option. To inspect the tessellations on your input dataset, run
NoiseChisel with ‘--checktiles’.
*Usage TIP:* Frequently use the options starting with ‘--check’. Since
the noise properties differ between different datasets, you can often
play with the parameters/options for a better result than the default
parameters. You can start with ‘--checkdetection’ for the main steps.
For the full list of NoiseChisel's checking options please run:
$ astnoisechisel --help | grep check
*Not detecting wings of bright galaxies:* In such cases, probably the
best solution is to increase ‘--outliernumngb’ (to reject tiles that are
affected by very flat diffuse signal). For more, see *note Quantifying
signal in a tile::.
When working on 3D data cubes, the tessellation options need three
values and updating them every time can be annoying/buggy. To simplify
the job, NoiseChisel also installs a ‘astnoisechisel-3d.conf’
configuration file (see *note Configuration files::). You can use this
for default values on data cubes. For example, if you installed
Gnuastro with the prefix ‘/usr/local’ (the default location, see *note
Installation directory::), you can benefit from this configuration file
by running NoiseChisel like the example below.
$ astnoisechisel cube.fits \
--config=/usr/local/etc/gnuastro/astnoisechisel-3d.conf
To further simplify the process, you can define a shell alias in any
startup file (for example, ‘~/.bashrc’, see *note Installation
directory::). Assuming that you installed Gnuastro in ‘/usr/local’, you
can add this line to the startup file (you may put it all in one line,
it is broken into two lines here for fitting within page limits).
alias astnoisechisel-3d="astnoisechisel \
--config=/usr/local/etc/gnuastro/astnoisechisel-3d.conf"
Using this alias, you can call NoiseChisel with the name
‘astnoisechisel-3d’ (instead of ‘astnoisechisel’). It will
automatically load the 3D specific configuration file first, and then
parse any other arguments, options or configuration files. You can
change the default values in this 3D configuration file by calling them
on the command-line as you do with ‘astnoisechisel’(1). For example:
$ astnoisechisel-3d --numchannels=3,3,1 cube.fits
Below, we will discuss NoiseChisel's options, classified into
separate sub-sections to help in easy navigation. *note NoiseChisel
input:: discusses the basic options relating to input file(s) and data;
these have no effect on the the detection process. Afterwards, *note
Detection options:: fully describes every configuration parameter
(option) related to detection and how they affect the final result. The
order of options in this section follow the logical order within
NoiseChisel. On first reading (while you are still new to NoiseChisel),
it is therefore strongly recommended to read the options in the given
order below. The output of ‘--printparams’ (or ‘-P’) also has this
order. However, the output of ‘--help’ is sorted alphabetically.
Finally, in *note NoiseChisel output:: the format of NoiseChisel's
output is discussed.
---------- Footnotes ----------
(1) Recall that for single-invocation options, the last command-line
invocation takes precedence over all previous invocations (including
those in the 3D configuration file). See the description of ‘--config’
in *note Operating mode options::.
7.2.2.1 NoiseChisel input
.........................
The options here can be used to configure the inputs and output of
NoiseChisel, along with some general processing options. Recall that
you can always see the full list of Gnuastro's options with the ‘--help’
(see *note Getting help::), or ‘--printparams’ (or ‘-P’) to see their
values (see *note Operating mode options::).
‘-k FITS’
‘--kernel=FITS’
File name of kernel to smooth the image before applying the
threshold, see *note Convolution kernel::. If no convolution is
needed, give this option a value of ‘none’.
The first step of NoiseChisel is to convolve/smooth the image and
use the convolved image in multiple steps including the finding and
applying of the quantile threshold (see ‘--qthresh’). The
‘--kernel’ option is not mandatory. If not called, for a 2D, image
a 2D Gaussian profile with a FWHM of 2 pixels truncated at 5 times
the FWHM is used. This choice of the default kernel is discussed
in Section 3.1.1 of Akhlaghi and Ichikawa [2015].
For a 3D cube, when no file name is given to ‘--kernel’, a Gaussian
with FWHM of 1.5 pixels in the first two dimensions and 0.75 pixels
in the third dimension will be used. The reason for this
particular configuration is that commonly in astronomical
applications, 3D datasets do not have the same nature in all three
dimensions, commonly the first two dimensions are spatial (RA and
Dec) while the third is spectral (for example, wavelength). The
samplings are also different, in the default case, the spatial
sampling is assumed to be larger than the spectral sampling, hence
a wider FWHM in the spatial directions, see *note Sampling
theorem::.
You can use MakeProfiles to build a kernel with any of its
recognized profile types and parameters. For more details, please
see *note MakeProfiles output dataset::. For example, the command
below will make a Moffat kernel (with $\beta=2.8$) with FWHM of 2
pixels truncated at 10 times the FWHM.
$ astmkprof --oversample=1 --kernel=moffat,2,2.8,10
Since convolution can be the slowest step of NoiseChisel, for large
datasets, you can convolve the image once with Gnuastro's Convolve
(see *note Convolve::), and use the ‘--convolved’ option to feed it
directly to NoiseChisel. This can help getting faster results when
you are playing/testing the higher-level options.
‘--khdu=STR’
HDU containing the kernel in the file given to the ‘--kernel’
option.
‘--convolved=FITS’
Use this file as the convolved image and do not do convolution
(ignore ‘--kernel’). NoiseChisel will just check the size of the
given dataset is the same as the input's size. If a wrong image
(with the same size) is given to this option, the results (errors,
bugs, etc.) are unpredictable. So please use this option with
care and in a highly controlled environment, for example, in the
scenario discussed below.
In almost all situations, as the input gets larger, the single most
CPU (and time) consuming step in NoiseChisel (and other programs
that need a convolved image) is convolution. Therefore minimizing
the number of convolutions can save a significant amount of time in
some scenarios. One such scenario is when you want to segment
NoiseChisel's detections using the same kernel (with *note
Segment::, which also supports this ‘--convolved’ option). This
scenario would require two convolutions of the same dataset: once
by NoiseChisel and once by Segment. Using this option in both
programs, only one convolution (prior to running NoiseChisel) is
enough.
Another common scenario where this option can be convenient is when
you are testing NoiseChisel (or Segment) for the best parameters.
You have to run NoiseChisel multiple times and see the effect of
each change. However, once you are happy with the kernel,
re-convolving the input on every change of higher-level parameters
will greatly hinder, or discourage, further testing. With this
option, you can convolve the input image with your chosen kernel
once before running NoiseChisel, then feed it to NoiseChisel on
each test run and thus save valuable time for better/more tests.
To build your desired convolution kernel, you can use *note
MakeProfiles::. To convolve the image with a given kernel you can
use *note Convolve::. Spatial domain convolution is mandatory: in
the frequency domain, blank pixels (if present) will cover the
whole image and gradients will appear on the edges, see *note
Spatial vs. Frequency domain::.
Below you can see an example of the second scenario: you want to
see how variation of the growth level (through the ‘--detgrowquant’
option) will affect the final result. Recall that you can ignore
all the extra spaces, new lines, and backslash's ('‘\’') if you are
typing in the terminal. In a shell script, remove the ‘$’ signs at
the start of the lines.
## Make the kernel to convolve with.
$ astmkprof --oversample=1 --kernel=gaussian,2,5
## Convolve the input with the given kernel.
$ astconvolve input.fits --kernel=kernel.fits \
--domain=spatial --output=convolved.fits
## Run NoiseChisel with seven growth quantile values.
$ for g in 60 65 70 75 80 85 90; do \
astnoisechisel input.fits --convolved=convolved.fits \
--detgrowquant=0.$g --output=$g.fits; \
done
‘--chdu=STR’
The HDU/extension containing the convolved image in the file given
to ‘--convolved’.
‘-w FITS’
‘--widekernel=FITS’
File name of a wider kernel to use in estimating the difference of
the mode and median in a tile (this difference is used to identify
the significance of signal in that tile, see *note Quantifying
signal in a tile::). As displayed in Figure 4 of Akhlaghi and
Ichikawa 2015 (https://arxiv.org/abs/1505.01664), a wider kernel
will help in identifying the skewness caused by data in noise. The
image that is convolved with this kernel is _only_ used for this
purpose. Once the mode is found to be sufficiently close to the
median, the quantile threshold is found on the image convolved with
the sharper kernel (‘--kernel’), see ‘--qthresh’).
Since convolution will significantly slow down the processing, this
feature is optional. When it is not given, the image that is
convolved with ‘--kernel’ will be used to identify good tiles _and_
apply the quantile threshold. This option is mainly useful in
conditions were you have a very large, extended, diffuse signal
that is still present in the usable tiles when using ‘--kernel’.
See *note Detecting large extended targets:: for a practical
demonstration on how to inspect the tiles used in identifying the
quantile threshold.
‘--whdu=STR’
HDU containing the kernel file given to the ‘--widekernel’ option.
‘-L INT[,INT]’
‘--largetilesize=INT[,INT]’
The size of each tile for the tessellation with the larger tile
sizes. Except for the tile size, all the other parameters for this
tessellation are taken from the common options described in *note
Processing options::. The format is identical to that of the
‘--tilesize’ option that is discussed in that section.
7.2.2.2 Detection options
.........................
Detection is the process of separating the pixels in the image into two
groups: 1) Signal, and 2) Noise. Through the parameters below, you can
customize the detection process in NoiseChisel. Recall that you can
always see the full list of NoiseChisel's options with the ‘--help’ (see
*note Getting help::), or ‘--printparams’ (or ‘-P’) to see their values
(see *note Operating mode options::).
‘-Q FLT’
‘--meanmedqdiff=FLT’
The maximum acceptable distance between the quantiles of the mean
and median in each tile, see *note Quantifying signal in a tile::.
The quantile threshold estimates are measured on tiles where the
quantiles of their mean and median are less distant than the value
given to this option. For example, ‘--meanmedqdiff=0.01’ means
that only tiles where the mean's quantile is between 0.49 and 0.51
(recall that the median's quantile is 0.5) will be used.
‘-a INT’
‘--outliernumngb=INT’
Number of neighboring tiles to use for outlier rejection (mostly
the wings of bright stars or galaxies). For optimal detection of
the wings of bright stars or galaxies, this is *the most important*
option in NoiseChisel. This is because the extended wings of
bright galaxies or stars (the PSF) can become flat over the tile.
In this case, they will satisfy the ‘--meanmedqdiff’ condition and
pass that step. Therefore, to correctly identify such bad tiles,
we need to look at the neighboring nearby tiles. A tile that is on
the wing of a bright galaxy/star will clearly be an outlier when
looking at the neighbors. For more on the details of the outlier
rejection algorithm, see the latter half of *note Quantifying
signal in a tile::. If this option is given a value of zero, no
outlier rejection will take place.
‘--outliersclip=FLT,FLT’
$\sigma$-clipping parameters for the outlier rejection of the
quantile threshold. The format of the given values is similar to
‘--sigmaclip’ below. In NoiseChisel, outlier rejection on tiles is
used when identifying the quantile thresholds (‘--qthresh’,
‘--noerodequant’, and ‘detgrowquant’).
Outlier rejection is useful when the dataset contains a large and
diffuse (almost flat within each tile) signal. The flatness of the
profile will cause it to successfully pass the mean-median quantile
difference test, so we will need to use the distribution of
successful tiles for removing these false positives. For more, see
the latter half of *note Quantifying signal in a tile::.
‘--outliersigma=FLT’
Multiple of sigma to define an outlier. If this option is given a
value of zero, no outlier rejection will take place. For more see
‘--outliersclip’ and the latter half of *note Quantifying signal in
a tile::.
‘-t FLT’
‘--qthresh=FLT’
The quantile threshold to apply to the convolved image. The
detection process begins with applying a quantile threshold to each
of the tiles in the small tessellation. The quantile is only
calculated for tiles that do not have any significant signal within
them, see *note Quantifying signal in a tile::. Interpolation is
then used to give a value to the unsuccessful tiles and it is
finally smoothed.
The quantile value is a floating point value between 0 and 1.
Assume that we have sorted the $N$ data elements of a distribution
(the pixels in each mesh on the convolved image). The quantile
($q$) of this distribution is the value of the element with an
index of (the nearest integer to) $q\times{N}$ in the sorted data
set. After thresholding is complete, we will have a binary (two
valued) image. The pixels above the threshold are known as
foreground pixels (have a value of 1) while those which lie below
the threshold are known as background (have a value of 0).
‘--smoothwidth=INT’
Width of flat kernel used to smooth the interpolated quantile
thresholds, see ‘--qthresh’ for more.
‘--checkqthresh’
Check the quantile threshold values on the mesh grid. A
multi-extension FITS file, suffixed with ‘_qthresh.fits’ will be
created showing each step of how the final quantile threshold is
found. With this option, NoiseChisel will abort as soon as
quantile estimation has been completed, allowing you to inspect the
steps leading to the final quantile threshold, this can be disabled
with ‘--continueaftercheck’. By default the output will have the
same pixel size as the input, but with the ‘--oneelempertile’
option, only one pixel will be used for each tile (see *note
Processing options::).
The key things to remember are:
• The measurements to find the thresholds are done on tiles that
cover the whole image in a tessellation. Recall that you can
set the size of tiles with ‘--tilesize’ and check them with
‘--checktiles’. Therefore except for the first and last
extensions, the rest only show tiles.
• NoiseChisel ultimately has three thresholds: the quantile
threshold (that you set with ‘--qthresh’), the no-erode
quantile (set with ‘--noerodequant’) and the growth quantile
(set with ‘--detgrowquant’). Therefore for each step, we have
three extensions.
The output file will have the following extensions. Below, the
extensions are put in the same order as you see in the file, with
their name.
‘CONVOLVED’
This is the input image after convolution with the kernel
(which is a FWHM=2 Gaussian by default, but you can change
with ‘--kernel’). Recall that the thresholds are defined on
the convolved image.
‘QTHRESH_ERODE’
‘QTHRESH_NOERODE’
‘QTHRESH_EXPAND’
In these three extensions, the tiles that have a
quantile-of-mean more/less than 0.5 (quantile of median) $\pm
d$ are set to NaN ($d$ is the value given to ‘--meanmedqdiff’,
see *note Quantifying signal in a tile::). Therefore the
non-NaN tiles that you see here are the tiles where there is
no significant skewness (changing signal) within that tile.
The only differing thing between the three extensions is the
values of the non-NaN tiles. These values will be used to
construct the final threshold map over the whole image.
‘VALUE1_NO_OUTLIER’
‘VALUE2_NO_OUTLIER’
‘VALUE3_NO_OUTLIER’
All outlier tiles have been masked. The reason for removing
outliers is that the quantile-of-mean is only sensitive to
signal that varies on a scale that is smaller than the tile
size. Therefore the extended wings of large galaxies or
bright stars (which vary on scales much larger than the tile
size) will pass that test. As described in *note Quantifying
signal in a tile:: outlier rejection is customized through
‘--outliernumngb’, ‘--outliersclip’ and ‘--outliersigma’.
‘THRESH1_INTERP’
‘THRESH2_INTERP’
‘THRESH3_INTERP’
Using the successful values that remain after the previous
step, give values to all (interpolate) the tiles in the image.
The interpolation is done using the nearest-neighbor method:
for each tile, the N nearest neighbors are found and the
median of their values is used to fill it. You can set the
value of N through the ‘--interpnumngb’ option.
‘THRESH1_SMOOTH’
‘THRESH2_SMOOTH’
‘THRESH3_SMOOTH’
Smooth the interpolated image to remove the strong differences
between touching tiles. Because we used the median value of
the N nearest neighbors in the previous step, there can be
strong discontinuities on the edges of tiles (which can
directly show in the image after applying the threshold). The
scale of the smoothing (number of nearby tiles to smooth with)
is set with the ‘--smoothwidth’ option.
‘QTHRESH-APPLIED’
The pixels in this image can only have three values:
‘0’
These pixels had a value below the quantile threshold.
‘1’
These pixels had a value above the quantile threshold,
but below the threshold for no erosion. Therefore in the
next step, NoiseChisel will erode (set them to 0) these
pixels if they are touching a 0-valued pixel.
‘2’
These pixels had a value above the no-erosion threshold.
So NoiseChisel will not erode these pixels, it will only
apply Opening to them afterwards. Recall that this was
done to avoid loosing sharp point-sources (like stars in
space-based imaging).
‘--blankasforeground’
In the erosion and opening steps below, treat blank elements as
foreground (regions above the threshold). By default, blank
elements in the dataset are considered to be background, so if a
foreground pixel is touching it, it will be eroded. This option is
irrelevant if the datasets contains no blank elements.
When there are many blank elements in the dataset, treating them as
foreground will systematically erode their regions less, therefore
systematically creating more false positives. So use this option
(when blank values are present) with care.
‘-e INT’
‘--erode=INT’
The number of erosions to apply to the binary thresholded image.
Erosion is simply the process of flipping (from 1 to 0) any of the
foreground pixels that neighbor a background pixel. In a 2D image,
there are two kinds of neighbors, 4-connected and 8-connected
neighbors. In a 3D dataset, there are three: 6-connected,
18-connected, and 26-connected. You can specify which class of
neighbors should be used for erosion with the ‘--erodengb’ option,
see below.
Erosion has the effect of shrinking the foreground pixels. To put
it another way, it expands the holes. This is a founding principle
in NoiseChisel: it exploits the fact that with very low thresholds,
the holes in the very low surface brightness regions of an image
will be smaller than regions that have no signal. Therefore by
expanding those holes, we are able to separate the regions
harboring signal.
‘--erodengb=INT’
The type of neighborhood (structuring element) used in erosion, see
‘--erode’ for an explanation on erosion. If the input is a 2D
image, only two integer values are acceptable: 4 or 8. For a 3D
input data cube, the acceptable values are: 6, 18 and 26.
In 2D 4-connectivity, the neighbors of a pixel are defined as the
four pixels on the top, bottom, right and left of a pixel that
share an edge with it. The 8-connected neighbors on the other hand
include the 4-connected neighbors along with the other 4 pixels
that share a corner with this pixel. See Figure 6 (a) and (b) in
Akhlaghi and Ichikawa (2015) for a demonstration. A similar
argument applies to 3D data cubes.
‘--noerodequant’
Pure erosion is going to carve off sharp and small objects
completely out of the detected regions. This option can be used to
avoid missing such sharp and small objects (which have significant
pixels, but not over a large area). All pixels with a value larger
than the significance level specified by this option will not be
eroded during the erosion step above. However, they will undergo
the erosion and dilation of the opening step below.
Like the ‘--qthresh’ option, the significance level is determined
using the quantile (a value between 0 and 1). Just as a reminder,
in the normal distribution, $1\sigma$, $1.5\sigma$, and $2\sigma$
are approximately on the 0.84, 0.93, and 0.98 quantiles.
‘-p INT’
‘--opening=INT’
Depth of opening to be applied to the eroded binary image. Opening
is a composite operation. When opening a binary image with a depth
of $n$, $n$ erosions (explained in ‘--erode’) are followed by $n$
dilations. Simply put, dilation is the inverse of erosion. When
dilating an image any background pixel is flipped (from 0 to 1) to
become a foreground pixel. Dilation has the effect of fattening
the foreground. Note that in NoiseChisel, the erosion which is
part of opening is independent of the initial erosion that is done
on the thresholded image (explained in ‘--erode’). The structuring
element for the opening can be specified with the ‘--openingngb’
option. Opening has the effect of removing the thin foreground
connections (mostly noise) between separate foreground 'islands'
(detections) thereby completely isolating them. Once opening is
complete, we have _initial_ detections.
‘--openingngb=INT’
The structuring element used for opening, see ‘--erodengb’ for more
information about a structuring element.
‘--skyfracnoblank’
Ignore blank pixels when estimating the fraction of undetected
pixels for Sky estimation. NoiseChisel only measures the Sky over
the tiles that have a sufficiently large fraction of undetected
pixels (value given to ‘--minskyfrac’). By default this fraction
is found by dividing number of undetected pixels in a tile by the
tile's area. But this default behavior ignores the possibility of
blank pixels. In situations that blank/masked pixels are scattered
across the image and if they are large enough, all the tiles can
fail the ‘--minskyfrac’ test, thus not allowing NoiseChisel to
proceed. With this option, such scenarios can be fixed: the
denominator of the fraction will be the number of non-blank
elements in the tile, not the total tile area.
‘-B FLT’
‘--minskyfrac=FLT’
Minimum fraction (value between 0 and 1) of Sky (undetected) areas
in a tile. Only tiles with a fraction of undetected pixels (Sky)
larger than this value will be used to estimate the Sky value.
NoiseChisel uses this option value twice to estimate the Sky value:
after initial detections and in the end when false detections have
been removed.
Because of the PSF and their intrinsic amorphous properties,
astronomical objects (except cosmic rays) never have a clear cutoff
and commonly sink into the noise very slowly. Even below the very
low thresholds used by NoiseChisel. So when a large fraction of
the area of one mesh is covered by detections, it is very plausible
that their faint wings are present in the undetected regions (hence
causing a bias in any measurement). To get an accurate measurement
of the above parameters over the tessellation, tiles that harbor
too many detected regions should be excluded. The used tiles are
visible in the respective ‘--check’ option of the given step.
‘--checkdetsky’
Check the initial approximation of the sky value and its standard
deviation in a FITS file ending with ‘_detsky.fits’. With this
option, NoiseChisel will abort as soon as the sky value used for
defining pseudo-detections is complete. This allows you to inspect
the steps leading to the final quantile threshold, this behavior
can be disabled with ‘--continueaftercheck’. By default the output
will have the same pixel size as the input, but with the
‘--oneelempertile’ option, only one pixel will be used for each
tile (see *note Processing options::).
‘-s FLT,FLT’
‘--sigmaclip=FLT,FLT’
The $\sigma$-clipping parameters for measuring the initial and
final Sky values from the undetected pixels, see *note Sigma
clipping::.
This option takes two values which are separated by a comma (<,>).
Each value can either be written as a single number or as a
fraction of two numbers (for example, ‘3,1/10’). The first value
to this option is the multiple of $\sigma$ that will be clipped
($\alpha$ in that section). The second value is the exit criteria.
If it is less than 1, then it is interpreted as tolerance and if it
is larger than one it is assumed to be the fixed number of
iterations. Hence, in the latter case the value must be an
integer.
‘-R FLT’
‘--dthresh=FLT’
The detection threshold: a multiple of the initial Sky standard
deviation added with the initial Sky approximation (which you can
inspect with ‘--checkdetsky’). This flux threshold is applied to
the initially undetected regions on the unconvolved image. The
background pixels that are completely engulfed in a 4-connected
foreground region are converted to background (holes are filled)
and one opening (depth of 1) is applied over both the initially
detected and undetected regions. The Signal to noise ratio of the
resulting 'pseudo-detections' are used to identify true vs. false
detections. See Section 3.1.5 and Figure 7 in Akhlaghi and
Ichikawa (2015) for a very complete explanation.
‘--dopening=INT’
The number of openings to do after applying ‘--dthresh’.
‘--dopeningngb=INT’
The connectivity used in the opening of ‘--dopening’. In a 2D
image this must be either 4 or 8. The stronger the connectivity,
the more smaller regions will be discarded.
‘--holengb=INT’
The connectivity (defined by the number of neighbors) to fill holes
after applying ‘--dthresh’ (above) to find pseudo-detections. For
example, in a 2D image it must be 4 (the neighbors that are most
strongly connected) or 8 (all neighbors). The stronger the
connectivity, the stronger the hole will be enclosed. So setting a
value of 8 in a 2D image means that the walls of the hole are
4-connected. If standard (near Sky level) values are given to
‘--dthresh’, setting ‘--holengb=4’, might fill the complete dataset
and thus not create enough pseudo-detections.
‘--pseudoconcomp=INT’
The connectivity (defined by the number of neighbors) to find
individual pseudo-detections. If it is a weaker connectivity (4 in
a 2D image), then pseudo-detections that are connected on the
corners will be treated as separate.
‘-m INT’
‘--snminarea=INT’
The minimum area to calculate the Signal to noise ratio on the
pseudo-detections of both the initially detected and undetected
regions. When the area in a pseudo-detection is too small, the
Signal to noise ratio measurements will not be accurate and their
distribution will be heavily skewed to the positive. So it is best
to ignore any pseudo-detection that is smaller than this area. Use
‘--detsnhistnbins’ to check if this value is reasonable or not.
‘--checksn’
Save the S/N values of the pseudo-detections (and possibly grown
detections if ‘--cleangrowndet’ is called) into separate tables.
If ‘--tableformat’ is a FITS table, each table will be written into
a separate extension of one file suffixed with ‘_detsn.fits’. If
it is plain text, a separate file will be made for each table
(ending in ‘_detsn_sky.txt’, ‘_detsn_det.txt’ and
‘_detsn_grown.txt’). For more on ‘--tableformat’ see *note Input
output options::.
You can use these to inspect the S/N values and their distribution
(in combination with the ‘--checkdetection’ option to see where the
pseudo-detections are). You can use Gnuastro's *note Statistics::
to make a histogram of the distribution or any other analysis you
would like for better understanding of the distribution (for
example, through a histogram).
‘--minnumfalse=INT’
The minimum number of 'pseudo-detections' over the undetected
regions to identify a Signal-to-Noise ratio threshold. The Signal
to noise ratio (S/N) of false pseudo-detections in each tile is
found using the quantile of the S/N distribution of the
pseudo-detections over the undetected pixels in each mesh. If the
number of S/N measurements is not large enough, the quantile will
not be accurate (can have large scatter). For example, if you set
‘--snquant=0.99’ (or the top 1 percent), then it is best to have at
least 100 S/N measurements.
‘-c FLT’
‘--snquant=FLT’
The quantile of the Signal to noise ratio distribution of the
pseudo-detections in each mesh to use for filling the large mesh
grid. Note that this is only calculated for the large mesh grids
that satisfy the minimum fraction of undetected pixels (value of
‘--minbfrac’) and minimum number of pseudo-detections (value of
‘--minnumfalse’).
‘--snthresh=FLT’
Manually set the signal-to-noise ratio of true pseudo-detections.
With this option, NoiseChisel will not attempt to find
pseudo-detections over the noisy regions of the dataset, but will
directly go onto applying the manually input value.
This option is useful in crowded images where there is no blank sky
to find the sky pseudo-detections. You can get this value on a
similarly reduced dataset (from another region of the Sky with more
undetected regions spaces).
‘-d FLT’
‘--detgrowquant=FLT’
Quantile limit to "grow" the final detections. As discussed in the
previous options, after applying the initial quantile threshold,
layers of pixels are carved off the objects to identify true
signal. With this step you can return those low surface brightness
layers that were carved off back to the detections. To disable
growth, set the value of this option to ‘1’.
The process is as follows: after the true detections are found, all
the non-detected pixels above this quantile will be put in a list
and used to "grow" the true detections (seeds of the growth). Like
all quantile thresholds, this threshold is defined and applied to
the convolved dataset. Afterwards, the dataset is dilated once
(with minimum connectivity) to connect very thin regions on the
boundary: imagine building a dam at the point rivers spill into an
open sea/ocean. Finally, all holes are filled. In the geography
metaphor, holes can be seen as the closed (by the dams) rivers and
lakes, so this process is like turning the water in all such rivers
and lakes into soil. See ‘--detgrowmaxholesize’ for configuring
the hole filling.
Note that since the growth occurs on all neighbors of a data
element, the quantile for 3D detection must be must larger than
that of 2D detection. Recall that in 2D each element has 8
neighbors while in 3D there are 27 neighbors.
‘--detgrowmaxholesize=INT’
The maximum hole size to fill during the final expansion of the
true detections as described in ‘--detgrowquant’. This is
necessary when the input contains many smaller objects and can be
used to avoid marking blank sky regions as detections.
For example, multiple galaxies can be positioned such that they
surround an empty region of sky. If all the holes are filled, the
Sky region in between them will be taken as a detection which is
not desired. To avoid such cases, the integer given to this option
must be smaller than the hole between such objects. However, we
should caution that unless the "hole" is very large, the combined
faint wings of the galaxies might actually be present in between
them, so be very careful in not filling such holes.
On the other hand, if you have a very large (and extended) galaxy,
the diffuse wings of the galaxy may create very large holes over
the detections. In such cases, a large enough value to this option
will cause all such holes to be detected as part of the large
galaxy and thus help in detecting it to extremely low surface
brightness limits. Therefore, especially when large and extended
objects are present in the image, it is recommended to give this
option (very) large values. For one real-world example, see *note
Detecting large extended targets::.
‘--cleangrowndet’
After dilation, if the signal-to-noise ratio of a detection is less
than the derived pseudo-detection S/N limit, that detection will be
discarded. In an ideal/clean noise, a true detection's S/N should
be larger than its constituent pseudo-detections because its area
is larger and it also covers more signal. However, on a false
detections (especially at lower ‘--snquant’ values), the increase
in size can cause a decrease in S/N below that threshold.
This will improve purity and not change completeness (a true
detection will not be discarded). Because a true detection has
flux in its vicinity and dilation will catch more of that flux and
increase the S/N. So on a true detection, the final S/N cannot be
less than pseudo-detections.
However, in many real images bad processing creates artifacts that
cannot be accurately removed by the Sky subtraction. In such
cases, this option will decrease the completeness (will
artificially discard true detections). So this feature is not
default and should to be explicitly called when you know the noise
is clean.
‘--checkdetection’
Every step of the detection process will be added as an extension
to a file with the suffix ‘_det.fits’. Going through each would
just be a repeat of the explanations above and also of those in
Akhlaghi and Ichikawa (2015). The extension label should be
sufficient to recognize which step you are observing. Viewing all
the steps can be the best guide in choosing the best set of
parameters. With this option, NoiseChisel will abort as soon as a
snapshot of all the detection process is saved. This behavior can
be disabled with ‘--continueaftercheck’.
‘--checksky’
Check the derivation of the final sky and its standard deviation
values on the mesh grid. With this option, NoiseChisel will abort
as soon as the sky value is estimated over the image (on each
tile). This behavior can be disabled with ‘--continueaftercheck’.
By default the output will have the same pixel size as the input,
but with the ‘--oneelempertile’ option, only one pixel will be used
for each tile (see *note Processing options::).
7.2.2.3 NoiseChisel output
..........................
NoiseChisel's output is a multi-extension FITS file. The main
extension/dataset is a (binary) detection map. It has the same size as
the input but with only two possible values for all pixels: 0 (for
pixels identified as noise) and 1 (for those identified as
signal/detections). The detection map is followed by a Sky and Sky
standard deviation dataset (which are calculated from the binary image).
By default (when ‘--rawoutput’ is not called), NoiseChisel will also
subtract the Sky value from the input and save the sky-subtracted input
as the first extension in the output with data. The zero-th extension
(that contains no data), contains NoiseChisel's configuration as FITS
keywords, see *note Output FITS files::.
The name of the output file can be set by giving a value to
‘--output’ (this is a common option between all programs and is
therefore discussed in *note Input output options::). If ‘--output’ is
not used, the input name will be suffixed with ‘_detected.fits’ and used
as output, see *note Automatic output::. If any of the options starting
with ‘--check*’ are given, NoiseChisel will not complete and will abort
as soon as the respective check images are created. For more
information on the different check images, see the description for the
‘--check*’ options in *note Detection options:: (this can be disabled
with ‘--continueaftercheck’).
The last two extensions of the output are the Sky and its Standard
deviation, see *note Sky value:: for a complete explanation. They are
calculated on the tile grid that you defined for NoiseChisel. By
default these datasets will have the same size as the input, but with
all the pixels in one tile given one value. To be more space-efficient
(keep only one pixel per tile), you can use the ‘--oneelempertile’
option, see *note Tessellation::.
To inspect any of NoiseChisel's output files, assuming you use SAO
DS9, you can configure your Graphic User Interface (GUI) to open
NoiseChisel's output as a multi-extension data cube. This will allow
you to flip through the different extensions and visually inspect the
results. This process has been described for the GNOME GUI (most common
GUI in GNU/Linux operating systems) in *note Viewing FITS file contents
with DS9 or TOPCAT::.
NoiseChisel's output configuration options are described in detail
below.
‘--continueaftercheck’
Continue NoiseChisel after any of the options starting with
‘--check’ (see *note Detection options::. NoiseChisel involves
many steps and as a result, there are many checks, allowing you to
inspect the status of the processing. The results of each step
affect the next steps of processing. Therefore, when you want to
check the status of the processing at one step, the time spent to
complete NoiseChisel is just wasted/distracting time.
To encourage easier experimentation with the option values, when
you use any of the NoiseChisel options that start with ‘--check’,
NoiseChisel will abort once its desired extensions have been
written. With ‘--continueaftercheck’ option, you can disable this
behavior and ask NoiseChisel to continue with the rest of the
processing, even after the requested check files are complete.
‘--ignoreblankintiles’
Do not set the input's blank pixels to blank in the tiled outputs
(for example, Sky and Sky standard deviation extensions of the
output). This is only applicable when the tiled output has the
same size as the input, in other words, when ‘--oneelempertile’ is
not called.
By default, blank values in the input (commonly on the edges which
are outside the survey/field area) will be set to blank in the
tiled outputs also. But in other scenarios this default behavior
is not desired; for example, if you have masked something in the
input, but want the tiled output under that also.
‘-l’
‘--label’
Run a connected-components algorithm on the finally detected pixels
to identify which pixels are connected to which. By default the
main output is a binary dataset with only two values: 0 (for noise)
and 1 (for signal/detections). See *note NoiseChisel output:: for
more.
The purpose of NoiseChisel is to detect targets that are extended
and diffuse, with outer parts that sink into the noise very
gradually (galaxies and stars for example). Since NoiseChisel digs
down to extremely low surface brightness values, many such targets
will commonly be detected together as a single large body of
connected pixels.
To properly separate connected objects, sophisticated segmentation
methods are commonly necessary on NoiseChisel's output. Gnuastro
has the dedicated *note Segment:: program for this job. Since
input images are commonly large and can take a significant volume,
the extra volume necessary to store the labels of the connected
components in the detection map (which will be created with this
‘--label’ option, in 32-bit signed integer type) can thus be a
major waste of space. Since the default output is just a binary
dataset, an 8-bit unsigned dataset is enough.
The binary output will also encourage users to segment the result
separately prior to doing higher-level analysis. As an alternative
to ‘--label’, if you have the binary detection image, you can use
the ‘connected-components’ operator in Gnuastro's Arithmetic
program to identify regions that are connected with each other.
For example, with this command (assuming NoiseChisel's output is
called ‘nc.fits’):
$ astarithmetic nc.fits 2 connected-components -hDETECTIONS
‘--rawoutput’
Do not include the Sky-subtracted input image as the first
extension of the output. By default, the Sky-subtracted input is
put in the first extension of the output. The next extensions are
NoiseChisel's main outputs described above.
The extra Sky-subtracted input can be convenient in checking
NoiseChisel's output and comparing the detection map with the
input: visually see if everything you expected is detected
(reasonable completeness) and that you do not have too many false
detections (reasonable purity). This visual inspection is
simplified if you use SAO DS9 to view NoiseChisel's output as a
multi-extension data-cube, see *note Viewing FITS file contents
with DS9 or TOPCAT::.
When you are satisfied with your NoiseChisel configuration
(therefore you do not need to check on every run), or you want to
archive/transfer the outputs, or the datasets become large, or you
are running NoiseChisel as part of a pipeline, this Sky-subtracted
input image can be a significant burden (take up a large volume).
The fact that the input is also noisy, makes it hard to compress it
efficiently.
In such cases, this ‘--rawoutput’ can be used to avoid the extra
sky-subtracted input in the output. It is always possible to
easily produce the Sky-subtracted dataset from the input (assuming
it is in extension ‘1’ of ‘in.fits’) and the ‘SKY’ extension of
NoiseChisel's output (let's call it ‘nc.fits’) with a command like
below (assuming NoiseChisel was not run with ‘--oneelempertile’,
see *note Tessellation::):
$ astarithmetic in.fits nc.fits - -h1 -hSKY
*Save space:* with the ‘--rawoutput’ and ‘--oneelempertile’,
NoiseChisel's output will only be one binary detection map and two much
smaller arrays with one value per tile. Since none of these have noise
they can be compressed very effectively (without any loss of data) with
exceptionally high compression ratios. This makes it easy to archive,
or transfer, NoiseChisel's output even on huge datasets. To compress it
with the most efficient method (take up less volume), run the following
command:
$ gzip --best noisechisel_output.fits
The resulting ‘.fits.gz’ file can then be fed into any of Gnuastro's
programs directly, or viewed in viewers like SAO DS9, without having to
decompress it separately (they will just take a little longer, because
they have to internally decompress it before starting). See *note
NoiseChisel optimization for storage:: for an example on a real dataset.
7.3 Segment
===========
Once signal is separated from noise (for example, with *note
NoiseChisel::), you have a binary dataset: each pixel is either signal
(1) or noise (0). Signal (for example, every galaxy in your image) has
been "detected", but all detections have a label of 1. Therefore while
we know which pixels contain signal, we still cannot find out how many
galaxies they contain or which detected pixels correspond to which
galaxy. At the lowest (most generic) level, detection is a kind of
segmentation (segmenting the whole dataset into signal and noise, see
*note NoiseChisel::). Here, we will define segmentation only on signal:
to separate sub-structure within the detections.
If the targets are clearly separated, or their detected regions are
not touching, a simple connected components(1) algorithm (very basic
segmentation) is enough to separate the regions that are
touching/connected. This is such a basic and simple form of
segmentation that Gnuastro's Arithmetic program has an operator for it:
see ‘connected-components’ in *note Arithmetic operators::. Assuming
the binary dataset is called ‘binary.fits’, you can use it with a
command like this:
$ astarithmetic binary.fits 2 connected-components
You can even do a very basic detection (a threshold, say at value ‘100’)
_and_ segmentation in Arithmetic with a single command like below:
$ astarithmetic in.fits 100 gt 2 connected-components
However, in most astronomical situations our targets are not nicely
separated or have a sharp boundary/edge (for a threshold to suffice):
they touch (for example, merging galaxies), or are simply in the same
line-of-sight (which is much more common). This causes their images to
overlap.
In particular, when you do your detection with NoiseChisel, you will
detect signal to very low surface brightness limits: deep into the faint
wings of galaxies or bright stars (which can extend very far and
irregularly from their center). Therefore, it often happens that
several galaxies are detected as one large detection. Since they are
touching, a simple connected components algorithm will not suffice. It
is therefore necessary to do a more sophisticated segmentation and break
up the detected pixels (even those that are touching) into multiple
target objects as accurately as possible.
Segment will use a detection map and its corresponding dataset to
find sub-structure over the detected areas and use them for its
segmentation. Until Gnuastro version 0.6 (released in 2018), Segment
was part of *note NoiseChisel::. Therefore, similar to NoiseChisel, the
best place to start reading about Segment and understanding what it does
(with many illustrative figures) is Section 3.2 of Akhlaghi and Ichikawa
2015 (https://arxiv.org/abs/1505.01664), and continue with Akhlaghi 2019
(https://arxiv.org/abs/1909.11230).
As a summary, Segment first finds true _clump_s over the detections.
Clumps are associated with local maxima/minima(2) and extend over the
neighboring pixels until they reach a local minimum/maximum
(_river_/_watershed_). By default, Segment will use the distribution of
clump signal-to-noise ratios over the undetected regions as reference to
find "true" clumps over the detections. Using the undetected regions
can be disabled by directly giving a signal-to-noise ratio to
‘--clumpsnthresh’.
The true clumps are then grown to a certain threshold over the
detections. Based on the strength of the connections
(rivers/watersheds) between the grown clumps, they are considered parts
of one _object_ or as separate _object_s. See Section 3.2 of Akhlaghi
and Ichikawa 2015 (https://arxiv.org/abs/1505.01664) for more.
Segment's main output are thus two labeled datasets: 1) clumps, and 2)
objects. See *note Segment output:: for more.
To start learning about Segment, especially in relation to detection
(*note NoiseChisel::) and measurement (*note MakeCatalog::), the
recommended references are Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664), Akhlaghi 2016
(https://arxiv.org/abs/1611.06387) and Akhlaghi 2019
(https://arxiv.org/abs/1909.11230). If you have used Segment within
your research, please run it with ‘--cite’ to list the papers you should
cite and how to acknowledge its funding sources.
Those papers cannot be updated any more but the software will evolve.
For example, Segment became a separate program (from NoiseChisel) in
2018 (after those papers were published). Therefore this book is the
definitive reference. Finally, in *note Invoking astsegment::, we will
discuss Segment's inputs, outputs and configuration options.
---------- Footnotes ----------
(1)
(2) By default the maximum is used as the first clump pixel, to
define clumps based on local minima, use the ‘--minima’ option.
7.3.1 Invoking Segment
----------------------
Segment will identify substructure within the detected regions of an
input image. Segment's output labels can be directly used for
measurements (for example, with *note MakeCatalog::). The executable
name is ‘astsegment’ with the following general template
$ astsegment [OPTION ...] InputImage.fits
One line examples:
## Segment NoiseChisel's detected regions.
$ astsegment default-noisechisel-output.fits
## Use a hand-input S/N value for keeping true clumps
## (avoid finding the S/N using the undetected regions).
$ astsegment nc-out.fits --clumpsnthresh=10
## Inspect all the segmentation steps after changing a parameter.
$ astsegment input.fits --snquant=0.9 --checksegmentaion
## Use the fixed value of 0.01 for the input's Sky standard deviation
## (in the units of the input), and assume all the pixels are a
## detection (for example, a large structure extending over the whole
## image), and only keep clumps with S/N>10 as true clumps.
$ astsegment in.fits --std=0.01 --detection=all --clumpsnthresh=10
If Segment is to do processing (for example, you do not want to get
help, or see the values of each option), at least one input dataset is
necessary along with detection and error information, either as separate
datasets (per-pixel) or fixed values, see *note Segment input::.
Segment shares a large set of common operations with other Gnuastro
programs, mainly regarding input/output, general processing steps, and
general operating modes. To help in a unified experience between all of
Gnuastro's programs, these common operations have the same names and
defined in *note Common options::.
As in all Gnuastro programs, options can also be given to Segment in
configuration files. For a thorough description of Gnuastro's
configuration file parsing, please see *note Configuration files::. All
of Segment's options with a short description are also always available
on the command-line with the ‘--help’ option, see *note Getting help::.
To inspect the option values without actually running Segment, append
your command with ‘--printparams’ (or ‘-P’).
To help in easy navigation between Segment's options, they are
separately discussed in the three sub-sections below: *note Segment
input:: discusses how you can customize the inputs to Segment. *note
Segmentation options:: is devoted to options specific to the high-level
segmentation process. Finally, in *note Segment output::, we will
discuss options that affect Segment's output.
7.3.1.1 Segment input
.....................
Besides the input dataset (for example, astronomical image), Segment
also needs to know the Sky standard deviation and the regions of the
dataset that it should segment. The values dataset is assumed to be Sky
subtracted by default. If it is not, you can ask Segment to subtract
the Sky internally by calling ‘--sky’. For the rest of this discussion,
we will assume it is already sky subtracted.
The Sky and its standard deviation can be a single value (to be used
for the whole dataset) or a separate dataset (for a separate value per
pixel). If a dataset is used for the Sky and its standard deviation,
they must either be the size of the input image, or have a single value
per tile (generated with ‘--oneelempertile’, see *note Processing
options:: and *note Tessellation::).
The detected regions/pixels can be specified as a detection map (for
example, see *note NoiseChisel output::). If ‘--detection=all’, Segment
will not read any detection map and assume the whole input is a single
detection. For example, when the dataset is fully covered by a large
nearby galaxy/globular cluster.
When dataset are to be used for any of the inputs, Segment will
assume they are multiple extensions of a single file by default (when
‘--std’ or ‘--detection’ are not called). For example, NoiseChisel's
default output *note NoiseChisel output::. When the Sky-subtracted
values are in one file, and the detection and Sky standard deviation are
in another, you just need to use ‘--detection’: in the absence of
‘--std’, Segment will look for both the detection labels and Sky
standard deviation in the file given to ‘--detection’. Ultimately, if
all three are in separate files, you need to call both ‘--detection’ and
‘--std’.
The extensions of the three mandatory inputs can be specified with
‘--hdu’, ‘--dhdu’, and ‘--stdhdu’. For a full discussion on what to
give to these options, see the description of ‘--hdu’ in *note Input
output options::. To see their default values (along with all the other
options), run Segment with the ‘--printparams’ (or ‘-P’) option. Just
recall that in the absence of ‘--detection’ and ‘--std’, all three are
assumed to be in the same file. If you only want to see Segment's
default values for HDUs on your system, run this command:
$ astsegment -P | grep hdu
By default Segment will convolve the input with a kernel to improve
the signal-to-noise ratio of true peaks. If you already have the
convolved input dataset, you can pass it directly to Segment for faster
processing (using the ‘--convolved’ and ‘--chdu’ options). Just do not
forget that the convolved image must also be Sky-subtracted before
calling Segment. If a value/file is given to ‘--sky’, the convolved
values will also be Sky subtracted internally. Alternatively, if you
prefer to give a kernel (with ‘--kernel’ and ‘--khdu’), Segment can do
the convolution internally. To disable convolution, use
‘--kernel=none’.
‘--sky=STR/FLT’
The Sky value(s) to subtract from the input. This option can
either be given a constant number or a file name containing a
dataset (multiple values, per pixel or per tile). By default,
Segment will assume the input dataset is Sky subtracted, so this
option is not mandatory.
If the value cannot be read as a number, it is assumed to be a file
name. When the value is a file, the extension can be specified
with ‘--skyhdu’. When it is not a single number, the given dataset
must either have the same size as the output or the same size as
the tessellation (so there is one pixel per tile, see *note
Tessellation::).
When this option is given, its value(s) will be subtracted from the
input and the (optional) convolved dataset (given to ‘--convolved’)
prior to starting the segmentation process.
‘--skyhdu=STR/INT’
The HDU/extension containing the Sky values. This is mandatory
when the value given to ‘--sky’ is not a number. Please see the
description of ‘--hdu’ in *note Input output options:: for the
different ways you can identify a special extension.
‘--std=STR/FLT’
The Sky standard deviation value(s) corresponding to the input.
The value can either be a constant number or a file name containing
a dataset (multiple values, per pixel or per tile). The Sky
standard deviation is mandatory for Segment to operate.
If the value cannot be read as a number, it is assumed to be a file
name. When the value is a file, the extension can be specified
with ‘--skyhdu’. When it is not a single number, the given dataset
must either have the same size as the output or the same size as
the tessellation (so there is one pixel per tile, see *note
Tessellation::).
When this option is not called, Segment will assume the standard
deviation is a dataset and in a HDU/extension (‘--stdhdu’) of
another one of the input file(s). If a file is given to
‘--detection’, it will assume that file contains the standard
deviation dataset, otherwise, it will look into input filename (the
main argument, without any option).
‘--stdhdu=INT/STR’
The HDU/extension containing the Sky standard deviation values,
when the value given to ‘--std’ is a file name. Please see the
description of ‘--hdu’ in *note Input output options:: for the
different ways you can identify a special extension.
‘--variance’
The input Sky standard deviation value/dataset is actually
variance. When this option is called, the square root of input Sky
standard deviation (see ‘--std’) is used internally, not its raw
value(s).
‘-d FITS’
‘--detection=FITS’
Detection map to use for segmentation. If given a value of ‘all’,
Segment will assume the whole dataset must be segmented, see below.
If a detection map is given, the extension can be specified with
‘--dhdu’. If not given, Segment will assume the desired
HDU/extension is in the main input argument (input file specified
with no option).
The final segmentation (clumps or objects) will only be over the
non-zero pixels of this detection map. The dataset must have the
same size as the input image. Only datasets with an integer type
are acceptable for the labeled image, see *note Numeric data
types::. If your detection map only has integer values, but it is
stored in a floating point container, you can use Gnuastro's
Arithmetic program (see *note Arithmetic::) to convert it to an
integer container, like the example below:
$ astarithmetic float.fits int32 --output=int.fits
It may happen that the whole input dataset is covered by signal,
for example, when working on parts of the Andromeda galaxy, or
nearby globular clusters (that cover the whole field of view). In
such cases, segmentation is necessary over the complete dataset,
not just specific regions (detections). By default Segment will
first use the undetected regions as a reference to find the proper
signal-to-noise ratio of "true" clumps (give a purity level
specified with ‘--snquant’). Therefore, in such scenarios you also
need to manually give a "true" clump signal-to-noise ratio with the
‘--clumpsnthresh’ option to disable looking into the undetected
regions, see *note Segmentation options::. In such cases, is
possible to make a detection map that only has the value ‘1’ for
all pixels (for example, using *note Arithmetic::), but for
convenience, you can also use ‘--detection=all’.
‘--dhdu’
The HDU/extension containing the detection map given to
‘--detection’. Please see the description of ‘--hdu’ in *note
Input output options:: for the different ways you can identify a
special extension.
‘-k FITS’
‘--kernel=FITS’
The name of file containing kernel that will be used to convolve
the input image. The usage of this option is identical to
NoiseChisel's ‘--kernel’ option (*note NoiseChisel input::).
Please see the descriptions there for more. To disable
convolution, you can give it a value of ‘none’.
‘--khdu’
The HDU/extension containing the kernel used for convolution. For
acceptable values, please see the description of ‘--hdu’ in *note
Input output options::.
‘--convolved=FITS’
The convolved image's file name to avoid internal convolution by
Segment. The usage of this option is identical to NoiseChisel's
‘--convolved’ option. Please see *note NoiseChisel input:: for a
thorough discussion of the usefulness and best practices of using
this option.
If you want to use the same convolution kernel for detection (with
*note NoiseChisel::) and segmentation, with this option, you can
use the same convolved image (that is also available in
NoiseChisel) and avoid two convolutions. However, just be careful
to use the input to NoiseChisel as the input to Segment also, then
use the ‘--sky’ and ‘--std’ to specify the Sky and its standard
deviation (from NoiseChisel's output). Recall that when
NoiseChisel is not called with ‘--rawoutput’, the first extension
of NoiseChisel's output is the _Sky-subtracted_ input (see *note
NoiseChisel output::). So if you use the same convolved image that
you fed to NoiseChisel, but use NoiseChisel's output with Segment's
‘--convolved’, then the convolved image will not be Sky subtracted.
‘--chdu’
The HDU/extension containing the convolved image (given to
‘--convolved’). For acceptable values, please see the description
of ‘--hdu’ in *note Input output options::.
‘-L INT[,INT]’
‘--largetilesize=INT[,INT]’
The size of the large tiles to use for identifying the clump S/N
threshold over the undetected regions. The usage of this option is
identical to NoiseChisel's ‘--largetilesize’ option (*note
NoiseChisel input::). Please see the descriptions there for more.
The undetected regions can be a significant fraction of the dataset
and finding clumps requires sorting of the desired regions, which
can be slow. To speed up the processing, Segment finds clumps in
the undetected regions over separate large tiles. This allows it
to have to sort a much smaller set of pixels and also to treat them
independently and in parallel. Both these issues greatly speed it
up. Just be sure to not decrease the large tile sizes too much
(less than 100 pixels in each dimension). It is important for them
to be much larger than the clumps.
7.3.1.2 Segmentation options
............................
The options below can be used to configure every step of the
segmentation process in the Segment program. For a more complete
explanation (with figures to demonstrate each step), please see Section
3.2 of Akhlaghi and Ichikawa 2015 (https://arxiv.org/abs/1505.01664),
and also *note Segment::. By default, Segment will follow the procedure
described in the paper to find the S/N threshold based on the noise
properties. This can be disabled by directly giving a trustable
signal-to-noise ratio to the ‘--clumpsnthresh’ option.
Recall that you can always see the full list of Gnuastro's options
with the ‘--help’ (see *note Getting help::), or ‘--printparams’ (or
‘-P’) to see their values (see *note Operating mode options::).
‘-B FLT’
‘--minskyfrac=FLT’
Minimum fraction (value between 0 and 1) of Sky (undetected) areas
in a large tile. Only (large) tiles with a fraction of undetected
pixels (Sky) greater than this value will be used for finding
clumps. The clumps found in the undetected areas will be used to
estimate a S/N threshold for true clumps. Therefore this is an
important option (to decrease) in crowded fields. Operationally,
this is almost identical to NoiseChisel's ‘--minskyfrac’ option
(*note Detection options::). Please see the descriptions there for
more.
‘--minima’
Build the clumps based on the local minima, not maxima. By
default, clumps are built starting from local maxima (see Figure 8
of Akhlaghi and Ichikawa 2015 (https://arxiv.org/abs/1505.01664)).
Therefore, this option can be useful when you are searching for
true local minima (for example, absorption features).
‘-m INT’
‘--snminarea=INT’
The minimum area which a clump in the undetected regions should
have in order to be considered in the clump Signal to noise ratio
measurement. If this size is set to a small value, the Signal to
noise ratio of false clumps will not be accurately found. It is
recommended that this value be larger than the value to
NoiseChisel's ‘--snminarea’. Because the clumps are found on the
convolved (smoothed) image while the pseudo-detections are found on
the input image. You can use ‘--checksn’ and ‘--checksegmentation’
to see if your chosen value is reasonable or not.
‘--checksn’
Save the S/N values of the clumps over the sky and detected regions
into separate tables. If ‘--tableformat’ is a FITS format, each
table will be written into a separate extension of one file
suffixed with ‘_clumpsn.fits’. If it is plain text, a separate
file will be made for each table (ending in ‘_clumpsn_sky.txt’ and
‘_clumpsn_det.txt’). For more on ‘--tableformat’ see *note Input
output options::.
You can use these tables to inspect the S/N values and their
distribution (in combination with the ‘--checksegmentation’ option
to see where the clumps are). You can use Gnuastro's *note
Statistics:: to make a histogram of the distribution (ready for
plotting in a text file, or a crude ASCII-art demonstration on the
command-line).
With this option, Segment will abort as soon as the two tables are
created. This allows you to inspect the steps leading to the final
S/N quantile threshold, this behavior can be disabled with
‘--continueaftercheck’.
‘--minnumfalse=INT’
The minimum number of clumps over undetected (Sky) regions to
identify the requested Signal-to-Noise ratio threshold.
Operationally, this is almost identical to NoiseChisel's
‘--minnumfalse’ option (*note Detection options::). Please see the
descriptions there for more.
‘-c FLT’
‘--snquant=FLT’
The quantile of the signal-to-noise ratio distribution of clumps in
undetected regions, used to define true clumps. After identifying
all the usable clumps in the undetected regions of the dataset, the
given quantile of their signal-to-noise ratios is used to define
the signal-to-noise ratio of a "true" clump. Effectively, this can
be seen as an inverse p-value measure. See Figure 9 and Section
3.2.1 of Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664) for a complete explanation. The
full distribution of clump signal-to-noise ratios over the
undetected areas can be saved into a table with ‘--checksn’ option
and visually inspected with ‘--checksegmentation’.
‘-v’
‘--keepmaxnearriver’
Keep a clump whose maximum (minimum if ‘--minima’ is called) flux
is 8-connected to a river pixel. By default such clumps over
detections are considered to be noise and are removed irrespective
of their significance measure; see Akhlaghi 2019
(https://arxiv.org/abs/1909.11230). Over large profiles, that sink
into the noise very slowly, noise can cause part of the profile
(which was flat without noise) to become a very large and with a
very high Signal to noise ratio. In such cases, the pixel with the
maximum flux in the clump will be immediately touching a river
pixel.
‘-s FLT’
‘--clumpsnthresh=FLT’
The signal-to-noise threshold for true clumps. If this option is
given, then the segmentation options above will be ignored and the
given value will be directly used to identify true clumps over the
detections. This can be useful if you have a large dataset with
similar noise properties. You can find a robust signal-to-noise
ratio based on a (sufficiently large) smaller portion of the
dataset. Afterwards, with this option, you can speed up the
processing on the whole dataset. Other scenarios where this option
may be useful is when, the image might not contain enough/any Sky
regions.
‘-G FLT’
‘--gthresh=FLT’
Threshold (multiple of the sky standard deviation added with the
sky) to stop growing true clumps. Once true clumps are found, they
are set as the basis to segment the detected region. They are
grown until the threshold specified by this option.
‘-y INT’
‘--minriverlength=INT’
The minimum length of a river between two grown clumps for it to be
considered in signal-to-noise ratio estimations. Similar to
‘--snminarea’, if the length of the river is too short, the
signal-to-noise ratio can be noisy and unreliable. Any existing
rivers shorter than this length will be considered as non-existent,
independent of their Signal to noise ratio. The clumps are grown
on the input image, therefore this value can be smaller than the
value given to ‘--snminarea’. Recall that the clumps were defined
on the convolved image so ‘--snminarea’ should be larger.
‘-O FLT’
‘--objbordersn=FLT’
The maximum Signal to noise ratio of the rivers between two grown
clumps in order to consider them as separate 'objects'. If the
Signal to noise ratio of the river between two grown clumps is
larger than this value, they are defined to be part of one
'object'. Note that the physical reality of these 'objects' can
never be established with one image, or even multiple images from
one broad-band filter. Any method we devise to define 'object's
over a detected region is ultimately subjective.
Two very distant galaxies or satellites in one halo might lie in
the same line of sight and be detected as clumps on one detection.
On the other hand, the connection (through a spiral arm or tidal
tail for example) between two parts of one galaxy might have such a
low surface brightness that they are broken up into multiple
detections or objects. In fact if you have noticed, exactly for
this purpose, this is the only Signal to noise ratio that the user
gives into NoiseChisel. The 'true' detections and clumps can be
objectively identified from the noise characteristics of the image,
so you do not have to give any hand input Signal to noise ratio.
‘--checksegmentation’
A file with the suffix ‘_seg.fits’ will be created. This file
keeps all the relevant steps in finding true clumps and segmenting
the detections into multiple objects in various extensions. Having
read the paper or the steps above. Examining this file can be an
excellent guide in choosing the best set of parameters. Note that
calling this function will significantly slow NoiseChisel. In
verbose mode (without the ‘--quiet’ option, see *note Operating
mode options::) the important steps (along with their extension
names) will also be reported.
With this option, NoiseChisel will abort as soon as the two tables
are created. This behavior can be disabled with
‘--continueaftercheck’.
7.3.1.3 Segment output
......................
The main output of Segment are two label datasets (with integer types,
separating the dataset's elements into different classes). They have
HDU/extension names of ‘CLUMPS’ and ‘OBJECTS’.
Similar to all Gnuastro's FITS outputs, the zero-th extension/HDU of
the main output file only contains header keywords and image or table.
It contains the Segment input files and parameters (option names and
values) as FITS keywords. Note that if an option name is longer than 8
characters, the keyword name is the second word. The first word is
‘HIERARCH’. Also note that according to the FITS standard, the keyword
names must be in capital letters, therefore, if you want to use Grep to
inspect these keywords, use the ‘-i’ option, like the example below.
$ astfits image_segmented.fits -h0 | grep -i snquant
By default, besides the ‘CLUMPS’ and ‘OBJECTS’ extensions, Segment's
output will also contain the (technically redundant) sky-subtracted
input dataset (‘INPUT-NO-SKY’) and the sky standard deviation dataset
(‘SKY_STD’, if it was not a constant number). This can help in visually
inspecting the result when viewing the images as a "Multi-extension data
cube" in SAO DS9 for example, (see *note Viewing FITS file contents with
DS9 or TOPCAT::). You can simply flip through the extensions and see
the same region of the image and its corresponding clumps/object labels.
It also makes it easy to feed the output (as one file) into MakeCatalog
when you intend to make a catalog afterwards (see *note MakeCatalog::.
To remove these redundant extensions from the output (for example, when
designing a pipeline), you can use ‘--rawoutput’.
The ‘OBJECTS’ and ‘CLUMPS’ extensions can be used as input into *note
MakeCatalog:: to generate a catalog for higher-level analysis. If you
want to treat each clump separately, you can give a very large value (or
even a NaN, which will always fail) to the ‘--gthresh’ option (for
example, ‘--gthresh=1e10’ or ‘--gthresh=nan’), see *note Segmentation
options::.
For a complete definition of clumps and objects, please see Section
3.2 of Akhlaghi and Ichikawa 2015 (https://arxiv.org/abs/1505.01664) and
*note Segmentation options::. The clumps are "true" local maxima
(minima if ‘--minima’ is called) and their surrounding pixels until a
local minimum/maximum (caused by noise fluctuations, or another "true"
clump). Therefore it may happen that some of the input detections are
not covered by clumps at all (very diffuse objects without any strong
peak), while some objects may contain many clumps. Even in those that
have clumps, there will be regions that are too diffuse. The diffuse
regions (within the input detected regions) are given a negative label
(-1) to help you separate them from the undetected regions (with a value
of zero).
Each clump is labeled with respect to its host object. Therefore, if
an object has three clumps for example, the clumps within it have labels
1, 2 and 3. As a result, if an initial detected region has multiple
objects, each with a single clump, all the clumps will have a label of
1. The total number of clumps in the dataset is stored in the ‘NCLUMPS’
keyword of the ‘CLUMPS’ extension and printed in the verbose output of
Segment (when ‘--quiet’ is not called).
The ‘OBJECTS’ extension of the output will give a positive
counter/label to every detected pixel in the input. As described in
Akhlaghi and Ichikawa [2015], the true clumps are grown until a certain
threshold. If the grown clumps touch other clumps and the connection is
strong enough, they are considered part of the same _object_. Once
objects (grown clumps) are identified, they are grown to cover the whole
detected area.
The options to configure the output of Segment are listed below:
‘--continueaftercheck’
Do not abort Segment after producing the check image(s). The usage
of this option is identical to NoiseChisel's ‘--continueaftercheck’
option (*note NoiseChisel input::). Please see the descriptions
there for more.
‘--noobjects’
Abort Segment after finding true clumps and do not continue with
finding options. Therefore, no ‘OBJECTS’ extension will be present
in the output. Each true clump in ‘CLUMPS’ will get a unique
label, but diffuse regions will still have a negative value.
To make a catalog of the clumps, the input detection map (where all
the labels are one) can be fed into *note MakeCatalog:: along with
the input detection map to Segment (that only had a value of ‘1’
for all detected pixels) with ‘--clumpscat’. In this way,
MakeCatalog will assume all the clumps belong to a single "object".
‘--grownclumps’
In the output ‘CLUMPS’ extension, store the grown clumps. If a
detected region contains no clumps or only one clump, then it will
be fully given a label of ‘1’ (no negative valued pixels).
‘--rawoutput’
Only write the ‘CLUMPS’ and ‘OBJECTS’ datasets in the output file.
Without this option (by default), the first and last extensions of
the output will the Sky-subtracted input dataset and the Sky
standard deviation dataset (if it was not a number). When the
datasets are small, these redundant extensions can make it
convenient to inspect the results visually or feed the output to
*note MakeCatalog:: for measurements. Ultimately both the input
and Sky standard deviation datasets are redundant (you had them
before running Segment). When the inputs are large/numerous, these
extra dataset can be a burden.
*Save space:* with the ‘--rawoutput’, Segment's output will only be two
labeled datasets (only containing integers). Since they have no noise,
such datasets can be compressed very effectively (without any loss of
data) with exceptionally high compression ratios. You can use the
following command to compress it with the best ratio:
$ gzip --best segment_output.fits
The resulting ‘.fits.gz’ file can then be fed into any of Gnuastro's
programs directly, without having to decompress it separately (it will
just take them a little longer, because they have to decompress it
internally before use).
When the input is a 2D image, to inspect NoiseChisel's output you can
configure SAO DS9 in your Graphic User Interface (GUI) to open
NoiseChisel's output as a multi-extension data cube. This will allow
you to flip through the different extensions and visually inspect the
results. This process has been described for the GNOME GUI (most common
GUI in GNU/Linux operating systems) in *note Viewing FITS file contents
with DS9 or TOPCAT::.
7.4 MakeCatalog
===============
At the lowest level, a dataset (for example, an image) is just a
collection of values, placed after each other in any number of
dimensions (for example, an image is a 2D dataset). Each data-element
(pixel) just has two properties: its position (relative to the rest) and
its value. In higher-level analysis, an entire dataset (an image for
example) is rarely treated as a singular entity(1). You usually want to
know/measure the properties of the (separate) scientifically interesting
targets that are embedded in it. For example, the magnitudes, positions
and elliptical properties of the galaxies that are in the image.
MakeCatalog is Gnuastro's program for localized measurements over a
dataset. In other words, MakeCatalog is Gnuastro's program to convert
low-level datasets (like images), to high level catalogs. The role of
MakeCatalog in a scientific analysis and the benefits of its model
(where detection/segmentation is separated from measurement) is
discussed in Akhlaghi 2016 (https://arxiv.org/abs/1611.06387v1)(2) and
summarized in *note Detection and catalog production::. We strongly
recommend reading this short paper for a better understanding of this
methodology. Understanding the effective usage of MakeCatalog, will
thus also help effective use of other (lower-level) Gnuastro's programs
like *note NoiseChisel:: or *note Segment::.
It is important to define your regions of interest for measurements
_before_ running MakeCatalog. MakeCatalog is specialized in doing
measurements accurately and efficiently. Therefore MakeCatalog will not
do detection, segmentation, or defining apertures on requested positions
in your dataset. Following Gnuastro's modularity principle, there are
separate and highly specialized and customizable programs in Gnuastro
for these other jobs as shown below (for a usage example in a real-world
analysis, see *note General program usage tutorial:: and *note Detecting
large extended targets::).
• *note Arithmetic::: Detection with a simple threshold.
• *note NoiseChisel::: Advanced detection.
• *note Segment::: Segmentation (substructure over detections).
• *note MakeProfiles::: Aperture creation for known positions.
These programs will/can return labeled dataset(s) to be fed into
MakeCatalog. A labeled dataset for measurement has the same
size/dimensions as the input, but with integer valued pixels that have
the label/counter for each sub-set of pixels that must be measured
together. For example, all the pixels covering one galaxy in an image,
get the same label.
The requested measurements are then done on similarly labeled pixels.
The final result is a catalog where each row corresponds to the
measurements on pixels with a specific label. For example, the flux
weighted average position of all the pixels with a label of 42 will be
written into the 42nd row of the output catalog/table's central position
column(3). Similarly, the sum of all these pixels will be the 42nd row
in the sum column, etc. Pixels with labels equal to, or smaller than,
zero will be ignored by MakeCatalog. In other words, the number of rows
in MakeCatalog's output is already known before running it (the maximum
value of the labeled dataset).
Before getting into the details of running MakeCatalog (in *note
Invoking astmkcatalog::, we will start with a discussion on the basics
of its approach to separating detection from measurements in *note
Detection and catalog production::. A very important factor in any
measurement is understanding its validity range, or limits. Therefore
in *note Quantifying measurement limits::, we will discuss how to
estimate the reliability of the detection and basic measurements. This
section will continue with a derivation of elliptical parameters from
the labeled datasets in *note Measuring elliptical parameters::. For
those who feel MakeCatalog's existing measurements/columns are not
enough and would like to add further measurements, in *note Adding new
columns to MakeCatalog::, a checklist of steps is provided for readily
adding your own new measurements/columns.
---------- Footnotes ----------
(1) You can derive the over-all properties of a complete dataset (1D
table column, 2D image, or 3D data-cube) treated as a single entity with
Gnuastro's Statistics program (see *note Statistics::).
(2) A published paper cannot undergo any more change, so this manual
is the definitive guide.
(3) See *note Measuring elliptical parameters:: for a discussion on
this and the derivation of positional parameters, which includes the
center.
7.4.1 Detection and catalog production
--------------------------------------
Most existing common tools in low-level astronomical data-analysis (for
example, SExtractor(1)) merge the two processes of detection and
measurement (catalog production) in one program. However, in light of
Gnuastro's modularized approach (modeled on the Unix system) detection
is separated from measurements and catalog production. This modularity
is therefore new to many experienced astronomers and deserves a short
review here. Further discussion on the benefits of this methodology can
be seen in Akhlaghi 2016 (https://arxiv.org/abs/1611.06387v1).
As discussed in the introduction of *note MakeCatalog::, detection
(identifying which pixels to do measurements on) can be done with
different programs. Their outputs (a labeled dataset) can be directly
fed into MakeCatalog to do the measurements and write the result as a
catalog/table. Beyond that, Gnuastro's modular approach has many
benefits that will become clear as you get more experienced in
astronomical data analysis and want to be more creative in using your
valuable data for the exciting scientific project you are working on.
In short the reasons for this modularity can be classified as below:
• Simplicity/robustness of independent, modular tools: making a
catalog is a logically separate process from labeling (detection,
segmentation, or aperture production). A user might want to do
certain operations on the labeled regions before creating a catalog
for them. Another user might want the properties of the same
pixels/objects in another image (another filter for example) to
measure the colors or SED fittings.
Here is an example of doing both: suppose you have images in
various broad band filters at various resolutions and orientations.
The image of one color will thus not lie exactly on another or even
be in the same scale. However, it is imperative that the same
pixels be used in measuring the colors of galaxies.
To solve the problem, NoiseChisel can be run on the reference image
to generate the labeled detection image. Afterwards, the labeled
image can be warped into the grid of the other color (using *note
Warp::). MakeCatalog will then generate the same catalog for both
colors (with the different labeled images). It is currently
customary to warp the images to the same pixel grid, however,
modification of the scientific dataset is very harmful for the data
and creates correlated noise. It is much more accurate to do the
transformations on the labeled image.
• Complexity of a monolith: Adding in a catalog functionality to the
detector program will add several more steps (and many more
options) to its processing that can equally well be done outside of
it. This makes following what the program does harder for the
users and developers, it can also potentially add many bugs.
As an example, if the parameter you want to measure over one
profile is not provided by the developers of MakeCatalog. You can
simply open this tiny little program and add your desired
calculation easily. This process is discussed in *note Adding new
columns to MakeCatalog::. However, if making a catalog was part of
NoiseChisel for example, adding a new column/measurement would
require a lot of energy to understand all the steps and internal
structures of that huge program. It might even be so intertwined
with its processing, that adding new columns might cause
problems/bugs in its primary job (detection).
---------- Footnotes ----------
(1)
7.4.2 Brightness, Flux, Magnitude and Surface brightness
--------------------------------------------------------
Astronomical data pixels are usually in units of counts(1) or electrons
or either one divided by seconds. To convert from the counts to
electrons, you will need to know the instrument gain. In any case, they
can be directly converted to energy or energy/time using the basic
hardware (telescope, camera and filter) information (that is summarized
in the _zero point_, and we will discuss below). We will continue the
discussion assuming the pixels are in units of energy/time.
Brightness
The _brightness_ of an object is defined as its measured energy in
units of time. If our detector pixels directly measured the energy
from the astronomical object(2), then the brightness would be the
total sum of pixel values (energy) associated to the object,
divided by the exposure time. The _flux_ of an object is defined
in units of energy/time/collecting-area. For an astronomical
target, the flux is therefore defined as its brightness divided by
the area used to collect the light from the source; or the
telescope aperture (for example, in units of $cm^2$). Knowing the
flux ($f$) and distance to the object ($r$), we can define its
_luminosity_: $L=4{\pi}r^2f$.
Therefore, while flux and luminosity are intrinsic properties of
the object, brightness depends on our detecting tools (hardware and
software). In low-level observational astronomy data analysis, we
are usually more concerned with measuring the brightness, because
it is the thing we directly measure from the image pixels and
create in catalogs. On the other hand, luminosity is used in
higher-level analysis (after image contents are measured as
catalogs to deduce physical interpretations, because high-level
things like distance/redshift need to be calculated). At this
stage, it is just important avoid confusion between luminosity and
brightness because both have the same units of energy per seconds.
Magnitude
Images of astronomical objects span over a very large range of
brightness: the Sun (as the brightest object) is roughly
$2.5^{60}=10^{24}$ times brighter than the fainter galaxies we can
currently detect in the deepest images. Therefore discussing
brightness directly will involve a large range of values which is
inconvenient. So astronomers have chosen to use a logarithmic
scale for the brightness of astronomical objects.
But the logarithm can only be usable with a dimensionless value
that is always positive. Fortunately brightness is always positive
(at least in theory(3)). To remove the dimensions, we divide the
brightness of the object ($B$) by a reference brightness ($B_r$).
We then define a logarithmic scale as $magnitude$ through the
relation below. The $-2.5$ factor in the definition of magnitudes
is a legacy of the our ancient colleagues and in particular
Hipparchus of Nicaea (190-120 BC).
$$m-m_r=-2.5\log_{10} \left( B \over B_r \right)$$
$m$ is defined as the magnitude of the object and $m_r$ is the
pre-defined magnitude of the reference brightness. For estimating
the error in measuring a magnitude, see *note Quantifying
measurement limits::.
Zero point
A unique situation in the magnitude equation above occurs when the
reference brightness is unity ($B_r=1$). This brightness will thus
summarize all the hardware-specific parameters discussed above
(like the conversion of pixel values to physical units) into one
number. That reference magnitude is commonly known as the _Zero
point_ magnitude because when $B=B_r=1$, the right side of the
magnitude definition above will be zero. Using the zero point
magnitude ($Z$), we can write the magnitude relation above in a
more simpler format:
$$m = -2.5\log_{10}(B) + Z$$
Gnuastro has an installed script to estimate the zero point of any
image, see *note Zero point estimation:: (it contains practical
tutorials to help you get started fast). Having the zero point of
an image, you can convert its pixel values to physical units like
microJanskys (or $\mu{}Jy$). This enables direct pixel-based
comparisons with images from other instruments(4). Jansky is a
commonly used unit for measuring spectral flux density and one
Jansky is equivalent to $10^{-26} W/m^2/Hz$ (watts per square meter
per hertz).
This conversion can be done with the fact that in the AB magnitude
standard(5), $3631Jy$ corresponds to the zero-th magnitude,
therefore $B\equiv3631\times10^{6}\mu{Jy}$ and $m\equiv0$. We can
therefore estimate the brightness ($B_z$, in $\mu{Jy}$)
corresponding to the image zero point ($Z$) using this equation:
$$m - Z = -2.5\log_{10}(B/B_z)$$
$$0 - Z = -2.5\log_{10}({3631\times10^{6}\over B_z})$$
$$B_z = 3631\times10^{\left(6 - {Z \over 2.5} \right)} \mu{Jy}$$
Because the image zero point corresponds to a pixel value of $1$,
the $B_z$ value calculated above also corresponds to a pixel value
of $1$. Therefore you simply have to multiply your image by $B_z$
to convert it to $\mu{Jy}$. Do not forget that this only applies
when your zero point was also estimated in the AB magnitude system.
On the command-line, you can estimate this value for a certain zero
point with AWK, then multiply it to all the pixels in the image
with *note Arithmetic::. For example, let's assume you are using
an SDSS image with a zero point of 22.5:
bz=$(echo 22.5 | awk '{print 3631 * 10^(6-$1/2.5)}')
astarithmetic sdss.fits $bz x --output=sdss-in-muJy.fits
But in Gnuastro, it gets even easier: Arithmetic has an operator
called ‘counts-to-jy’. This will directly convert your image
pixels (in units of counts) to Janskys though a provided AB
Magnitude-based zero point like below. See *note Arithmetic
operators:: for more.
$ astarithmetic sdss.fits 22.5 counts-to-jy
*Be careful with the exposure time:* as described at the start of
this section, we are assuming your data are in units of counts/sec.
As a result, the counts you get from the command above, are only
for one second of exposure! Please see the discussion below in
"Magnitude to counts" for more.
Magnitude to counts (accounting for exposure time)
Until now, we had assumed that the data are in units of counts/sec.
As a result, the equations given above (in the "Zero point" item to
convert magnitudes to pixel counts), give the count level for the
reference (1 second) exposure. But we rarely take 1 second
exposures! It is therefore very important to take the exposure
time into account in scenarios like simulating observations with
varying exposure times (where you need to know how many counts the
object of a certain magnitude will add to a certain image with a
certain exposure time).
To clarify the concept, let's define $C$ as the _counted_ electrons
(which has a linear relation with the photon energy entering the
CCD pixel). In this case, if an object of brightness $B$ is
observed for $t$ seconds, it will accumulate $C=B\times t$
counts(6). Therefore, the generic magnitude equation above can be
written as:
$$m = -2.5\log_{10}(B) + Z = -2.5\log_{10}(C/t) + Z$$
From this, we can derive $C(t)$ in relation to $C(1)$, or counts
from a 1 second exposure, using this relation:
$$C(t) = t\times10^{(m-Z)/2.5} = t\times C(1)$$
In other words, you should simply multiply the counts for one
second with the number of observed seconds.
Another approach is to shift the time-dependence of the counts into
the zero point (after all exposure time is also a hardware issue).
Let's derive the equation below:
$$m = -2.5\log_{10}(C/t) + Z = -2.5\log_{10}(C) + 2.5\log_{10}(t) + Z$$
Therefore, defining an exposure-time-dependent zero point as
$Z(t)$, we can directly correlate a certain object's magnitude with
counts after an exposure of $t$ seconds:
$$m = -2.5\log_{10}(C) + Z(t) \quad\rm{where}\quad Z(t)=Z + 2.5\log_{10}(t)$$
This solution is useful in programs like *note MakeCatalog:: or
*note MakeProfiles::, when you cannot (or do not want to: because
of the extra storage/speed costs) manipulate the values image (for
example, divide it by the exposure time to use a counts/sec zero
point).
Surface brightness
Another important concept is the distribution of an object's
brightness over its area. For this, we define the _surface
brightness_ to be the magnitude of an object's brightness divided
by its solid angle over the celestial sphere (or coverage in the
sky, commonly in units of arcsec$^2$). The solid angle is
expressed in units of arcsec$^2$ because astronomical targets are
usually much smaller than one steradian. Recall that the steradian
is the dimension-less SI unit of a solid angle and 1 steradian
covers $1/4\pi$ (almost $8\%$) of the full celestial sphere.
Surface brightness is therefore most commonly expressed in units of
mag/arcsec$^2$. For example, when the brightness is measured over
an area of A arcsec$^2$, then the surface brightness becomes:
$$S = -2.5\log_{10}(B/A) + Z = -2.5\log_{10}(B) + 2.5\log_{10}(A) + Z$$
In other words, the surface brightness (in units of mag/arcsec$^2$)
is related to the object's magnitude ($m$) and area ($A$, in units
of arcsec$^2$) through this equation:
$$S = m + 2.5\log_{10}(A)$$
A common mistake is to follow the mag/arcsec$^2$ unit literally,
and divide the object's magnitude by its area. But this is wrong
because magnitude is a logarithmic scale while area is linear. It
is the brightness that should be divided by the solid angle because
both have linear scales. The magnitude of that ratio is then
defined to be the surface brightness.
One usual application of this is to convert an image's pixel values
to surface brightness, when you know its zero point. This can be
done with the two simple commands below. First, we derive the
pixel area (in arcsec$^2$) then we use Arithmetic to convert the
pixels into surface brightness, see below for the details.
$ zeropoint=22.5
$ pixarea=$(astfits image.fits --pixelareaarcsec2)
$ astarithmetic image.fits $zeropoint $pixarea counts-to-sb \
--output=image-sb.fits
See *note Reverse polish notation:: for more on Arithmetic's
notation and *note Arithmetic operators:: for a description of each
operator. And see *note FITS images in a publication:: for a fully
working tutorial on how to optimally convert a FITS image to a PDF
image for usage in a publication using the surface brightness
conversion shown above.
*Do not warp or convolve magnitude or surface brightness images:*
Warping an image involves calculating new pixel values (of the new
pixel grid) from the old pixel values. Convolution is also a
process of finding the weighted mean of pixel values. During these
processes, many arithmetic operations are done on the original
pixel values, for example, addition or multiplication. However,
$log_{10}(a+b)\ne log_{10}(a)+log_{10}(b)$. Therefore after
calculating a magnitude or surface brightness image, do not apply
any such operations on it! If you need to warp or convolve the
image, do it _before_ the conversion.
---------- Footnotes ----------
(1) Counts are also known as analog to digital units (ADU).
(2) In practice, the measured pixels don't just count the
astronomical object's energy: imaging detectors insert a certain bias
level before the exposure, they amplify the photo-electrons, there are
optical artifacts like flat-fielding, and finally, there is the
background light.
(3) In practice, for very faint objects, if the background brightness
is over-subtracted, we may end up with a negative "brightness" or sum of
pixels in a real object.
(4) Comparing data from different instruments assumes instrument and
observation signatures are properly corrected, things like the
flat-field or the Sky absorption. It is also valid for pixel values,
assuming that factors that can change the morphology (like the *note
PSF::) are the same.
(5)
(6) Recall that counts another name for ADUs, which already includes
the CCD gain.
7.4.3 Quantifying measurement limits
------------------------------------
No measurement on a real dataset can be perfect: you can only reach a
certain level/limit of accuracy and a meaningful (scientific) analysis
requires an understanding of these limits. Different datasets have
different noise properties and different detection methods (one
method/algorithm/software that is run with a different set of parameters
is considered as a different detection method) will have different
abilities to detect or measure certain kinds of signal (astronomical
objects) and their properties in the dataset. Hence, quantifying the
detection and measurement limitations with a particular dataset and
analysis tool is the most crucial/critical aspect of any high-level
analysis. In two separate tutorials, we have touched upon some of these
points. So to see the discussions below in action (on real data), see
*note Measuring the dataset limits:: and *note Image surface brightness
limit::.
Here, we will review some of the most commonly used methods to
quantify the limits in astronomical data analysis and how MakeCatalog
makes it easy to measure them. Depending on the higher-level analysis,
there are more tests that must be done, but these are relatively
low-level and usually necessary in most cases. In astronomy, it is
common to use the magnitude (a unit-less scale) and physical units, see
*note Brightness flux magnitude::. Therefore the measurements discussed
here are commonly used in units of magnitudes.
7.4.3.1 Standard deviation vs error
...................................
The error and the standard deviation are sometimes confused with each
other. Therefore, before continuing with the various measurement limits
below, let's review these two fundamental concepts. Instead of going
into the theoretical definitions of the two (which you can see in their
respective Wikipedia pages), we'll discuss the concepts in a hands-on
and practical way here.
Let's simulate an observation of the sky, but without any
astronomical sources! In other words, we only have a background flux
level (from the sky emission). With the first command below, let's make
an image called ‘1.fits’ that contains $200\times200$ pixels that are
filled with random noise from a Poisson distribution with a mean of 100
counts (the flux from the background sky). With the second command,
we'll have a look at the image. Recall that the Poisson distribution is
equal to a normal distribution for large mean values (as in this case).
$ astarithmetic 200 200 2 makenew 100 mknoise-poisson \
--output=1.fits
$ astscript-fits-view 1.fits
The standard deviation ($\sigma$) of the Poisson distribution is the
square root of the mean, see *note Photon counting noise::. Note that
due to the random nature of the noise, the values reported in the next
steps on your computer will be very slightly different. To reproducible
exactly the same values in different runs, see *note Generating random
numbers::, and for more on the first command, see *note Arithmetic::.
Each pixel shows the result of one sampling from the Poisson
distribution. In other words, assuming the sky emission in our
simulation is constant over our field of view, each pixel's value shows
one measurement of the sky emission. Statistically speaking, a
"measurement" is a sampling from an underlying distribution of values.
Through our measurements, we aim to identify that underlying
distribution (the "truth")! With the command below, let's look at the
pixel statistics of ‘1.fits’ (output is shown immediately under it).
$ aststatistics 1.fits
Statistics (GNU Astronomy Utilities) 0.23
-------
Input: 1.fits (hdu: 1)
-------
Number of elements: 40000
Minimum: 61
Maximum: 155
Median: 100
Mean: 100.044925
Standard deviation: 10.00066032
-------
Histogram:
| * *
| * * *
| * * * *
| * * * * *
| * * * * *
| * * ******** * *
| * ************* *
| * ****************** *
| ************************ *
| *********************************
|* ********************************************************** ** *
|----------------------------------------------------------------------
As expected, you see that the ASCII histogram nicely resembles a
normal distribution. The measured mean and standard deviation
($\sigma_x$) are also very similar to the input (mean of 100, standard
deviation of $\sigma=10$). But the measured mean (and standard
deviation) aren't exactly equal to the input!
Every time we make a different simulated image from the same
distribution, the measured mean and standard deviation will slightly
differ. With the second command below, let's build 500 images like
above and measure their mean and standard deviation. The outputs will
be written into a file (‘mean-stds.txt’; in the first command we are
deleting it to make sure we write into an empty file within the loop).
With the third command, let's view the top 10 rows:
$ rm -f mean-stds.txt
$ for i in $(seq 500); do \
astarithmetic 200 200 2 makenew 100 mknoise-poisson \
--output=$i.fits --quiet; \
aststatistics $i.fits --mean --std >> mean-stds.txt; \
echo "$i: complete"; \
done
$ asttable mean-stds.txt -Y --head=10
99.989381 9.936407
100.036622 10.059997
100.006054 9.985470
99.944535 9.960069
100.050318 9.970116
100.002718 9.905395
100.067555 9.964038
100.027167 10.018562
100.051951 9.995859
100.000212 9.970293
From this table, you see that each simulation has produced a slightly
different measured mean and measured standard deviation ($\sigma_x$)
that are just fluctuating around the input mean (which was 100) and
input standard deviation ($\sigma=10$). Let's have a look at the
distribution of mean measurements:
$ aststatistics mean-stds.txt -c1
Statistics (GNU Astronomy Utilities) 0.23
-------
Input: mean-stds.txt
Column: 1
-------
Number of elements: 500
Minimum: 9.98183528700191e+01
Maximum: 1.00146490891332e+02
Mode: 99.99709739
Mode quantile: 0.49498998
Median: 9.99977393190436e+01
Mean: 99.99891826
Standard deviation: 0.04901635275
-------
Histogram:
| *
| * **
| ****** **** * *
| ****** **** * * *
| * * ************* * *
| * ****************** **
| * ********************* *** *
| * ***************************** ***
| *** ********************************** *
| *** ******************************************* **
| * ************************************************* ** *
|----------------------------------------------------------------------
The standard deviation of the various mean measurements above shows
the scatter in measuring the mean with an image of this size from this
underlying distribution. This is therefore defined as the _standard
error of the mean_, or "error" for short (since most measurements are
actually the mean of a population) and shown with
$\widehat\sigma_{\bar{x}}$.
From the example above, you see that the error is smaller than the
standard deviation (smaller when you have a larger sample). In fact, it
can be shown (https://en.wikipedia.org/wiki/Standard_error#Derivation)
that this "error of the mean" ($\sigma_{\bar{x}}$) is related to the
distribution standard deviation ($\sigma$) through the following
equation. Where $N$ is the number of points used to measure the mean in
one sample ($200\times200=40000$ in this case). Note that the $10.0$
below was reported as "standard deviation" in the first run of
‘aststatistics’ on ‘1.fits’ above):
$$\sigma_{\bar{x}}=\frac{\sigma}{\sqrt{N}} \quad\quad {\rm or} \quad\quad \widehat\sigma_{\bar{x}}\approx\frac{\sigma_x}{\sqrt{N}} = \frac{10.0}{200} = 0.05$$
Taking the considerations above into account, we should clearly
distinguish the following concepts when talking about the standard
deviation or error:
Standard deviation of population
This is the standard deviation of the underlying distribution (10
in the example above), and shown by $\sigma$. This is something
you can never measure, and is just the ideal value.
Standard deviation of mean
Ideal error of measuring the mean (assuming we know $\sigma$).
Standard deviation of sample (i.e., _Standard deviation_)
Measured Standard deviation from a sampling of the ideal
distribution. This is the second column of ‘mean-stds.txt’ above
and is shown with $\sigma_x$ above. In astronomical literature,
this is simply referred to as the "standard deviation".
In other words, the standard deviation is computed on the input
itself and MakeCatalog just needs a "values" file. For example,
when measuring the standard deviation of an astronomical object
using MakeCatalog it is computed directly from the input values.
Standard error (i.e., _error_)
Measurable scatter of measuring the mean
($\widehat\sigma_{\bar{x}}$) that can be estimated from the size of
the sample and the measured standard deviation ($\sigma_x$). In
astronomical literature, this is simply referred to as the "error".
In other words, when asking for an "error" measurement with
MakeCatalog, a separate standard deviation dataset should be always
provided. This dataset should take into account all sources of
scatter. For example, during the reduction of an image, the
standard deviation dataset should take into account the dispersion
of each pixel that comes from the bias, dark, flat fielding, etc.
If this image is not available, it is possible to use the ‘SKY_STD’
extension from NoiseChisel as an estimation. For more see *note
NoiseChisel output::.
7.4.3.2 Magnitude measurement error of each detection
.....................................................
The raw error in measuring the magnitude is only meaningful when the
object's magnitude is brighter than the upper-limit magnitude (see
below). As discussed in *note Brightness flux magnitude::, the
magnitude ($M$) of an object with brightness $B$ and zero point
magnitude $z$ can be written as:
$$M=-2.5\log_{10}(B)+z$$
Calculating the derivative with respect to $B$, we get:
$${dM\over dB} = {-2.5\over {B\times ln(10)}}$$
From the Tailor series ($\Delta{M}=dM/dB\times\Delta{B}$), we can write:
$$\Delta{M} = \left|{-2.5\over ln(10)}\right|\times{\Delta{B}\over{B}}$$
But, $\Delta{B}/B$ is just the inverse of the Signal-to-noise ratio
($S/N$), so we can write the error in magnitude in terms of the
signal-to-noise ratio:
$$\Delta{M} = {2.5\over{S/N\times ln(10)}} $$
MakeCatalog uses this relation to estimate the magnitude errors. The
signal-to-noise ratio is calculated in different ways for clumps and
objects, see Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664)), but this single equation can be
used to estimate the measured magnitude error afterwards for any type of
target.
7.4.3.3 Surface brightness error of each detection
..................................................
We can derive the error in measuring the surface brightness based on the
surface brightness (SB) equation of *note Brightness flux magnitude::
and the generic magnitude error ($\Delta{M}$) of *note Magnitude
measurement error of each detection::. Let's set $A$ to represent the
area and $\Delta{A}$ to represent the error in measuring the area. For
more on $\Delta{A}$, see the description of ‘--spatialresolution’ in
*note MakeCatalog inputs and basic settings::.
$$\Delta{(SB)} = \Delta{M} + \left|{-2.5\over ln(10)}\right|\times{\Delta{A}\over{A}}$$
In the surface brightness equation mentioned above, $A$ is in units
of arcsecond squared and the conversion between arcseconds to pixels is
a multiplication factor. Therefore as long as $A$ and $\Delta{A}$ have
the same units, it does not matter if they are in arcseconds or pixels.
Since the measure of spatial resolution (or area error) is the FWHM of
the PSF which is usually defined in terms of pixels, its more intuitive
to use pixels for $A$ and $\Delta{A}$.
7.4.3.4 Completeness limit of each detection
............................................
As the surface brightness of the objects decreases, the ability to
detect them will also decrease. An important statistic is thus the
fraction of objects of similar morphology and magnitude that will be
detected with our detection algorithm/parameters in a given image. This
fraction is known as _completeness_. For brighter objects, completeness
is 1: all bright objects that might exist over the image will be
detected. However, as we go to objects of lower overall surface
brightness, we will fail to detect a fraction of them, and fainter than
a certain surface brightness level (for each morphology), nothing will
be detectable in the image: you will need more data to construct a
"deeper" image. For a given profile and dataset, the magnitude where
the completeness drops below a certain level (usually above $90\%$) is
known as the completeness limit.
Another important parameter in measuring completeness is purity: the
fraction of true detections to all detections. In effect purity is the
measure of contamination by false detections: the higher the purity, the
lower the contamination. Completeness and purity are anti-correlated:
if we can allow a large number of false detections (that we might be
able to remove by other means), we can significantly increase the
completeness limit.
One traditional way to measure the completeness and purity of a given
sample is by embedding mock profiles in regions of the image with no
detection. However in such a study we must be really careful to choose
model profiles as similar to the target of interest as possible.
7.4.3.5 Upper limit magnitude of each detection
...............................................
Due to the noisy nature of data, it is possible to get arbitrarily faint
magnitudes, especially when you use labels from another image (for
example see *note Working with catalogs estimating colors::). Given the
scatter caused by the dataset's noise, values fainter than a certain
level are meaningless: another similar depth observation will give a
radically different value. In such cases, measurements like the image
magnitude limit are not useful because it is estimated for a certain
morphology and is given for the whole image (it is a crude
generalization; see see *note Magnitude limit of image::). You want a
quality measure that is specific to each object.
For example, assume that you have done your detection and
segmentation on one filter and now you do measurements over the same
labeled regions, but on other filters to measure colors (as we did in
the tutorial *note Segmentation and making a catalog::). Some objects
are not going to have any significant signal in the other filters, but
for example, you measure magnitude of 36 for one of them! This is
clearly unreliable (no dataset in current astronomy is able to detect
such a faint signal). In another image with the same depth, using the
same filter, you might measure a magnitude of 30 for it, and yet another
might give you 33. Furthermore, the total sum of pixel values might
actually be negative in some images of the same depth (due to noise).
In these cases, no magnitude can be defined and MakeCatalog will place a
NaN there (recall that a magnitude is a base-10 logarithm).
Using such unreliable measurements will directly affect our analysis,
so we must not use the raw measurements. When approaching the limits of
your detection method, it is therefore important to be able to identify
such cases. But how can we know how reliable a measurement of one
object on a given dataset is?
When we confront such unreasonably faint magnitudes, there is one
thing we can deduce: that if something actually exists under our labeled
pixels (possibly buried deep under the noise), it's inherent magnitude
is fainter than an _upper limit magnitude_. To find this upper limit
magnitude, we place the object's footprint (segmentation map) over a
random part of the image where there are no detections, and measure the
sum of pixel values within the footprint. Doing this a large number of
times will give us a distribution of measurements of the sum. The
standard deviation ($\sigma$) of that distribution can be used to
quantify the upper limit magnitude for that particular object (given its
particular shape and area):
$$M_{up,n\sigma}=-2.5\times\log_{10}{(n\sigma_m)}+z \quad\quad [mag/target]$$
Traditionally, faint/small object photometry was done using fixed
circular apertures (for example, with a diameter of $N$ arc-seconds) and
there was not much processing involved (to make a deep stack). Hence,
the upper limit was synonymous with the surface brightness limit
discussed above: one value for the whole image. The problem with this
simplified approach is that the number of pixels in the aperture
directly affects the final distribution and thus magnitude. Also the
image correlated noise might actually create certain patterns, so the
shape of the object can also affect the final result. Fortunately, with
the much more advanced hardware and software of today, we can make
customized segmentation maps (footprint) for each object and have enough
computing power to actually place that footprint over many random
places. As a result, the per-target upper-limit magnitude and general
surface brightness limit have diverged.
When any of the upper-limit-related columns requested, MakeCatalog
will randomly place each target's footprint over the undetected parts of
the dataset as described above, and estimate the required properties.
The procedure is fully configurable with the options in *note
Upper-limit settings::. You can get the full list of upper-limit
related columns of MakeCatalog with this command (the extra ‘--’ before
‘--upperlimit’ is necessary(1)):
$ astmkcatalog --help | grep -- --upperlimit
---------- Footnotes ----------
(1) Without the extra ‘--’, grep will assume that ‘--upperlimit’ is
one of its own options, and will thus abort, complaining that it has no
option with this name.
7.4.3.6 Magnitude limit of image
................................
Suppose we have taken two images of the same field of view with the same
CCD, once with a smaller telescope, and once with a larger one. Because
we used the same CCD, the noise will be very similar. However, the
larger telescope has gathered more light, therefore the same star or
galaxy will have a higher signal-to-noise ratio (S/N) in the image taken
with the larger one. The same applies for a stacked image of the field
compared to a single-exposure image of the same telescope.
This concept is used by some researchers to define the "magnitude
limit" or "detection limit" at a certain S/N (sometimes 10, 5 or 3 for
example, also written as $10\sigma$, $5\sigma$ or $3\sigma$). To do
this, they measure the magnitude and signal-to-noise ratio of all the
objects within an image and measure the mean (or median) magnitude of
objects at the desired S/N. A fully working example of deriving the
magnitude limit is available in the tutorials section: *note Measuring
the dataset limits::.
However, this method should be used with extreme care! This is
because the shape of the object becomes important in this method: a
sharper object will have a higher _measured_ S/N compared to a more
diffuse object at the same original magnitude. Besides the inherent
shape/sharpness of the object, issues like the PSF also become important
in this method (because the finally observed shapes of objects are
important here): two surveys with the same surface brightness limit (see
*note Surface brightness limit of image::) will have different magnitude
limits if one is taken from space and the other from the ground.
7.4.3.7 Surface brightness limit of image
.........................................
As we make more observations on one region of the sky and add/combine
the observations into one dataset, both the signal and the noise
increase. However, the signal increases much faster than the noise:
Assuming you add $N$ datasets with equal exposure times, the signal will
increases as a multiple of $N$, while noise increases as $\sqrt{N}$.
Therefore the signal-to-noise ratio increases by a factor of $\sqrt{N}$.
Visually, fainter (per pixel) parts of the objects/signal in the image
will become more visible/detectable. The noise-level is known as the
dataset's surface brightness limit.
You can think of the noise as muddy water that is completely covering
a flat ground(1). The signal (coming from astronomical objects in real
data) will be summits/hills that start from the flat sky level (under
the muddy water) and their summits can sometimes reach above the muddy
water. Let's assume that in your first observation the muddy water has
just been stirred and except a few small peaks, you cannot see anything
through the mud. As you wait and make more observations/exposures, the
mud settles down and the _depth_ of the transparent water increases. As
a result, more and more summits become visible and the lower parts of
the hills (parts with lower surface brightness) can be seen more
clearly. In this analogy(2), height (from the ground) is the _surface
brightness_ and the height of the muddy water at the moment you combine
your data, is your _surface brightness limit_ for that moment.
The outputs of NoiseChisel include the Sky standard deviation
($\sigma$) on every group of pixels (a tile) that were calculated from
the undetected pixels in each tile, see *note Tessellation:: and *note
NoiseChisel output::. Let's take $\sigma_m$ as the median $\sigma$ over
the successful meshes in the image (prior to interpolation or
smoothing). It is recorded in the ‘MEDSTD’ keyword of the ‘SKY_STD’
extension of NoiseChisel's output.
On different instruments, pixels cover different spatial angles over
the sky. For example, the width of each pixel on the ACS camera on the
Hubble Space Telescope (HST) is roughly 0.05 seconds of arc, while the
pixels of SDSS are each 0.396 seconds of arc (almost eight times
wider(3)). Nevertheless, irrespective of its sky coverage, a pixel is
our unit of data collection.
To start with, we define the low-level Surface brightness limit or
_depth_, in units of magnitude/pixel with the equation below (assuming
the image has zero point magnitude $z$ and we want the $n$th multiple of
$\sigma_m$).
$$SB_{n\sigma,\rm pixel}=-2.5\times\log_{10}{(n\sigma_m)}+z \quad\quad [mag/pixel]$$
As an example, the XDF survey covers part of the sky that the HST has
observed the most (for 85 orbits) and is consequently very small
($\sim4$ minutes of arc, squared). On the other hand, the CANDELS
survey, is one of the widest multi-color surveys done by the HST
covering several fields (about 720 arcmin$^2$) but its deepest fields
have only 9 orbits observation. The $1\sigma$ depth of the XDF and
CANDELS-deep surveys in the near infrared WFC3/F160W filter are
respectively 34.40 and 32.45 magnitudes/pixel. In a single orbit image,
this same field has a $1\sigma$ depth of 31.32 magnitudes/pixel. Recall
that a larger magnitude corresponds to fainter objects, see *note
Brightness flux magnitude::.
The low-level magnitude/pixel measurement above is only useful when
all the datasets you want to use, or compare, have the same pixel size.
However, you will often find yourself using, or comparing, datasets from
various instruments with different pixel scales (projected pixel width,
in arc-seconds). If we know the pixel scale, we can obtain a more
easily comparable surface brightness limit in units of:
magnitude/arcsec$^2$. But another complication is that astronomical
objects are usually larger than 1 arcsec$^2$. As a result, it is common
to measure the surface brightness limit over a larger (but fixed,
depending on context) area.
Let's assume that every pixel is $p$ arcsec$^2$ and we want the
surface brightness limit for an object covering A arcsec$^2$ (so $A/p$
is the number of pixels that cover an area of $A$ arcsec$^2$). On the
other hand, noise is added in RMS(4), hence the noise level in $A$
arcsec$^2$ is $n\sigma_m\sqrt{A/p}$. But we want the result in units of
arcsec$^2$, so we should divide this by $A$ arcsec$^2$:
$n\sigma_m\sqrt{A/p}/A=n\sigma_m\sqrt{A/(pA^2)}=n\sigma_m/\sqrt{pA}$.
Plugging this into the magnitude equation, we get the $n\sigma$ surface
brightness limit, over an area of A arcsec$^2$, in units of
magnitudes/arcsec$^2$:
$$SB_{{n\sigma,\rm A arcsec}^2}=-2.5\times\log_{10}{\left(n\sigma_m\over \sqrt{pA}\right)+z} \quad\quad [mag/arcsec^2]$$
MakeCatalog will calculate the input dataset's $SB_{n\sigma,\rm
pixel}$ and $SB_{{n\sigma,\rm A arcsec}^2}$ and will write them as the
‘SBLMAGPIX’ and ‘SBLMAG’ keywords the output catalog(s), see *note
MakeCatalog output::. You can set your desired $n$-th multiple of
$\sigma$ and the $A$ arcsec$^2$ area using the following two options
respectively: ‘--sfmagnsigma’ and ‘--sfmagarea’ (see *note MakeCatalog
output::). Just note that $SB_{{n\sigma,\rm A arcsec}^2}$ is only
calculated if the input has World Coordinate System (WCS). Without WCS,
the pixel scale cannot be derived.
As you saw in its derivation, the calculation above extrapolates the
noise in one pixel over all the input's pixels! In other words, all
pixels are treated independently in the measurement of the standard
deviation. It therefore implicitly assumes that the noise is the same
in all of the pixels. But this only happens in individual exposures:
reduced data will have correlated noise because they are a stack of many
individual exposures that have been warped (thus mixing the pixel
values). A more accurate measure which will provide a realistic value
for every labeled region is known as the _upper-limit magnitude_, which
is discussed in the next section (*note Upper limit surface brightness
of image::).
---------- Footnotes ----------
(1) The ground is the sky value in this analogy, see *note Sky
value::. Note that this analogy only holds for a flat sky value across
the surface of the image or ground.
(2) Note that this muddy water analogy is not perfect, because while
the water-level remains the same all over a peak, in data analysis, the
Poisson noise increases with the level of data.
(3) Ground-based instruments like the SDSS suffer from strong
smoothing due to the atmosphere. Therefore, increasing the pixel
resolution (or decreasing the width of a pixel) will not increase the
received information).
(4) If you add three datasets with noise $\sigma_1$, $\sigma_2$ and
$\sigma_3$, the resulting noise level is
$\sigma_t=\sqrt{\sigma_1^2+\sigma_2^2+\sigma_3^2}$, so when
$\sigma_1=\sigma_2=\sigma_3\equiv\sigma$, then
$\sigma_t=\sigma\sqrt{3}$. In this case, the area $A$ is covered by
$A/p$ pixels, so the noise level is $\sigma_t=\sigma\sqrt{A/p}$.
7.4.3.8 Upper limit surface brightness of image
...............................................
As mentioned in *note Surface brightness limit of image::, the surface
brightness limit assumes independent pixels when deriving the standard
deviation (the main input in the equation). It just extrapolates the
standard deviation derived from one pixel to the requested area. But as
mentioned at the end of that section, we have correlated noise in our
science-ready (deep) images and the noise of the pixels are not
independent.
Because of this, the surface brightness limit will always
under-estimate the surface brightness (give fainter values than what is
statistically possible in the data for the requested area). To account
for the correlated noise in the images, we need to derive the standard
deviation over a group of pixels that fall within a certain
footprint/shape. For example over a circular aperture of radius 5.6419
arcsec, or a square with a side length of $10$ arcsec. Depending on the
correlated noise systematics, the limit can be (very) different for
different shapes, even if they have the same area (as in the circle and
square mentioned in the previous sentence: both have an area of 100
arcsec$^2$).
Therefore we need to derive the standard deviation that goes into the
surface brightness limit equation over a certain footprint/shape. To do
that, we should:
1. Place the desired footprint many times randomly over all the
undetected pixels in an image. In MakeCatalog, the number of these
random positions can be configured with ‘--upnum’ and you can check
their positions with ‘--checkuplim’.
2. Calculate the sum of pixel values in each randomly placed
footprint.
3. Calculate the sigma-clipped standard deviation of the resulting
distribution (of sum of pixel values in the randomly placed
apertures). Therefore, each footprint's measurement is independent
of the other.
4. Calculate the surface brightness of that standrad deviation (after
multiplying it with your desired multiple of sigma). For the
definition of surface brightness, see *note Brightness flux
magnitude::.
If you have reviewed the previous sections, the measurements over
randomly placed apertures should remind you of *note Upper limit
magnitude of each detection::. Generally, the "upper limit" prefix is
given to all measurements with this way of measurement. Therefore this
limit is called "Upper limit surface brightness" of an image (for a
multiple of sigma, over a certain shape).
Traditionally a circular aperture of a fixed radius (in arcseconds)
has been used. In Gnuastro, a labeled image containing the desired
shape/aperture can be generated with MakeProfiles. The position of the
label is irrelevant because the upper limit measurements are done on the
many randomly placed footprints in undetected regions (independent of
where the label is positioned). That labeled image should then be given
to MakeCatalog, while requesting ‘--upperlimit-sb’. Of course, all
detected signal in the image needs to be masked (set to blank/NaN) so
MakeCatalog doesn't use randomly placed apertures that overlap with
detected signal in the image.
Going into the implementation details can get pretty hard to follow
in English, so a full hands-on tutorial is available in the second half
of *note Image surface brightness limit::. Read that tutorial with the
same input images and run the commands, and see each output image to get
a good understanding of how to properly measure the upper limit surface
brightness of your images.
7.4.4 Measuring elliptical parameters
-------------------------------------
The shape or morphology of a target is one of the most commonly desired
parameters of a target. Here, we will review the derivation of the most
basic/simple morphological parameters: the elliptical parameters for a
set of labeled pixels. The elliptical parameters are: the (semi-)major
axis, the (semi-)minor axis and the position angle along with the
central position of the profile. The derivations below follow the
SExtractor manual derivations with some added explanations for easier
reading.
Let's begin with one dimension for simplicity: Assume we have a set
of $N$ values $B_i$ (for example, showing the spatial distribution of a
target's brightness), each at position $x_i$. The simplest parameter we
can define is the geometric center of the object ($x_g$) (ignoring the
brightness values): $x_g=(\sum_ix_i)/N$. _Moments_ are defined to
incorporate both the value (brightness) and position of the data. The
first moment can be written as:
$$\overline{x}={\sum_iB_ix_i \over \sum_iB_i}$$
This is essentially the weighted (by $B_i$) mean position. The
geometric center ($x_g$, defined above) is a special case of this with
all $B_i=1$. The second moment is essentially the variance of the
distribution:
$$\overline{x^2}\equiv{\sum_iB_i(x_i-\overline{x})^2 \over
\sum_iB_i} = {\sum_iB_ix_i^2 \over \sum_iB_i} -
2\overline{x}{\sum_iB_ix_i\over\sum_iB_i} + \overline{x}^2
={\sum_iB_ix_i^2 \over \sum_iB_i} - \overline{x}^2$$
The last step was done from the definition of $\overline{x}$. Hence,
the square root of $\overline{x^2}$ is the spatial standard deviation
(along the one-dimension) of this particular brightness distribution
($B_i$). Crudely (or qualitatively), you can think of its square root
as the distance (from $\overline{x}$) which contains a specific amount
of the flux (depending on the $B_i$ distribution). Similar to the first
moment, the geometric second moment can be found by setting all $B_i=1$.
So while the first moment quantified the position of the brightness
distribution, the second moment quantifies how that brightness is
dispersed about the first moment. In other words, it quantifies how
"sharp" the object's image is.
Before continuing to two dimensions and the derivation of the
elliptical parameters, let's pause for an important implementation
technicality. You can ignore this paragraph and the next two if you do
not want to implement these concepts. The basic definition (first
definition of $\overline{x^2}$ above) can be used without any major
problem. However, using this fraction requires two runs over the data:
one run to find $\overline{x}$ and another run to find $\overline{x^2}$
from $\overline{x}$, this can be slow. The advantage of the last
fraction above, is that we can estimate both the first and second
moments in one run (since the $-\overline{x}^2$ term can easily be added
later).
The logarithmic nature of floating point number digitization creates
a complication however: suppose the object is located between pixels
10000 and 10020. Hence the target's pixels are only distributed over 20
pixels (with a standard deviation $<20$), while the mean has a value of
$\sim10000$. The $\sum_iB_i^2x_i^2$ will go to very very large values
while the individual pixel differences will be orders of magnitude
smaller. This will lower the accuracy of our calculation due to the
limited accuracy of floating point operations. The variance only
depends on the distance of each point from the mean, so we can shift all
position by a constant/arbitrary $K$ which is much closer to the mean:
$\overline{x-K}=\overline{x}-K$. Hence we can calculate the second
order moment using:
$$\overline{x^2}={\sum_iB_i(x_i-K)^2 \over \sum_iB_i} -
(\overline{x}-K)^2 $$
The closer $K$ is to $\overline{x}$, the better (the sums of squares
will involve smaller numbers), as long as $K$ is within the object
limits (in the example above: $10000\leq{K}\leq10020$), the floating
point error induced in our calculation will be negligible. For the most
simplest implementation, MakeCatalog takes $K$ to be the smallest
position of the object in each dimension. Since $K$ is arbitrary and an
implementation/technical detail, we will ignore it for the remainder of
this discussion.
In two dimensions, the mean and variances can be written as:
$$\overline{x}={\sum_iB_ix_i\over B_i}, \quad
\overline{x^2}={\sum_iB_ix_i^2 \over \sum_iB_i} -
\overline{x}^2$$
$$\overline{y}={\sum_iB_iy_i\over B_i}, \quad
\overline{y^2}={\sum_iB_iy_i^2 \over \sum_iB_i} -
\overline{y}^2$$
$$\quad\quad\quad\quad\quad\quad\quad\quad\quad
\overline{xy}={\sum_iB_ix_iy_i \over \sum_iB_i} -
\overline{x}\times\overline{y}$$
If an elliptical profile's major axis exactly lies along the $x$
axis, then $\overline{x^2}$ will be directly proportional with the
profile's major axis, $\overline{y^2}$ with its minor axis and
$\overline{xy}=0$. However, in reality we are not that lucky and
(assuming galaxies can be parameterized as an ellipse) the major axis of
galaxies can be in any direction on the image (in fact this is one of
the core principles behind weak-lensing by shear estimation). So the
purpose of the remainder of this section is to define a strategy to
measure the position angle and axis ratio of some randomly positioned
ellipses in an image, using the raw second moments that we have
calculated above in our image coordinates.
Let's assume we have rotated the galaxy by $\theta$, the new second
order moments are:
$$\overline{x_\theta^2} = \overline{x^2}\cos^2\theta +
\overline{y^2}\sin^2\theta -
2\overline{xy}\cos\theta\sin\theta $$
$$\overline{y_\theta^2} = \overline{x^2}\sin^2\theta +
\overline{y^2}\cos^2\theta +
2\overline{xy}\cos\theta\sin\theta$$
$$\overline{xy_\theta} = \overline{x^2}\cos\theta\sin\theta -
\overline{y^2}\cos\theta\sin\theta +
\overline{xy}(\cos^2\theta-\sin^2\theta)$$
The best $\theta$ ($\theta_0$, where major axis lies along the
$x_\theta$ axis) can be found by:
$$\left.{\partial \overline{x_\theta^2} \over \partial \theta}\right|_{\theta_0}=0$$
Taking the derivative, we get:
$$2\cos\theta_0\sin\theta_0(\overline{y^2}-\overline{x^2}) +
2(\cos^2\theta_0-\sin^2\theta_0)\overline{xy}=0$$ When
$\overline{x^2}\neq\overline{y^2}$, we can write:
$$\tan2\theta_0 =
2{\overline{xy} \over \overline{x^2}-\overline{y^2}}.$$
MakeCatalog uses the standard C math library's ‘atan2’ function to
estimate $\theta_0$, which we define as the position angle of the
ellipse. To recall, this is the angle of the major axis of the ellipse
with the $x$ axis. By definition, when the elliptical profile is
rotated by $\theta_0$, then $\overline{xy_{\theta_0}}=0$,
$\overline{x_{\theta_0}^2}$ will be the extent of the maximum variance
and $\overline{y_{\theta_0}^2}$ the extent of the minimum variance
(which are perpendicular for an ellipse). Replacing $\theta_0$ in the
equations above for $\overline{x_\theta}$ and $\overline{y_\theta}$, we
can get the semi-major ($A$) and semi-minor ($B$) lengths:
$$A^2\equiv\overline{x_{\theta_0}^2}= {\overline{x^2} +
\overline{y^2} \over 2} + \sqrt{\left({\overline{x^2}-\overline{y^2} \over 2}\right)^2 + \overline{xy}^2}$$
$$B^2\equiv\overline{y_{\theta_0}^2}= {\overline{x^2} +
\overline{y^2} \over 2} - \sqrt{\left({\overline{x^2}-\overline{y^2} \over 2}\right)^2 + \overline{xy}^2}$$
As a summary, it is important to remember that the units of $A$ and
$B$ are in pixels (the standard deviation of a positional distribution)
and that they represent the spatial light distribution of the object in
both image dimensions (rotated by $\theta_0$). When the object cannot
be represented as an ellipse, this interpretation breaks down:
$\overline{xy_{\theta_0}}\neq0$ and $\overline{y_{\theta_0}^2}$ will not
be the direction of minimum variance.
7.4.5 Adding new columns to MakeCatalog
---------------------------------------
MakeCatalog is designed to allow easy addition of different measurements
over a labeled image; see Akhlaghi 2016
(https://arxiv.org/abs/1611.06387v1). A check-list style description of
necessary steps to do that is described in this section. The common
development characteristics of MakeCatalog and other Gnuastro programs
is explained in *note Developing::. We strongly encourage you to have a
look at that chapter to greatly simplify your navigation in the code.
After adding and testing your column, you are most welcome (and
encouraged) to share it with us so we can add to the next release of
Gnuastro for everyone else to also benefit from your efforts.
MakeCatalog will first pass over each label's pixels two times and do
necessary raw/internal calculations. Once the passes are done, it will
use the raw information for filling the final catalog's columns. In the
first pass it will gather mainly object information and in the second
run, it will mainly focus on the clumps, or any other measurement that
needs an output from the first pass. These two passes are designed to
be raw summations: no extra processing. This will allow parallel
processing and simplicity/clarity. So if your new calculation, needs
new raw information from the pixels, then you will need to also modify
the respective ‘mkcatalog_first_pass’ and ‘mkcatalog_second_pass’
functions (both in ‘bin/mkcatalog/mkcatalog.c’) and define new raw table
columns in ‘main.h’ (hopefully the comments in the code are clear
enough).
In all these different places, the final columns are sorted in the
same order (same order as *note Invoking astmkcatalog::). This allows a
particular column/option to be easily found in all steps. Therefore in
adding your new option, be sure to keep it in the same relative place in
the list in all the separate places (it does not necessarily have to be
in the end), and near conceptually similar options.
‘main.h’
The ‘objectcols’ and ‘clumpcols’ enumerated variables (‘enum’)
define the raw/internal calculation columns. If your new column
requires new raw calculations, add a row to the respective list.
If your calculation requires any other settings parameters, you
should add a variable to the ‘mkcatalogparams’ structure.
‘ui.c’
If the new column needs raw calculations (an entry was added in
‘objectcols’ and ‘clumpcols’), specify which inputs it needs in
‘ui_necessary_inputs’, similar to the other options. Afterwards,
if your column includes any particular settings (you needed to add
a variable to the ‘mkcatalogparams’ structure in ‘main.h’), you
should do the sanity checks and preparations for it here.
‘ui.h’
The ‘option_keys_enum’ associates a unique value for each option to
MakeCatalog. The options that have a short option version, the
single character short comment is used for the value. Those that
do not have a short option version, get a large integer
automatically. You should add a variable here to identify your
desired column.
‘args.h’
This file specifies all the parameters for the GNU C library, Argp
structure that is in charge of reading the user's options. To
define your new column, just copy an existing set of parameters and
change the first, second and 5th values (the only ones that differ
between all the columns), you should use the macro you defined in
‘ui.h’ here.
‘columns.c’
This file contains the main definition and high-level calculation
of your new column through the ‘columns_define_alloc’ and
‘columns_fill’ functions. In the first, you specify the basic
information about the column: its name, units, comments, type (see
*note Numeric data types::) and how it should be printed if the
output is a text file. You should also specify the raw/internal
columns that are necessary for this column here as the many
existing examples show. Through the types for objects and rows,
you can specify if this column is only for clumps, objects or both.
The second main function (‘columns_fill’) writes the final value
into the appropriate column for each object and clump. As you can
see in the many existing examples, you can define your processing
on the raw/internal calculations here and save them in the output.
‘mkcatalog.c’
This file contains the low-level parsing functions. To be
optimized, the parsing is done in parallel through the
‘mkcatalog_single_object’ function. This function initializes the
necessary arrays and calls the lower-level ‘parse_objects’ and
‘parse_clumps’ for actually going over the pixels. They are all
heavily commented, so you should be able to follow where to add
your necessary low-level calculations.
‘doc/gnuastro.texi’
Update this manual and add a description for the new column.
7.4.6 MakeCatalog measurements
------------------------------
MakeCatalog's output measurements/columns can be specified using
command-line options (*note Options::). The current measurements in
MakeCatalog are those which only produce one final value for each label
(for example, its magnitude: a single number). All the different
label's measurements can be written as one column in a final
table/catalog that contains other columns for other similar
single-number measurements.
In this case, all the different label's measurements can be written
as one column in a final table/catalog that contains other columns for
other similar single-number measurements. The majority of this section
is devoted to MakeCatalog's single-valued measurements. However,
MakeCatalog can also do measurements that produce more than one value
for each label. Currently the only such measurement is generation of
spectra from 3D cubes with the ‘--spectrum’ option and it is discussed
in the end of this section.
Command-line options are used to identify which measurements you want
in the final catalog(s) and in what order. If any of the options below
is called on the command-line or in any of the configuration files, it
will be included as a column in the output catalog. The order of the
columns is in the same order as the options were seen by MakeCatalog
(see *note Configuration file precedence::). Some of the columns apply
to both "objects" and "clumps" and some are particular to only one of
them (for the definition of "objects" and "clumps", see *note
Segment::). Columns/options that are unique to one catalog (only
objects, or only clumps), are explicitly marked with [Objects] or
[Clumps] to specify the catalog they will be placed in.
7.4.6.1 Identifier columns
..........................
The identifier of each row (group of measurements) is usually the first
thing you will be requesting from MakeCatalog. Without the identifier,
it is not clear which measurement corresponds to which label for the
input.
Since MakeCatalog can also optionally take sub-structure label
(clumps; see *note Segment::), there are various identifiers in general
that are listed below. The most generic (and shortest and easiest to
type!) is the ‘--ids’ option which can be used in object-only or
object-clump catalogs.
‘--i’
‘--ids’
This is a unique option which can add multiple columns to the final
catalog(s). Calling this option will put the object IDs
(‘--obj-id’) in the objects catalog and host-object-ID
(‘--host-obj-id’) and ID-in-host-object (‘--id-in-host-obj’) into
the clumps catalog. Hence if only object catalogs are required, it
has the same effect as ‘--obj-id’.
‘--obj-id’
[Objects] ID of this object.
‘-j’
‘--host-obj-id’
[Clumps] The ID of the object which hosts this clump.
‘--id-in-host-obj’
[Clumps] The ID of this clump in its host object.
7.4.6.2 Position measurements in pixels
.......................................
The position of a labeled region within your input dataset (in its own
units) can be measured with the options in this section. By "in its own
units" we mean pixels in a 2D image or voxels in a 3D cube. For example
if the flux-weighted center of a label lies 123 pixels on the horizontal
and 456 pixels on the vertical, the ‘--x’ and ‘--y’ options will put a
value of 123 and 456 in their respective columns. As you see below,
there are various ways to define the "position" of an object, so read
the differences carefully to choose the one that corresponds best to
your usage.
‘-x’
‘--x’
The flux weighted center of all objects and clumps along the first
FITS axis (horizontal when viewed in SAO DS9), see $\overline{x}$
in *note Measuring elliptical parameters::. The weight has to have
a positive value (pixel value larger than the Sky value) to be
meaningful! Specially when doing matched photometry, this might
not happen: no pixel value might be above the Sky value. For such
detections, the geometric center will be reported in this column
(see ‘--geo-x’). You can use ‘--weight-area’ to see which was
used.
‘-y’
‘--y’
The flux weighted center of all objects and clumps along the second
FITS axis (vertical when viewed in SAO DS9). See ‘--x’.
‘-z’
‘--z’
The flux weighted center of all objects and clumps along the third
FITS axis. See ‘--x’.
‘--geo-x’
The geometric center of all objects and clumps along the first FITS
axis axis. The geometric center is the average pixel positions
irrespective of their pixel values.
‘--geo-y’
The geometric center of all objects and clumps along the second
FITS axis axis, see ‘--geo-x’.
‘--geo-z’
The geometric center of all objects and clumps along the third FITS
axis axis, see ‘--geo-x’.
‘--min-val-x’
Position of pixel with minimum value in objects and clumps, along
the first FITS axis.
‘--max-val-x’
Position of pixel with maximum value in objects and clumps, along
the first FITS axis.
‘--min-val-y’
Position of pixel with minimum value in objects and clumps, along
the first FITS axis.
‘--max-val-y’
Position of pixel with maximum value in objects and clumps, along
the first FITS axis.
‘--min-val-z’
Position of pixel with minimum value in objects and clumps, along
the first FITS axis.
‘--max-val-z’
Position of pixel with maximum value in objects and clumps, along
the first FITS axis.
‘--min-x’
The minimum position of all objects and clumps along the first FITS
axis.
‘--max-x’
The maximum position of all objects and clumps along the first FITS
axis.
‘--min-y’
The minimum position of all objects and clumps along the second
FITS axis.
‘--max-y’
The maximum position of all objects and clumps along the second
FITS axis.
‘--min-z’
The minimum position of all objects and clumps along the third FITS
axis.
‘--max-z’
The maximum position of all objects and clumps along the third FITS
axis.
‘--clumps-x’
[Objects] The flux weighted center of all the clumps in this object
along the first FITS axis. See ‘--x’.
‘--clumps-y’
[Objects] The flux weighted center of all the clumps in this object
along the second FITS axis. See ‘--x’.
‘--clumps-z’
[Objects] The flux weighted center of all the clumps in this object
along the third FITS axis. See ‘--x’.
‘--clumps-geo-x’
[Objects] The geometric center of all the clumps in this object
along the first FITS axis. See ‘--geo-x’.
‘--clumps-geo-y’
[Objects] The geometric center of all the clumps in this object
along the second FITS axis. See ‘--geo-x’.
‘--clumps-geo-z’
[Objects] The geometric center of all the clumps in this object
along the third FITS axis. See ‘--geo-z’.
7.4.6.3 Position measurements in WCS
....................................
The position of a labeled region within your input dataset (in the World
Coordinate System, or WCS) can be measured with the options in this
section. As you see below, there are various ways to define the
"position" of an object, so read the differences carefully to choose the
one that corresponds best to your usage.
The most common WCS coordinates are Right Ascension (RA) and
Declination in an equatorial system. Therefore, to simplify their
usage, we have special ‘--ra’ and ‘--dec’ options. However, the WCS of
datasets are in Galactic coordinates, so to be generic, you can use the
‘--w1’, ‘--w2’ or ‘--w3’ (if you have a 3D cube) options. In case your
dataset's WCS is not in your desired system (for example it is Galactic,
but you want equatorial 2000), you can use the ‘--wcscoordsys’ option of
Gnuastro's Fits program on the labeled image before running MakeCatalog
(see *note Keyword inspection and manipulation::).
‘-r’
‘--ra’
Flux weighted right ascension of all objects or clumps, see ‘--x’.
This is just an alias for one of the lower-level ‘--w1’ or ‘--w2’
options. Using the FITS WCS keywords (‘CTYPE’), MakeCatalog will
determine which axis corresponds to the right ascension. If no
‘CTYPE’ keywords start with ‘RA’, an error will be printed when
requesting this column and MakeCatalog will abort.
‘-d’
‘--dec’
Flux weighted declination of all objects or clumps, see ‘--x’.
This is just an alias for one of the lower-level ‘--w1’ or ‘--w2’
options. Using the FITS WCS keywords (‘CTYPE’), MakeCatalog will
determine which axis corresponds to the declination. If no ‘CTYPE’
keywords start with ‘DEC’, an error will be printed when requesting
this column and MakeCatalog will abort.
‘--w1’
Flux weighted first WCS axis of all objects or clumps, see ‘--x’.
The first WCS axis is commonly used as right ascension in images.
‘--w2’
Flux weighted second WCS axis of all objects or clumps, see ‘--x’.
The second WCS axis is commonly used as declination in images.
‘--w3’
Flux weighted third WCS axis of all objects or clumps, see ‘--x’.
The third WCS axis is commonly used as wavelength in integral field
unit data cubes.
‘--geo-w1’
Geometric center in first WCS axis of all objects or clumps, see
‘--geo-x’. The first WCS axis is commonly used as right ascension
in images.
‘--geo-w2’
Geometric center in second WCS axis of all objects or clumps, see
‘--geo-x’. The second WCS axis is commonly used as declination in
images.
‘--geo-w3’
Geometric center in third WCS axis of all objects or clumps, see
‘--geo-x’. The third WCS axis is commonly used as wavelength in
integral field unit data cubes.
‘--clumps-w1’
[Objects] Flux weighted center in first WCS axis of all clumps in
this object, see ‘--x’. The first WCS axis is commonly used as
right ascension in images.
‘--clumps-w2’
[Objects] Flux weighted declination of all clumps in this object,
see ‘--x’. The second WCS axis is commonly used as declination in
images.
‘--clumps-w3’
[Objects] Flux weighted center in third WCS axis of all clumps in
this object, see ‘--x’. The third WCS axis is commonly used as
wavelength in integral field unit data cubes.
‘--clumps-geo-w1’
[Objects] Geometric center right ascension of all clumps in this
object, see ‘--geo-x’. The first WCS axis is commonly used as
right ascension in images.
‘--clumps-geo-w2’
[Objects] Geometric center declination of all clumps in this
object, see ‘--geo-x’. The second WCS axis is commonly used as
declination in images.
‘--clumps-geo-w3’
[Objects] Geometric center in third WCS axis of all clumps in this
object, see ‘--geo-x’. The third WCS axis is commonly used as
wavelength in integral field unit data cubes.
7.4.6.4 Brightness measurements
...............................
Within an image, pixels have both a position and a value. In the
sections above all the measurements involved position (see *note
Position measurements in pixels:: or *note Position measurements in
WCS::). The measurements in this section only deal with pixel values
and ignore the pixel positions completely. In other words, for the
options of this section each labeled region within the input is just a
group of values (and their associated error values if necessary), and
they let you do various types of measurements on the resulting
distribution of values.
‘--sum’
The sum of all pixel values associated to this label (object or
clump). Note that if a sky value or image has been given, it will
be subtracted before any column measurement. For clumps, the
ambient values (average of river pixels around the clump,
multiplied by the area of the clump) is subtracted, see
‘--river-mean’. So the sum of all the clump-sums in the clump
catalog of one object will be smaller than the ‘--clumps-sum’
column of the objects catalog.
If no usable pixels are present over the clump or object (for
example, they are all blank), the returned value will be NaN (note
that zero is meaningful).
‘--sum-error’
The ($1\sigma$) error in measuring the sum of values of a label
(objects or clumps).
The returned value will be NaN when the label covers only NaN
pixels in the values image, or a pixel is NaN in the ‘--instd’
image, but non-NaN in the values image. The latter situation
usually happens when there is a bug in the previous steps of your
analysis, and is important because those pixels with a NaN in the
‘--instd’ image may contribute significantly to the final error.
If you want to ignore those pixels in the error measurement, set
them to zero (which is a meaningful number in such scenarios).
‘--clumps-sum’
[Objects] The total sum of the pixels covered by clumps (before
subtracting the river) within each object. This is simply the sum
of ‘--sum-no-river’ in the clumps catalog (see below). If no
usable pixels are present over the clump or object (for example,
they are all blank), the stored value will be NaN (note that zero
is meaningful).
‘--sum-no-river’
[Clumps] The sum of Sky (not river) subtracted clump pixel values.
By definition, for the clumps, the average value of the rivers
surrounding it are subtracted from it for a first order accounting
for contamination by neighbors.
If no usable pixels are present over the clump or object (for
example, they are all blank), the stored value will be NaN (note
that zero is meaningful).
‘--mean’
The mean sky subtracted value of pixels within the object or clump.
For clumps, the average river flux is subtracted from the sky
subtracted mean.
‘--std’
The standard deviation of the pixels within the object or clump.
For clumps, the river pixels are not subtracted because that is a
constant (per pixel) value and should not affect the standard
deviation.
‘--median’
The median sky subtracted value of pixels within the object or
clump. For clumps, the average river flux is subtracted from the
sky subtracted median.
‘--maximum’
The maximum value of pixels within the object or clump. When the
label (object or clump) is larger than three pixels, the maximum is
actually derived by the mean of the brightest three pixels, not the
largest pixel value of the same label. This is because noise
fluctuations can be very strong in the extreme values of the
objects/clumps due to Poisson noise (which gets stronger as the
mean gets higher). Simply using the maximum pixel value will
create a strong scatter in results that depend on the maximum (for
example, the ‘--fwhm’ option also uses this value internally).
‘--sigclip-number’
The number of elements/pixels in the dataset after sigma-clipping
the object or clump. The sigma-clipping parameters can be set with
the ‘--sigmaclip’ option described in *note MakeCatalog inputs and
basic settings::. For more on Sigma-clipping, see *note Sigma
clipping::.
‘--sigclip-median’
The sigma-clipped median value of the object of clump's pixel
distribution. For more on sigma-clipping and how to define it, see
‘--sigclip-number’.
‘--sigclip-mean’
The sigma-clipped mean value of the object of clump's pixel
distribution. For more on sigma-clipping and how to define it, see
‘--sigclip-number’.
‘--sigclip-std’
The sigma-clipped standard deviation of the object of clump's pixel
distribution. For more on sigma-clipping and how to define it, see
‘--sigclip-number’.
‘-m’
‘--magnitude’
The magnitude of clumps or objects, see ‘--sum’.
‘--magnitude-error’
The magnitude error of clumps or objects. The magnitude error is
calculated from the signal-to-noise ratio (see ‘--sn’ and *note
Quantifying measurement limits::). Note that until now this error
assumes uncorrelated pixel values and also does not include the
error in estimating the aperture (or error in generating the
labeled image).
For now these factors have to be found by other means. Task 14124
(https://savannah.gnu.org/task/index.php?14124) has been defined
for work on adding these sources of error too.
The returned value will be NaN when the label covers only NaN
pixels in the values image, or a pixel is NaN in the ‘--instd’
image, but non-NaN in the values image. The latter situation
usually happens when there is a bug in the previous steps of your
analysis, and is important because those pixels with a NaN in the
‘--instd’ image may contribute significantly to the final error.
If you want to ignore those pixels in the error measurement, set
them to zero (which is a meaningful number in such scenarios).
‘--clumps-magnitude’
[Objects] The magnitude of all clumps in this object, see
‘--clumps-sum’.
‘--upperlimit’
The upper limit value (in units of the input image) for this object
or clump. This is the sigma-clipped standard deviation of the
random distribution, multiplied by the value of ‘--upnsigma’). See
*note Quantifying measurement limits:: and *note Upper-limit
settings:: for a complete explanation. This is very important for
the fainter and smaller objects in the image where the measured
magnitudes are not reliable.
‘--upperlimit-mag’
The upper limit magnitude for this object or clump. See *note
Quantifying measurement limits:: and *note Upper-limit settings::
for a complete explanation. This is very important for the fainter
and smaller objects in the image where the measured magnitudes are
not reliable.
‘--upperlimit-onesigma’
The $1\sigma$ upper limit value (in units of the input image) for
this object or clump. See *note Quantifying measurement limits::
and *note Upper-limit settings:: for a complete explanation. When
‘--upnsigma=1’, this column's values will be the same as
‘--upperlimit’.
‘--upperlimit-sigma’
The position of the label's sum measured within the distribution of
randomly placed upperlimit measurements in units of the
distribution's $\sigma$ or standard deviation. See *note
Quantifying measurement limits:: and *note Upper-limit settings::
for a complete explanation.
‘--upperlimit-quantile’
The position of the label's sum within the distribution of randomly
placed upperlimit measurements as a quantile (value between 0 or
1). See *note Quantifying measurement limits:: and *note
Upper-limit settings:: for a complete explanation. If the object
is brighter than the brightest randomly placed profile, a value of
‘inf’ is returned. If it is less than the minimum, a value of
‘-inf’ is reported.
‘--upperlimit-skew’
This column contains the non-parametric skew of the
$\sigma$-clipped random distribution that was used to estimate the
upper-limit magnitude. Taking $\mu$ as the mean, $\nu$ as the
median and $\sigma$ as the standard deviation, the traditional
definition of skewness is defined as: $(\mu-\nu)/\sigma$.
This can be a good measure to see how much you can trust the random
measurements, or in other words, how accurately the regions with
signal have been masked/detected. If the skewness is strong (and
to the positive), then you can tell that you have a lot of
undetected signal in the dataset, and therefore that the
upper-limit measurement (and other measurements) are not reliable.
‘--river-mean’
[Clumps] The average of the river pixel values around this clump.
River pixels were defined in Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664). In short they are the pixels
immediately outside of the clumps. This value is used internally
to find the sum (or magnitude) and signal to noise ratio of the
clumps. It can generally also be used as a scale to gauge the base
(ambient) flux surrounding the clump. In case there was no river
pixels, then this column will have the value of the Sky under the
clump. So note that this value is _not_ sky subtracted.
‘--river-num’
[Clumps] The number of river pixels around this clump, see
‘--river-mean’.
‘--river-min’
[Clumps] Minimum river value around this clump, see ‘--river-mean’.
‘--river-max’
[Clumps] Maximum river value around this clump, see ‘--river-mean’.
‘--sn’
The Signal to noise ratio (S/N) of all clumps or objects. See
Akhlaghi and Ichikawa (2015) for the exact equations used.
The returned value will be NaN when the label covers only NaN
pixels in the values image, or a pixel is NaN in the ‘--instd’
image, but non-NaN in the values image. The latter situation
usually happens when there is a bug in the previous steps of your
analysis, and is important because those pixels with a NaN in the
‘--instd’ image may contribute significantly to the final error.
If you want to ignore those pixels in the error measurement, set
them to zero (which is a meaningful number).
‘--sky’
The sky flux (per pixel) value under this object or clump. This is
actually the mean value of all the pixels in the sky image that lie
on the same position as the object or clump.
‘--sky-std’
The sky value standard deviation (per pixel) for this clump or
object. This is the square root of the mean variance under the
object, or the root mean square.
7.4.6.5 Surface brightness measurements
.......................................
In astronomy, Surface brightness is most commonly measured in units of
magnitudes per arcsec$^2$ (for the formal definition, see *note
Brightness flux magnitude::). Therefore it involves both the values of
the pixels within each input label (or output row) and their position.
‘--sb’
The surface brightness (in units of mag/arcsec$^2$) of the labeled
region (objects or clumps). For more on the definition of the
surface brightness, see *note Brightness flux magnitude::.
‘--sb-error’
Error in measuring the surface brightness (the ‘--sb’ column).
This column will use the value given to ‘--spatialresolution’ in
the processing (in pixels). For more on ‘--spatialresolution’, see
*note MakeCatalog inputs and basic settings:: and for the equation
used to derive the surface brightness error, see *note Surface
brightness error of each detection::.
‘--upperlimit-sb’
The upper-limit surface brightness (in units of mag/arcsec$^2$) of
this labeled region (object or clump). In other words, this option
measures the surface brightness of noise within the footprint of
each input label.
This is just a simple wrapper over lower-level columns: setting B
and A as the value in the columns ‘--upperlimit’ and
‘--area-arcsec2’, we fill this column by simply use the surface
brightness equation of *note Brightness flux magnitude::.
‘--half-sum-sb’
Surface brightness (in units of mag/arcsec$^2$) within the area
that contains half the total sum of the label's pixels (object or
clump). This is useful as a measure of the sharpness of an
astronomical object: for example a star will have very few pixels
at half the maximum, so its ‘--half-sum-sb’ will be much brighter
than a galaxy at the same magnitude. Also consider ‘--half-max-sb’
below.
This column just plugs in the values of half the value of the
‘--sum’ column and the ‘--half-sum-area’ column, into the surface
brightness equation. Therefore please see the description in
‘--half-sum-area’ to understand the systematics of this column and
potential biases (see *note Morphology measurements
nonparametric::).
‘--half-max-sb’
The surface brightness (in units of mag/arcsec$^2$) within the
region that contains half the maximum value of the labeled region.
Like ‘--half-sum-sb’ this option this is a good way to identify the
"central" surface brightness of an object. To know when this
measurement is reasonable, see the description of ‘--fwhm’ in *note
Morphology measurements nonparametric::.
‘--sigclip-mean-sb’
Surface brightness (over 1 pixel's area in arcsec$^2$) of the
sigma-clipped mean value of the pixel values distribution
associated to each label (object or clump). This is useful in
scenarios where your labels have approximately _constant_ surface
brightness values _after_ after removing outliers: for example in a
radial profile, see *note Invoking astscript-radial-profile::).
In other scenarios it should be used with extreme care. For
example over the full area of a galaxy/star the pixel distribution
is not constant (or symmetric after adding noise), their pixel
distributions are inherently skewed (with fewer pixels in the
center, having a very large value and many pixels in the outer
parts having lower values). Therefore, sigma-clipping is not
meaningful for such objects! For more on the definition of the
surface brightness, see *note Brightness flux magnitude::, for more
on sigma-clipping, see *note Sigma clipping::.
The error in this magnitude can be retrieved from the
‘--sigclip-mean-sb-delta’ column described below, and you can use
the ‘--sigclip-std-sb’ column to find when the magnitude has become
noise-dominated (signal-to-noise ratio is roughly 1). See the
description of these two options for more.
‘--sigclip-mean-sb-delta’
Scatter in the ‘--sigclip-mean-sb’ without using the standard
deviation of each pixel (that is given by ‘--instd’ in *note
MakeCatalog inputs and basic settings::). The scatter here is
measured from the values of the label themselves. This measurement
is therefore most meaningful when you expect the flux across one
label to be constant (as in a radial profile for example).
This is calculated using the equation in *note Surface brightness
error of each detection::, where $\Delta{A}=0$ (since sigma-clip is
calculated per pixel and there is no error in a single pixel).
Within the equation to derive $\Delta{M}$ (the error in magnitude,
derived in *note Magnitude measurement error of each detection::),
the signal-to-noise ratio is defined by dividing the sigma-clipped
mean by the sigma-clipped standard deviation.
‘--sigclip-std-sb’
The surface brightness of the sigma-clipped standard deviation of
all the pixels with the same label. For labels that are expected
to have the same value in all their pixels (for example each
annulus of a radial profile) this can be used to find the reliable
($1\sigma$) surface brightness for that label. In other words, if
‘--sigclip-mean-sb’ is fainter than the value of this column, you
know that noise is becoming significant. However, as described in
‘--sigclip-mean-sb’, the sigma-clipped measurements of MakeCatalog
should only be used in certain situations like radial profiles, see
the description there for more.
7.4.6.6 Morphology measurements (non-parametric)
................................................
Morphology defined as a way to quantify the "shape" of an object in your
input image. This includes both the position and value of the pixels
within your input labels. There are many ways to define the morphology
of an object. In this section, we will review the available
non-parametric measures of morphology. By non-parametric, we mean that
no functional shape is assumed for the measurement.
In *note Morphology measurements elliptical:: you can see some
parametric elliptical measurements (which are only valid when the object
is actually an ellipse).
‘--num-clumps’
[Objects] The number of clumps in this object.
‘--area’
The raw area (number of pixels/voxels) in any clump or object
independent of what pixel it lies over (if it is NaN/blank or
unused for example).
‘--arcsec2-area’
The used (non-blank in values image) area of the labeled region in
units of arc-seconds squared. This column is just the value of the
‘--area’ column, multiplied by the area of each pixel in the input
image (in units of arcsec^2). Similar to the ‘--ra’ or ‘--dec’
columns, for this option to work, the objects extension used has to
have a WCS structure.
‘--area-min-val’
The number of pixels that are equal to the minimum value of the
labeled region (clump or object).
‘--area-max-val’
The number of pixels that are equal to the maximum value of the
labeled region (clump or object).
‘--area-xy’
Similar to ‘--area’, when the clump or object is projected onto the
first two dimensions. This is only available for 3-dimensional
datasets. When working with Integral Field Unit (IFU) datasets,
this projection onto the first two dimensions would be a
narrow-band image.
‘--fwhm’
The full width at half maximum (in units of pixels, along the
semi-major axis) of the labeled region (object or clump). The
maximum value is estimated from the mean of the top-three pixels
with the highest values, see the description under ‘--maximum’.
The number of pixels that have half the value of that maximum are
then found (value in the ‘--half-max-area’ column) and a radius is
estimated from the area. See the description under
‘--half-sum-radius’ for more on converting area to radius along
major axis.
Because of its non-parametric nature, this column is most reliable
on clumps and should only be used in objects with great caution.
This is because objects can have more than one clump (peak with
true signal) and multiple peaks are not treated separately in
objects, so the result of this column will be biased.
Also, because of its non-parametric nature, this FWHM it does not
account for the PSF, and it will be strongly affected by noise if
the object is faint/diffuse So when half the maximum value (which
can be requested using the ‘--maximum’ column) is too close to the
local noise level (which can be requested using the ‘--sky-std’
column), the value returned in this column is meaningless (its just
noise peaks which are randomly distributed over the area). You can
therefore use the ‘--maximum’ and ‘--sky-std’ columns to remove, or
flag, unreliable FWHMs. For example, if a labeled region's maximum
is less than 2 times the sky standard deviation, the value will
certainly be unreliable (half of that is $1\sigma$!). For a more
reliable value, this fraction should be around 4 (so half the
maximum is 2$\sigma$).
‘--half-max-area’
The number of pixels with values larger than half the maximum flux
within the labeled region. This option is used to estimate
‘--fwhm’, so please read the notes there for the caveats and
necessary precautions.
‘--half-max-radius’
The radius of region containing half the maximum flux within the
labeled region. This is just half the value reported by ‘--fwhm’.
‘--half-max-sum’
The sum of the pixel values containing half the maximum flux within
the labeled region (or those that are counted in ‘--halfmaxarea’).
This option uses the pixels within ‘--fwhm’, so please read the
notes there for the caveats and necessary precautions.
‘--half-sum-area’
The number of pixels that contain half the object or clump's total
sum of pixels (half the value in the ‘--sum’ column). To count
this area, all the non-blank values associated with the given label
(object or clump) will be sorted and summed in order (starting from
the maximum), until the sum becomes larger than half the total sum
of the label's pixels.
This option is thus good for clumps (which are defined to have a
single peak in their morphology), but for objects you should be
careful: if the object includes multiple peaks/clumps at roughly
the same level, then the area reported by this option will be
distributed over all the peaks.
‘--half-sum-radius’
Radius (in units of pixels) derived from the area that contains
half the total sum of the label's pixels (value reported by
‘--halfsumarea’). If the area is $A_h$ and the axis ratio is $q$,
then the value returned in this column is $\sqrt{A_h/({\pi}q)}$.
This option is a good measure of the concentration of the
_observed_ (after PSF convolution and noisy) object or clump, But
as described below it underestimates the effective radius. Also,
it should be used in caution with objects that may have multiple
clumps. It is most reliable with clumps or objects that have one
or zero clumps, see the note under ‘--halfsumarea’.
Recall that in general, for an ellipse with semi-major axis $a$,
semi-minor axis $b$, and axis ratio $q=b/a$ the area ($A$) is
$A={\pi}ab={\pi}qa^2$. For a circle (where $q=1$), this simplifies
to the familiar $A={\pi}a^2$.
This option should not be confused with the _effective radius_ for
Sérsic profiles, commonly written as $r_e$. For more on the Sérsic
profile and $r_e$, please see *note Galaxies::. Therefore, when
$r_e$ is meaningful for the target (the target is elliptically
symmetric and can be parameterized as a Sérsic profile), $r_e$
should be derived from fitting the profile with a Sérsic function
which has been convolved with the PSF. But from the equation above,
you see that this radius is derived from the raw image's labeled
values (after convolution, with no parametric profile), so this
column's value will generally be (much) smaller than $r_e$,
depending on the PSF, depth of the dataset, the morphology, or if a
fraction of the profile falls on the edge of the image.
In other words, this option can only be interpreted as an effective
radius if there is no noise and no PSF and the profile within the
image extends to infinity (or a very large multiple of the
effective radius) and it not near the edge of the image.
‘--frac-max1-area’
‘--frac-max2-area’
Number of pixels brighter than the given fraction(s) of the maximum
pixel value. For the maximum value, see the description of
‘--maximum’ column. The fraction(s) are given through the
‘--frac-max’ option (that can take two values) and is described in
*note MakeCatalog inputs and basic settings::. Recall that in
‘--halfmaxarea’, the fraction is fixed to 0.5. Hence, added with
these two columns, you can sample three parts of the profile area.
‘--frac-max1-sum’
‘--frac-max2-sum’
Sum of pixels brighter than the given fraction(s) of the maximum
pixel value. For the maximum value, see the description of
‘--maximum’ column below. The fraction(s) are given through the
‘--frac-max’ option (that can take two values) and is described in
*note MakeCatalog inputs and basic settings::. Recall that in
‘--halfmaxsum’, the fraction is fixed to 0.5. Hence, added with
these two columns, you can sample three parts of the profile's sum
of pixels.
‘--frac-max1-radius’
‘--frac-max2-radius’
Radius (in units of pixels) derived from the area that contains the
given fractions of the maximum valued pixel(s) of the label's
pixels (value reported by ‘--frac-max1-area’ or
‘--frac-max2-area’). For the maximum value, see the description of
‘--maximum’ column below. The fractions are given through the
‘--frac-max’ option (that can take two values) and is described in
*note MakeCatalog inputs and basic settings::. Recall that in
‘--fwhm’, the fraction is fixed to 0.5. Hence, added with these
two columns, you can sample three parts of the profile's radius.
‘--clumps-area’
[Objects] The total area of all the clumps in this object.
‘--weight-area’
The area (number of pixels) used in the flux weighted position
calculations.
‘--geo-area’
The area of all the pixels labeled with an object or clump. Note
that unlike ‘--area’, pixel values are completely ignored in this
column. For example, if a pixel value is blank, it will not be
counted in ‘--area’, but will be counted here.
‘--geo-area-xy’
Similar to ‘--geo-area’, when the clump or object is projected onto
the first two dimensions. This is only available for 3-dimensional
datasets. When working with Integral Field Unit (IFU) datasets,
this projection onto the first two dimensions would be a
narrow-band image.
7.4.6.7 Morphology measurements (elliptical)
............................................
When your target objects are sufficiently ellipse-like, you can use the
measurements below to quantify the various parameters of the ellipse.
For details of how the elliptical parameters are measured, see *note
Measuring elliptical parameters::. For non-parametric morphological
measurements, see *note Morphology measurements nonparametric::. The
measures that start with ‘--geo-*’ ignore the pixel values and just do
the measurements on the label's "geometric" shape.
‘--semi-major’
The pixel-value weighted root mean square (RMS) along the
semi-major axis of the profile (assuming it is an ellipse) in units
of pixels.
‘--semi-minor’
The pixel-value weighted root mean square (RMS) along the
semi-minor axis of the profile (assuming it is an ellipse) in units
of pixels.
‘--axis-ratio’
The pixel-value weighted axis ratio (semi-minor/semi-major) of the
object or clump.
‘--position-angle’
The pixel-value weighted angle of the semi-major axis with the
first FITS axis in degrees.
‘--geo-semi-major’
The geometric (ignoring pixel values) root mean square (RMS) along
the semi-major axis of the profile, assuming it is an ellipse, in
units of pixels.
‘--geo-semi-minor’
The geometric (ignoring pixel values) root mean square (RMS) along
the semi-minor axis of the profile, assuming it is an ellipse, in
units of pixels.
‘--geo-axis-ratio’
The geometric (ignoring pixel values) axis ratio of the profile,
assuming it is an ellipse.
‘--geo-position-angle’
The geometric (ignoring pixel values) angle of the semi-major axis
with the first FITS axis in degrees.
7.4.6.8 Measurements per slice (spectra)
........................................
When the input is a 3D data cube, MakeCatalog has the following
multi-valued measurements per label. For a tutorial on how to use these
options and interpret their values, see *note Detecting lines and
extracting spectra in 3D data::.
These options will do measurements on each 2D slice of the input 3D
cube; hence the common the format of ‘--*-in-slice’. Each slice usually
corresponds to a certain wavelength, you can also think of these
measurements as spectra.
For each row (input label), each of the columns described here will
contain multiple values as a vector column. The number of measurements
in each column is the number of slices in the cube, or the size of the
cube along the third dimension. To learn more about vector columns and
how to manipulate them, see *note Vector columns::. For example usage
of these columns in the tutorial above, see *note 3D measurements and
spectra:: and *note Extracting a single spectrum and plotting it::.
There are two ways to do each measurement on a slice for each label:
Only label
The measurement will only be done on the voxels in the slice that
are associated to that label. These types of per-slice measurement
therefore have the following properties:
• This will only be a measurement of that label and will not be
affected by any other label.
• The number of voxels used in each slice can be different
(usually only one or two voxels at the two extremes of the
label (along the third dimension), and many in the middle.
• Since most labels are localized along the third dimension
(maybe only covering 20 slices out of thousands!), many of the
measurements (on slices where the label doesn't exist) will be
NaN (for the sum measurements for example) or 0 (for the area
measurements).
Projected label
MakeCatalog will first project the 3D label into a 2D surface
(along the third dimension) to get its 2D footprint. Afterwards,
all the voxels in that 2D footprint will be measured all slices.
All these measurements will have a ‘-proj-’ component in their
name. These types of per-slice measurement therefore has the
following properties:
• A measurement will be done on each slice of the cube.
• All measurements will be done on the same surface area.
• Labels can overlap when they are projected onto the first two
FITS dimensions (the spatial coordinates, not spectral). As a
result, other emission lines or objects may contaminate the
resulting spectrum for each label.
To help separate other labels, MakeCatalog can do a third type of
measurement on each slice: measurements on the voxels that belong
to other labels but overlap with the 2D projection. This can be
used to see how much your projected measurement is affected by
other emission sources (on the projected spectra) and also if
multiple lines (labeled regions) belong to the same physical
object. These measurements contain ‘-other-’ in their name.
‘--sum-in-slice’
[Only label] Sum of values in each slice.
‘--sum-err-in-slice’
[Only label] Error in '-sum-in-slice'.
‘--area-in-slice’
[Only label] Number of labeled in each slice.
‘--sum-proj-in-slice’
[Projected label] Sum of projected area in each slice.
‘--area-proj-in-slice:’
[Projected label] Number of voxels that are used in
‘--sum-proj-in-slice’.
‘--sum-proj-err-in-slice’
[Projected label] Error of ‘--sum-proj-in-slice’.
‘--area-other-in-slice’
[Projected label] Area of other label in projected area on each
slice.
‘--sum-other-in-slice’
[Projected label] Sum of other label in projected area on each
slice.
‘--sum-other-err-in-slice:’
[Projected label] Area in ‘--sum-other-in-slice’.
7.4.7 Invoking MakeCatalog
--------------------------
MakeCatalog will do measurements and produce a catalog from a labeled
dataset and optional values dataset(s). The executable name is
‘astmkcatalog’ with the following general template
$ astmkcatalog [OPTION ...] InputImage.fits
One line examples:
## Create catalog with RA, Dec, Magnitude and Magnitude error,
## from Segment's output:
$ astmkcatalog --ra --dec --magnitude seg-out.fits
## Same catalog as above (using short options):
$ astmkcatalog -rdm seg-out.fits
## Write the catalog to a text table:
$ astmkcatalog -rdm seg-out.fits --output=cat.txt
## Output columns specified in `columns.conf':
$ astmkcatalog --config=columns.conf seg-out.fits
## Use object and clump labels from a K-band image, but pixel values
## from an i-band image.
$ astmkcatalog K_segmented.fits --hdu=DETECTIONS --clumpscat \
--clumpsfile=K_segmented.fits --clumpshdu=CLUMPS \
--valuesfile=i_band.fits
If MakeCatalog is to do processing (not printing help or option values),
an input labeled image should be provided. The options described in
this section are those that are particular to MakeProfiles. For
operations that MakeProfiles shares with other programs (mainly
involving input/output or general processing steps), see *note Common
options::. Also see *note Common program behavior:: for some general
characteristics of all Gnuastro programs including MakeCatalog.
The various measurements/columns of MakeCatalog are requested as
options, either on the command-line or in configuration files, see *note
Configuration files::. The full list of available columns is available
in *note MakeCatalog measurements::. Depending on the requested
columns, MakeCatalog needs more than one input dataset, for more
details, please see *note MakeCatalog inputs and basic settings::. The
upper-limit measurements in particular need several configuration
options which are thoroughly discussed in *note Upper-limit settings::.
Finally, in *note MakeCatalog output:: the output file(s) created by
MakeCatalog are discussed.
7.4.7.1 MakeCatalog inputs and basic settings
.............................................
MakeCatalog works by using a localized/labeled dataset (see *note
MakeCatalog::). This dataset maps/labels pixels to a specific target
(row number in the final catalog) and is thus the only necessary input
dataset to produce a minimal catalog in any situation. Because it only
has labels/counters, it must have an integer type (see *note Numeric
data types::), see below if your labels are in a floating point
container. When the requested measurements only need this dataset (for
example, ‘--geo-x’, ‘--geo-y’, or ‘--geo-area’), MakeCatalog will not
read any more datasets.
Low-level measurements that only use the labeled image are rarely
sufficient for any high-level science case. Therefore necessary input
datasets depend on the requested columns in each run. For example,
let's assume you want the brightness/magnitude and signal-to-noise ratio
of your labeled regions. For these columns, you will also need to
provide an extra dataset containing values for every pixel of the
labeled input (to measure magnitude) and another for the Sky standard
deviation (to measure error). All such auxiliary input files have to
have the same size (number of pixels in each dimension) as the input
labeled image. Their numeric data type is irrelevant (they will be
converted to 32-bit floating point internally). For the full list of
available measurements, see *note MakeCatalog measurements::.
The "values" dataset is used for measurements like
brightness/magnitude, or flux-weighted positions. If it is a real
image, by default it is assumed to be already Sky-subtracted prior to
running MakeCatalog. If it is not, you use the ‘--subtractsky’ option
to, so MakeCatalog reads and subtracts the Sky dataset before any
processing. To obtain the Sky value, you can use the ‘--sky’ option of
*note Statistics::, but the best recommended method is *note
NoiseChisel::, see *note Sky value::.
MakeCatalog can also do measurements on sub-structures of detections.
In other words, it can produce two catalogs. Following the nomenclature
of Segment (see *note Segment::), the main labeled input dataset is
known as "object" labels and the (optional) sub-structure input dataset
is known as "clumps". If MakeCatalog is run with the ‘--clumpscat’
option, it will also need a labeled image containing clumps, similar to
what Segment produces (see *note Segment output::). Since clumps are
defined within detected regions (they exist over signal, not noise),
MakeCatalog uses their boundaries to subtract the level of signal under
them.
There are separate options to explicitly request a file name and
HDU/extension for each of the required input datasets as fully described
below (with the ‘--*file’ format). When each dataset is in a separate
file, these options are necessary. However, one great advantage of the
FITS file format (that is heavily used in astronomy) is that it allows
the storage of multiple datasets in one file. So in most situations
(for example, if you are using the outputs of *note NoiseChisel:: or
*note Segment::), all the necessary input datasets can be in one file.
When none of the ‘--*file’ options are given (for example
‘--clumpsfile’ or ‘--valuesfile’), MakeCatalog will assume the necessary
input datasets are available as HDUs in the file given as its argument
(without any option). When the Sky or Sky standard deviation datasets
are necessary and the only ‘--*file’ option called is ‘--valuesfile’,
MakeCatalog will search for these datasets (with the default/given HDUs)
in the file given to ‘--valuesfile’ (before looking into the main
argument file).
It may happen that your labeled objects image was created with a
program that only outputs floating point files. However, you know it
only has integer valued pixels that are stored in a floating point
container. In such cases, you can use Gnuastro's Arithmetic program
(see *note Arithmetic::) to change the numerical data type of the image
(‘float.fits’) to an integer type image (‘int.fits’) with a command like
below:
$ astarithmetic float.fits int32 --output=int.fits
To summarize: if the input file to MakeCatalog is the default/full
output of Segment (see *note Segment output::) you do not have to worry
about any of the ‘--*file’ options below. You can just give Segment's
output file to MakeCatalog as described in *note Invoking
astmkcatalog::. To feed NoiseChisel's output into MakeCatalog, just
change the labeled dataset's header (with ‘--hdu=DETECTIONS’). The full
list of input dataset options and general setting options are described
below.
‘-l FITS’
‘--clumpsfile=FITS’
The FITS file containing the labeled clumps dataset when
‘--clumpscat’ is called (see *note MakeCatalog output::). When
‘--clumpscat’ is called, but this option is not, MakeCatalog will
look into the main input file (given as an argument) for the
required extension/HDU (value to ‘--clumpshdu’).
‘--clumpshdu=STR’
The HDU/extension of the clump labels dataset. Only pixels with
values above zero will be considered. The clump labels dataset has
to be an integer data type (see *note Numeric data types::) and
only pixels with a value larger than zero will be used. See *note
Segment output:: for a description of the expected format.
‘-v FITS’
‘--valuesfile=FITS’
The file name of the (sky-subtracted) values dataset. When any of
the columns need values to associate with the input labels (for
example, to measure the sum of pixel values or magnitude of a
galaxy, see *note Brightness flux magnitude::), MakeCatalog will
look into a "values" for the respective pixel values. In most
common processing, this is the actual astronomical image that the
labels were defined, or detected, over. The HDU/extension of this
dataset in the given file can be specified with ‘--valueshdu’. If
this option is not called, MakeCatalog will look for the given
extension in the main input file.
‘--valueshdu=STR/INT’
The name or number (counting from zero) of the extension containing
the "values" dataset, see the descriptions above and those in
‘--valuesfile’ for more.
‘-s FITS/FLT’
‘--insky=FITS/FLT’
Sky value as a single number, or the file name containing a dataset
(different values per pixel or tile). The Sky dataset is only
necessary when ‘--subtractsky’ is called or when a column directly
related to the Sky value is requested (currently ‘--sky’). This
dataset may be a tessellation, with one element per tile (see
‘--oneelempertile’ of NoiseChisel's *note Processing options::).
When the Sky dataset is necessary but this option is not called,
MakeCatalog will assume it is an HDU/extension (specified by
‘--skyhdu’) in one of the already given files. First it will look
for it in the ‘--valuesfile’ (if it is given) and then the main
input file (given as an argument).
By default the values dataset is assumed to be already Sky
subtracted, so this dataset is not necessary for many of the
columns.
‘--skyhdu=STR’
HDU/extension of the Sky dataset, see ‘--skyfile’.
‘--subtractsky’
Subtract the sky value or dataset from the values file prior to any
processing.
‘-t STR/FLT’
‘--instd=STR/FLT’
Sky standard deviation value as a single number, or the file name
containing a dataset (different values per pixel or tile). With
the ‘--variance’ option you can tell MakeCatalog to interpret this
value/dataset as a variance image, not standard deviation.
*Important note:* This must only be the SKY standard deviation or
variance (not including the signal's contribution to the error).
In other words, the final standard deviation of a pixel depends on
how much signal there is in it. MakeCatalog will find the amount
of signal within each pixel (while subtracting the Sky, if
‘--subtractsky’ is called) and account for the extra error due to
it's value (signal). Therefore if the input standard deviation (or
variance) image also contains the contribution of signal to the
error, then the final error measurements will be over-estimated.
‘--stdhdu=STR’
The HDU of the Sky value standard deviation image.
‘--variance’
The dataset given to ‘--instd’ (and ‘--stdhdu’ has the Sky variance
of every pixel, not the Sky standard deviation.
‘--forcereadstd’
Read the input STD image even if it is not required by any of the
requested columns. This is because some of the output catalog's
metadata may need it, for example, to calculate the dataset's
surface brightness limit (see *note Quantifying measurement
limits::, configured with ‘--sfmagarea’ and ‘--sfmagnsigma’ in
*note MakeCatalog output::).
Furthermore, if the input STD image does not have the ‘MEDSTD’
keyword (that is meant to contain the representative standard
deviation of the full image), with this option, the median will be
calculated and used for the surface brightness limit.
‘-z FLT’
‘--zeropoint=FLT’
The zero point magnitude for the input image, see *note Brightness
flux magnitude::.
‘--sigmaclip FLT,FLT’
The sigma-clipping parameters when any of the sigma-clipping
related columns are requested (for example, ‘--sigclip-median’ or
‘--sigclip-number’).
This option takes two values: the first is the multiple of
$\sigma$, and the second is the termination criteria. If the
latter is larger than 1, it is read as an integer number and will
be the number of times to clip. If it is smaller than 1, it is
interpreted as the tolerance level to stop clipping. See *note
Sigma clipping:: for a complete explanation.
‘--frac-max=FLT[,FLT]’
The fractions (one or two) of maximum value in objects or clumps to
be used in the related columns, for example, ‘--frac-max1-area’,
‘--frac-max1-sum’ or ‘--frac-max1-radius’, see *note MakeCatalog
measurements::. For the maximum value, see the description of
‘--maximum’ column below. The value(s) of this option must be
larger than 0 and smaller than 1 (they are a fraction). When only
‘--frac-max1-area’ or ‘--frac-max1-sum’ is requested, one value
must be given to this option, but if ‘--frac-max2-area’ or
‘--frac-max2-sum’ are also requested, two values must be given to
this option. The values can be written as simple floating point
numbers, or as fractions, for example, ‘0.25,0.75’ and ‘0.25,3/4’
are the same.
‘--spatialresolution=FLT’
The error in measuring spatial properties (for example, the area)
in units of pixels. You can think of this as the FWHM of the
dataset's PSF and is used in measurements like the error in surface
brightness (‘--sb-error’, see *note MakeCatalog measurements::).
Ideally, images are taken in the optimal Nyquist sampling *note
Sampling theorem::, so the default value for this option is 2. But
in practice real images my be over-sampled (usually ground-based
images, where you will need to increase the default value) or
undersampled (some space-based images, where you will need to
decrease the default value).
‘--inbetweenints’
Output will contain one row for all integers between 1 and the
largest label in the input (irrespective of their existance in the
input image). By default, MakeCatalog's output will only contain
rows with integers that actually corresponded to at least one pixel
in the input dataset.
For example, if the input's only labeled pixel values are 11 and
13, MakeCatalog's default output will only have two rows. If you
use this option, it will have 13 rows and all the columns
corresponding to integer identifiers that did not correspond to any
pixel will be 0 or NaN (depending on context).
7.4.7.2 Upper-limit settings
............................
The upper-limit magnitude was discussed in *note Quantifying measurement
limits::. Unlike other measured values/columns in MakeCatalog, the
upper limit magnitude needs several extra parameters which are discussed
here. All the options specific to the upper-limit measurements start
with ‘up’ for "upper-limit". The only exception is ‘--envseed’ that is
also present in other programs and is general for any job requiring
random number generation in Gnuastro (see *note Generating random
numbers::).
One very important consideration in Gnuastro is reproducibility.
Therefore, the values to all of these parameters along with others (like
the random number generator type and seed) are also reported in the
comments of the final catalog when the upper limit magnitude column is
desired. The random seed that is used to define the random positions
for each object or clump is unique and set based on the (optionally)
given seed, the total number of objects and clumps and also the labels
of the clumps and objects. So with identical inputs, an identical
upper-limit magnitude will be found. However, even if the seed is
identical, when the ordering of the object/clump labels differs between
different runs, the result of upper-limit measurements will not be
identical.
MakeCatalog will randomly place the object/clump footprint over the
dataset. When the randomly placed footprint does not fall on any object
or masked region (see ‘--upmaskfile’) it will be used in the final
distribution. Otherwise that particular random position will be ignored
and another random position will be generated. Finally, when the
distribution has the desired number of successfully measured random
samples (‘--upnum’) the distribution's properties will be measured and
placed in the catalog.
When the profile is very large or the image is significantly covered
by detections, it might not be possible to find the desired number of
samplings in a reasonable time. MakeProfiles will continue searching
until it is unable to find a successful position (since the last
successful measurement(1)), for a large multiple of ‘--upnum’
(currently(2) this is 10). If ‘--upnum’ successful samples cannot be
found until this limit is reached, MakeCatalog will set the upper-limit
magnitude for that object to NaN (blank).
MakeCatalog will also print a warning if the range of positions
available for the labeled region is smaller than double the size of the
region. In such cases, the limited range of random positions can
artificially decrease the standard deviation of the final distribution.
If your dataset can allow it (it is large enough), it is recommended to
use a larger range if you see such warnings.
‘--upmaskfile=FITS’
File name of mask image to use for upper-limit calculation. In
some cases (especially when doing matched photometry), the object
labels specified in the main input and mask image might not be
adequate. In other words they do not necessarily have to cover
_all_ detected objects: the user might have selected only a few of
the objects in their labeled image. This option can be used to
ignore regions in the image in these situations when estimating the
upper-limit magnitude. All the non-zero pixels of the image
specified by this option (in the ‘--upmaskhdu’ extension) will be
ignored in the upper-limit magnitude measurements.
For example, when you are using labels from another image, you can
give NoiseChisel's objects image output for this image as the value
to this option. In this way, you can be sure that regions with
data do not harm your distribution. See *note Quantifying
measurement limits:: for more on the upper limit magnitude.
‘--upmaskhdu=STR’
The extension in the file specified by ‘--upmask’.
‘--upnum=INT’
The number of random samples to take for all the objects. A larger
value to this option will give a more accurate result
(asymptotically), but it will also slow down the process. When a
randomly positioned sample overlaps with a detected/masked pixel it
is not counted and another random position is found until the
object completely lies over an undetected region. So you can be
sure that for each object, this many samples over undetected
objects are made. See the upper limit magnitude discussion in
*note Quantifying measurement limits:: for more.
‘--uprange=INT,INT’
The range/width of the region (in pixels) to do random sampling
along each dimension of the input image around each object's
position. This is not a mandatory option and if not given (or
given a value of zero in a dimension), the full possible range of
the dataset along that dimension will be used. This is useful when
the noise properties of the dataset vary gradually. In such cases,
using the full range of the input dataset is going to bias the
result. However, note that decreasing the range of available
positions too much will also artificially decrease the standard
deviation of the final distribution (and thus bias the upper-limit
measurement).
‘--envseed’
Read the random number generator type and seed value from the
environment (see *note Generating random numbers::). Random
numbers are used in calculating the random positions of different
samples of each object.
‘--upsigmaclip=FLT,FLT’
The raw distribution of random values will not be used to find the
upper-limit magnitude, it will first be $\sigma$-clipped (see *note
Sigma clipping::) to avoid outliers in the distribution (mainly the
faint undetected wings of bright/large objects in the image). This
option takes two values: the first is the multiple of $\sigma$, and
the second is the termination criteria. If the latter is larger
than 1, it is read as an integer number and will be the number of
times to clip. If it is smaller than 1, it is interpreted as the
tolerance level to stop clipping. See *note Sigma clipping:: for a
complete explanation.
‘--upnsigma=FLT’
The multiple of the final ($\sigma$-clipped) standard deviation (or
$\sigma$) used to measure the upper-limit sum or magnitude.
‘--checkuplim=INT[,INT]’
Print a table of positions and measured values for all the full
random distribution used for one particular object or clump. If
only one integer is given to this option, it is interpreted to be
an object's label. If two values are given, the first is the
object label and the second is the ID of requested clump within it.
The output is a table with three columns (whether it is FITS or
plain-text is determined with the ‘--tableformat’ option, see *note
Input output options::). The first two columns are the pixel X,Y
positions of the center of each label's tile (see next paragraph),
in each random sampling of this particular object/clump. The third
column is the measured flux over that region. If the region
overlapped with a detection or masked pixel, then its measured
value will be a NaN (not-a-number). The total number of rows is
thus unknown before running. However, if an upper-limit
measurement was made in the main output of MakeCatalog, you can be
sure that the number of rows with non-NaN measurements is the
number given to the ‘--upnum’ option.
The "tile" of each label is defined by the minimum and maximum
positions of each label: values of the ‘--min-x’, ‘--max-x’,
‘--min-y’ and ‘--max-y’ columns in the main output table for each
label. Therefore, the tile center position that is recorded in the
output of this column ignores the distribution of labeled pixels
within the tile.
Precise interpretation of the position is only relevant when the
footprint of your label is highly un-symmetrical and you want to
use this catalog to insert your object into the image. In such a
case, you can also ask for ‘--min-x’ and ‘--min-y’ and manually
calculate their difference with the following two positional
measurements of your desired label: ‘--geo-x’ and ‘--geo-y’ (which
report the label's "geometric" center; only using the label
positions ignoring any "values") or ‘--x’ and ‘--y’ (which report
the value-weighted center of the label). Adding the difference
with the position reported by this column, will let you define
alternative "center"s for your label in particular situations (this
will usually not be necessary!). For more on these positional
columns, see *note Position measurements in pixels::.
---------- Footnotes ----------
(1) The counting of failed positions restarts on every successful
measurement.
(2) In Gnuastro's source, this constant number is defined as the
‘MKCATALOG_UPPERLIMIT_MAXFAILS_MULTIP’ macro in ‘bin/mkcatalog/main.h’,
see *note Downloading the source::.
7.4.7.3 MakeCatalog output
..........................
After it has completed all the requested measurements (see *note
MakeCatalog measurements::), MakeCatalog will store its measurements in
table(s). If an output filename is given (see ‘--output’ in *note Input
output options::), the format of the table will be deduced from the
name. When it is not given, the input name will be appended with a
‘_cat’ suffix (see *note Automatic output::) and its format will be
determined from the ‘--tableformat’ option, which is also discussed in
*note Input output options::. ‘--tableformat’ is also necessary when
the requested output name is a FITS table (recall that FITS can accept
ASCII and binary tables, see *note Table::).
By default (when ‘--spectrum’ or ‘--clumpscat’ are not called) only a
single catalog/table will be created for the labeled "objects".
• if ‘--clumpscat’ is called, a secondary catalog/table will also be
created for "clumps" (one of the outputs of the Segment program,
for more on "objects" and "clumps", see *note Segment::). In
short, if you only have one labeled image, you do not have to worry
about clumps and just ignore this.
• When ‘--spectrum’ is called, it is not mandatory to specify any
single-valued measurement columns. In this case, the output will
only be the spectra of each labeled region within a 3D data cube.
For more, see the description of ‘--spectrum’ in *note MakeCatalog
measurements::.
When possible, MakeCatalog will also measure the full input's noise
level (also known as surface brightness limit, see *note Quantifying
measurement limits::). Since these measurements are related to the
noise and not any particular labeled object, they are stored as keywords
in the output table. Furthermore, they are only possible when a
standard deviation image has been loaded (done automatically for any
column measurement that involves noise, for example, ‘--sn’,
‘--magnitude-error’ or ‘--sky-std’). But if you just want the surface
brightness limit and no noise-related column, you can use
‘--forcereadstd’. All these keywords start with ‘SBL’ (for "surface
brightness limit") and are described below:
‘SBLSTD’
Per-pixel standard deviation. If a ‘MEDSTD’ keyword exists in the
standard deviation dataset, then that value is directly used.
‘SBLNSIG’
Sigma multiple for surface brightness limit (value you gave to
‘--sfmagnsigma’), used for ‘SBLMAGPX’ and ‘SBLMAG’.
‘SBLMAGPX’
Per-pixel surface brightness limit (in units of magnitudes/pixel).
‘SBLAREA’
Area (in units of arcsec$^2$) used in ‘SBLMAG’ (value you gave to
‘--sfmagarea’).
‘SBLMAG’
Surface brightness limit of data calculated over ‘SBLAREA’ (in
units of mag/arcsec$^2$).
When any of the upper-limit measurements are requested, the input
parameters for the upper-limit measurement are stored in the keywords
starting with ‘UP’: ‘UPNSIGMA’, ‘UPNUMBER’, ‘UPRNGNAM’, ‘UPRNGSEE’,
‘UPSCMLTP’, ‘UPSCTOL’. These are primarily input arguments, so they
correspond to the options with a similar name.
The full list of MakeCatalog's options relating to the output file
format and keywords are listed below. See *note MakeCatalog
measurements:: for specifying which columns you want in the final
catalog.
‘-C’
‘--clumpscat’
Do measurements on clumps and produce a second catalog (only
devoted to clumps). When this option is given, MakeCatalog will
also look for a secondary labeled dataset (identifying
substructure) and produce a catalog from that. For more on the
definition on "clumps", see *note Segment::.
When the output is a FITS file, the objects and clumps
catalogs/tables will be stored as multiple extensions of one FITS
file. You can use *note Table:: to inspect the column meta-data
and contents in this case. However, in plain text format (see
*note Gnuastro text table format::), it is only possible to keep
one table per file. Therefore, if the output is a text file, two
output files will be created, ending in ‘_o.txt’ (for objects) and
‘_c.txt’ (for clumps).
‘--noclumpsort’
Do not sort the clumps catalog based on object ID (only relevant
with ‘--clumpscat’). This option will benefit the performance(1)
of MakeCatalog when it is run on multiple threads _and_ the
position of the rows in the clumps catalog is irrelevant (for
example, you just want the number-counts).
MakeCatalog does all its measurements on each _object_
independently and in parallel. As a result, while it is writing
the measurements on each object's clumps, it does not know how many
clumps there were in previous objects. Each thread will just fetch
the first available row and write the information of clumps (in
order) starting from that row. After all the measurements are
done, by default (when this option is not called), MakeCatalog will
reorder/permute the clumps catalog to have both the object and
clump ID in an ascending order.
If you would like to order the catalog later (when it is a plain
text file), you can run the following command to sort the rows by
object ID (and clump ID within each object), assuming they are
respectively the first and second columns:
$ awk '!/^#/' out_c.txt | sort -g -k1,1 -k2,2
‘--sfmagnsigma=FLT’
Value to multiply with the median standard deviation (from a
‘MEDSTD’ keyword in the Sky standard deviation image) for
estimating the surface brightness limit. Note that the surface
brightness limit is only reported when a standard deviation image
is read, in other words a column using it is requested (for
example, ‘--sn’) or ‘--forcereadstd’ is called.
This value is a per-pixel value, not per object/clump and is not
found over an area or aperture, like the common $5\sigma$ values
that are commonly reported as a measure of depth or the upper-limit
measurements (see *note Quantifying measurement limits::).
‘--sfmagarea=FLT’
Area (in arc-seconds squared) to convert the per-pixel estimation
of ‘--sfmagnsigma’ in the comments section of the output tables.
Note that the surface brightness limit is only reported when a
standard deviation image is read, in other words a column using it
is requested (for example, ‘--sn’) or ‘--forcereadstd’ is called.
Note that this is just a unit conversion using the World Coordinate
System (WCS) information in the input's header. It does not
actually do any measurements on this area. For random measurements
on any area, please use the upper-limit columns of MakeCatalog (see
the discussion on upper-limit measurements in *note Quantifying
measurement limits::).
---------- Footnotes ----------
(1) The performance boost due to ‘--noclumpsort’ can only be felt
when there are a huge number of objects. Therefore, by default the
output is sorted to avoid miss-understandings or bugs in the user's
scripts when the user forgets to sort the outputs.
7.5 Match
=========
Data can come come from different telescopes, filters, software and even
different configurations for a single software. As a result, one of the
primary things to do after generating catalogs from each of these
sources (for example, with *note MakeCatalog::), is to find which
sources in one catalog correspond to which in the other(s). In other
words, to 'match' the two catalogs with each other.
Gnuastro's Match program is in charge of such operations. The
nearest objects in the two catalogs, within the given aperture, will be
found and given as output. The aperture can be a circle or an ellipse
with any orientation.
7.5.1 Matching algorithms
-------------------------
Matching involves two catalogs, let's call them catalog A (with N rows)
and catalog B (with M rows). The most basic matching algorithm that
immediately comes to mind is this: for each row in A (let's call it
$A_i$), go over all the rows in B ($B_j$, where $0j$). Therefore, as you go down B (and more matches are
found), you have to calculate less distances (there are fewer elements
in A that remain to be checked). However, this will introduce an
important bias: $A_i$ may actually be closer to $B_k$ than to $B_j$!
But because $B_j$ happened to be before $B_k$ in your table, $A_i$ was
removed from the potential search domain of $B_k$. The good match
($B_k$ with $A_i$ will therefore be lost, and replaced by a false match
between $B_j$ and $A_i$!
In a single-dimensional match, this bias depends on the sorting of
your two datasets (leading to different matches if you shuffle your
datasets). But it will get more complex as you add dimensionality. For
example, catalogs derived from 2D images or 3D cubes, where you have 2
and 3 different coordinates for each point.
To address this problem, in Gnuastro (the Match program, or the
matching functions of the library) similar to above, we first parse over
the elements of B. But we will not associate the first nearest-neighbor
with a match! Instead, we will use an array (with the number of rows in
A, let's call it "B-in-A") to keep the list of all nearest element(s) in
B that match each A-point. Once all the points in B are parsed, each
A-point in B-in-A will (possibly) have a sorted list of B-points (there
may be multiple B-points that fall within the acceptable aperture of
each A-point). In the previous example, the $i$ element (corresponding
to $A_i$) of B-in-A will contain the following list of B-points: $B_j$
and $B_k$.
A new array (with the number of points in B, let's call it A-in-B) is
then used to find the final match. We parse over B-in-A (that was
completed above), and from it extract the nearest B-point to each
A-point ($B_k$ for $A_i$ in the example above). If this is the first
A-point that is found for this B-point, then we put this A-point into
A-in-B (in the example above, element $k$ is filled with $A_k$). If
another A-point was previously found for this B-point, then the distance
of the two A-points to that B-point are compared, and the A-point with
the smaller distance is kept in A-in-B. This will give us the best match
between the two catalogs, independent of any sorting issues. Both the
B-in-A and A-in-B will also keep the distances, so distances are only
measured once.
In summary, here are the points to consider when selecting an algorithm,
or the order of your inputs (for optimal speed, the match will be the
same):
• For larger datasets, the k-d tree based method (when running on all
threads possible) is much more faster than the classical sort-based
method.
• The k-d tree is constructed for the first input table and the
multi-threading is done on the rows of the second input table. The
construction of a larger dataset's k-d tree will take longer, but
multi-threading will work better when you have more rows. As a
result, the optimal way to place your inputs is to give the smaller
input table (with fewer rows) as the first argument (so its k-d
tree is constructed), and the larger table as the second argument
(so its rows are checked in parallel).
• If you always need to match against one catalog (that is large!),
the k-d tree construction itself can take a significant fraction of
the running time. Therefore you can save its k-d tree into a file
and simply give it to later calls, like the example given in the
description of the k-d algorithm mentioned above.
7.5.2 Invoking Match
--------------------
When given two catalogs, Match finds the rows that are nearest to each
other within an input aperture. The executable name is ‘astmatch’ with
the following general template
$ astmatch [OPTION ...] input-1 input-2
One line examples:
## 1D wavelength match (within 5 angstroms) of the two inputs.
## The wavelengths are in the 5th and 10th columns respectively.
$ astmatch --aperture=5e-10 --ccol1=5 --ccol2=10 in1.fits in2.txt
## Find the row that is closest to (RA,DEC) of (12.3456,6.7890)
## with a maximum distance of 1 arcseconds (1/3600 degrees).
## The coordinates can also be given in sexagesimal.
$ astmatch input1.txt --ccol1=ra,dec --coord=12.3456,6.7890 \
--aperture=1/3600
## Find matching rows of two catalogs with a circular aperture
## of width 2 (same unit as position columns: pixels in this case).
$ astmatch input1.txt input2.fits --aperture=2 \
--ccol1=X,Y --ccol2=IMG_X,IMG_Y
## Similar to before, but the output is created by merging various
## columns from the two inputs: columns 1, RA, DEC from the first
## input, followed by all columns starting with `MAG' and the `BRG'
## column from second input and the 10th column from first input.
$ astmatch input1.txt input2.fits --aperture=1/3600 \
--ccol1=ra,dec --ccol2=RAJ2000,DEJ2000 \
--outcols=a1,aRA,aDEC,b/^MAG/,bBRG,a10
## Assuming both inputs have the same column metadata (same name
## and numeric type), the output will contain all the rows of the
## first input, appended with the non-matching rows of the second
## input (good when you need to merge multiple catalogs that
## may have matching items, which you do not want to repeat).
$ astmatch input1.fits input2.fits --ccol1=RA,DEC --ccol2=RA,DEC \
--aperture=1/3600 --notmatched --outcols=_all
## Match the two catalogs within an elliptical aperture of 1 and 2
## arc-seconds along RA and Dec respectively.
$ astmatch --aperture=1/3600,2/3600 in1.fits in2.txt
## Match the RA and DEC columns of the first input with the RA_D
## and DEC_D columns of the second within a 0.5 arcseconds aperture.
$ astmatch --ccol1=RA,DEC --ccol2=RA_D,DEC_D --aperture=0.5/3600 \
in1.fits in2.fits
## Match in 3D (RA, Dec and Wavelength).
$ astmatch --ccol1=2,3,4 --ccol2=2,3,4 -a0.5/3600,0.5/3600,5e-10 \
in1.fits in2.txt
Match will find the rows that are nearest to each other in two
catalogs (given some coordinate columns). Alternatively, it can
construct the k-d tree of one catalog to save in a FITS file for future
matching of the same catalog with many others. To understand the inner
working of Match and its algorithms, see *note Matching algorithms::.
When matching, two catalogs are necessary for input. But for
constructing a k-d tree, only a single catalog should be given. The
input tables can be plain text tables or FITS tables, for more see *note
Tables::. But other ways of feeding inputs area also supported:
• The _first_ catalog can also come from the standard input (for
example, a pipe that feeds the output of a previous command to
Match, see *note Standard input::);
• When you only want to match one point with another catalog, you can
use the ‘--coord’ option to avoid creating a file for the _second_
input catalog.
Match follows the same basic behavior of all Gnuastro programs as
fully described in *note Common program behavior::. If the first input
is a FITS file, the common ‘--hdu’ option (see *note Input output
options::) should be used to identify the extension. When the second
input is FITS, the extension must be specified with ‘--hdu2’.
When ‘--quiet’ is not called, Match will print its various processing
phases (including the number of matches found) in standard output (on
the command-line). When matches are found, by default, two tables will
be output (if in FITS format, as two HDUs). Each output table will
contain the re-arranged rows of the respective input table. In other
words, both tables will have the same number of rows, and row N in both
corresponds to the 10th match between the two. If no matches are found,
the columns of the output table(s) will have zero rows (with proper
meta-data). The output format can be changed with the following
options:
• ‘--outcols’: The output will be a single table with rows chosen
from either of the two inputs in any order.
• ‘--notmatched’: The output tables will contain the rows that did
not match between the two tables. If called with ‘--outcols’, the
output will be a single table with all non-matched rows of both
tables.
• ‘--logasoutput’: The output will be a single table with the
contents of the log file, see below.
If no output file name is given with the ‘--output’ option, then
automatic output *note Automatic output:: will be used to determine the
output name(s). Depending on ‘--tableformat’ (see *note Input output
options::), the output will be a (possibly multi-extension) FITS file or
(possibly two) plain text file(s). Generally, giving a filename to
‘--output’ is recommended.
When the ‘--log’ option is called (see *note Operating mode
options::), and there was a match, Match will also create a file named
‘astmatch.fits’ (or ‘astmatch.txt’, depending on ‘--tableformat’, see
*note Input output options::) in the directory it is run in. This log
table will have three columns. The first and second columns show the
matching row/record number (counting from 1) of the first and second
input catalogs respectively. The third column is the distance between
the two matched positions. The units of the distance are the same as
the given coordinates (given the possible ellipticity, see description
of ‘--aperture’ below). When ‘--logasoutput’ is called, no log file
(with a fixed name) will be created. In this case, the output file
(possibly given by the ‘--output’ option) will have the contents of this
log file.
*‘--log’ is not thread-safe*: As described above, when ‘--logasoutput’
is not called, the Log file has a fixed name for all calls to Match.
Therefore if a separate log is requested in two simultaneous calls to
Match in the same directory, Match will try to write to the same file.
This will cause problems like unreasonable log file, undefined behavior,
or a crash. Remember that ‘--log’ is mainly intended for debugging
purposes, if you want the log file with a specific name, simply use
‘--logasoutput’ (which will also be faster, since no arranging of the
input columns is necessary).
‘-H STR’
‘--hdu2=STR’
The extension/HDU of the second input if it is a FITS file. When
it is not a FITS file, this option's value is ignored. For the
first input, the common option ‘--hdu’ must be used.
‘-k STR’
‘--kdtree=STR’
Select the algorithm and/or the way to construct or import the k-d
tree. A summary of the four acceptable strings for this option are
described here for completeness. However, for a much more detailed
discussion on Match's algorithms with examples, see *note Matching
algorithms::.
‘internal’
Construct a k-d tree for the first input internally (within
the same run of Match), and parallelize over the rows of the
second to find the nearest points. This is the default
algorithm/method used by Match (when this option is not
called).
‘build’
Only construct a k-d tree of a single input and abort. The
name of the k-d tree is value to ‘--output’.
‘CUSTOM-FITS-FILE’
Use the given FITS file as a k-d tree (that was previously
constructed with Match itself) of the first input, and do not
construct any k-d tree internally. The FITS file should have
two columns with an unsigned 32-bit integer data type and a
‘KDTROOT’ keyword that contains the index of the root of the
k-d tree. For more on Gnuastro's k-d tree format, see *note
K-d tree::.
‘disable’
Do not use the k-d tree algorithm for finding the nearest
neighbor, instead, use the sort-based method.
‘--kdtreehdu=STR’
The HDU of the FITS file, when a FITS file is given to the
‘--kdtree’ option that was described above.
‘--outcols=STR[,STR,[...]]’
Columns (from both inputs) to write into a single matched table
output. The value to ‘--outcols’ must be a comma-separated list of
column identifiers (number or name, see *note Selecting table
columns::). The expected format depends on ‘--notmatched’ and
explained below. By default (when ‘--nomatched’ is not called),
the number of rows in the output will be equal to the number of
matches. However, when ‘--notmatched’ is called, all the rows
(from the requested columns) of the first input are placed in the
output, and the not-matched rows of the second input are inserted
afterwards (useful when you want to merge unique entries of
multiple catalogs into one).
Default (only matching rows)
The first character of each string specifies the input
catalog: ‘a’ for the first and ‘b’ for the second. The rest
of the characters of the string will be directly used to
identify the proper column(s) in the respective table. See
*note Selecting table columns:: for how columns can be
specified in Gnuastro.
For example, the output of ‘--outcols=a1,bRA,bDEC’ will have
three columns: the first column of the first input, along with
the ‘RA’ and ‘DEC’ columns of the second input.
If the string after ‘a’ or ‘b’ is ‘_all’, then all the columns
of the respective input file will be written in the output.
For example, the command below will print all the input
columns from the first catalog along with the 5th column from
the second:
$ astmatch a.fits b.fits --outcols=a_all,b5
‘_all’ can be used multiple times, possibly on both inputs.
Tip: if an input's column is called ‘_all’ (an unlikely name!)
and you do not want all the columns from that table the
output, use its column number to avoid confusion.
Another example is given in the one-line examples above.
Compared to the default case (where two tables with all their
columns) are saved separately, using this option is much
faster: it will only read and re-arrange the necessary columns
and it will write a single output table. Combined with
regular expressions in large tables, this can be a very
powerful and convenient way to merge various tables into one.
When ‘--coord’ is given, no second catalog will be read. The
second catalog will be created internally based on the values
given to ‘--coord’. So column names are not defined and you
can only request integer column numbers that are less than the
number of coordinates given to ‘--coord’. For example, if you
want to find the row matching RA of 1.2345 and Dec of 6.7890,
then you should use ‘--coord=1.2345,6.7890’. But when using
‘--outcols’, you cannot give ‘bRA’, or ‘b25’.
With ‘--notmatched’
Only the column names/numbers should be given (for example,
‘--outcols=RA,DEC,MAGNITUDE’). It is assumed that both input
tables have the requested column(s) and that the numerical
data types of each column in each input (with same name) is
the same as the corresponding column in the other. Therefore
if one input has a ‘MAGNITUDE’ column with a 32-bit floating
point type, but the ‘MAGNITUDE’ column of the other is 64-bit
floating point, Match will crash with an error. The metadata
of the columns will come from the first input.
As an example, let's assume ‘input1.txt’ and ‘input2.fits’
each have a different number of columns and rows. However,
they both have the ‘RA’ (64-bit floating point), ‘DEC’ (64-bit
floating point) and ‘MAGNITUDE’ (32-bit floating point)
columns. If ‘input1.txt’ has 100 rows and ‘input2.fits’ has
300 rows (such that 50 of them match within 1 arcsec of the
first), then the output of the command above will have
$100+(300-50)=350$ rows and only three columns. Other columns
in each catalog, which may be different, are ignored.
$ astmatch input1.txt --ccol1=RA,DEC \
input2.fits --ccol2=RA,DEC \
--aperture=1/3600 \
--notmatched --outcols=RA,DEC,MAGNITUDE
‘-l’
‘--logasoutput’
The output file will have the contents of the log file: indexes in
the two catalogs that match with each other along with their
distance, see description of the log file above.
When this option is called, a separate log file will not be created
and the output will not contain any of the input columns (either as
two tables containing the re-arranged columns of each input, or a
single table mixing columns), only their indices in the log format.
‘--notmatched’
Write the non-matching rows into the outputs, not the matched ones.
By default, this will produce two output tables, that will not
necessarily have the same number of rows. However, when called
with ‘--outcols’, it is possible to import non-matching rows of the
second into the first. See the description of ‘--outcols’ for
more.
‘-c INT/STR[,INT/STR]’
‘--ccol1=INT/STR[,INT/STR]’
The coordinate columns of the first input. The number of
dimensions for the match is determined by the number of
comma-separated values given to this option. The values can be the
column number (counting from 1), exact column name or a regular
expression. For more, see *note Selecting table columns::. See
the one-line examples above for some usages of this option.
‘-C INT/STR[,INT/STR]’
‘--ccol2=INT/STR[,INT/STR]’
The coordinate columns of the second input. See the example in
‘--ccol1’ for more.
‘-d FLT[,FLT]’
‘--coord=FLT[,FLT]’
Manually specify the coordinates to match against the given
catalog. With this option, Match will not look for a second input
file/table and will directly use the coordinates given to this
option. When the coordinates are RA and Dec, the comma-separated
values can either be in degrees (a single number), or sexagesimal
(‘_h_m_’ for RA, ‘_d_m_’ for Dec, or ‘_:_:_’ for both).
When this option is called, the output changes in the following
ways: 1) when ‘--outcols’ is specified, for the second input, it
can only accept integer numbers that are less than the number of
values given to this option, see description of that option for
more. 2) By default (when ‘--outcols’ is not used), only the
matching row of the first table will be output (a single file), not
two separate files (one for each table).
This option is good when you have a (large) catalog and only want
to match a single coordinate to it (for example, to find the
nearest catalog entry to your desired point). With this option,
you can write the coordinates on the command-line and thus avoid
the need to make a single-row file.
‘-a FLT[,FLT[,FLT]]’
‘--aperture=FLT[,FLT[,FLT]]’
Parameters of the aperture for matching. The values given to this
option can be fractions, for example, when the position columns are
in units of degrees, ‘1/3600’ can be used to ask for one
arc-second. The interpretation of the values depends on the
requested dimensions (determined from ‘--ccol1’ and ‘--ccol2’) and
how many values are given to this option.
When multiple objects are found within the aperture, the match is
defined as the nearest one. In a multi-dimensional dataset, when
the aperture is a general ellipse or ellipsoid (and not a circle or
sphere), the distance is calculated in the elliptical space along
the major axis. For the defintion of this distance, see $r_{el}$
in *note Defining an ellipse and ellipsoid::.
1D match
The aperture/interval can only take one value: half of the
interval around each point (maximum distance from each point).
2D match
In a 2D match, the aperture can be a circle, an ellipse
aligned in the axes or an ellipse with a rotated major axis.
To simply the usage, you can determine the shape based on the
number of free parameters for each.
1 number
for example, ‘--aperture=2’. The aperture will be a
circle of the given radius. The value will be in the
same units as the columns in ‘--ccol1’ and ‘--ccol2’).
2 numbers
for example, ‘--aperture=3,4e-10’. The aperture will be
an ellipse (if the two numbers are different) with the
respective value along each dimension. The numbers are
in units of the first and second axis. In the example
above, the semi-axis value along the first axis will be 3
(in units of the first coordinate) and along the second
axis will be $4\times10^{-10}$ (in units of the second
coordinate). Such values can happen if you are comparing
catalogs of a spectra for example. If more than one
object exists in the aperture, the nearest will be found
along the major axis as described in *note Defining an
ellipse and ellipsoid::.
3 numbers
for example, ‘--aperture=2,0.6,30’. The aperture will be
an ellipse (if the second value is not 1). The first
number is the semi-major axis, the second is the axis
ratio and the third is the position angle (in degrees).
If multiple matches are found within the ellipse, the
distance (to find the nearest) is calculated along the
major axis in the elliptical space, see *note Defining an
ellipse and ellipsoid::.
3D match
The aperture (matching volume) can be a sphere, an ellipsoid
aligned on the three axes or a genenral ellipsoid rotated in
any direction. To simplifythe usage, the shape can be
determined based on the number of values given to this option.
1 number
for example, ‘--aperture=3’. The matching volume will be
a sphere of the given radius. The value is in the same
units as the input coordinates.
3 numbers
for example, ‘--aperture=4,5,6e-10’. The aperture will
be a general ellipsoid with the respective extent along
each dimension. The numbers must be in the same units as
each axis. This is very similar to the two number case
of 2D inputs. See there for more.
6 numbers
for example, ‘--aperture=4,0.5,0.6,10,20,30’. The
numbers represent the full general ellipsoid definition
(in any orientation). For the definition of a general
ellipsoid, see *note Defining an ellipse and ellipsoid::.
The first number is the semi-major axis. The second and
third are the two axis ratios. The last three are the
three Euler angles in units of degrees in the ZXZ order
as fully described in *note Defining an ellipse and
ellipsoid::.
8 Data modeling
***************
In order to fully understand observations after initial analysis on the
image, it is very important to compare them with the existing models to
be able to further understand both the models and the data. The tools
in this chapter create model galaxies and will provide 2D fittings to be
able to understand the detections.
8.1 MakeProfiles
================
MakeProfiles will create mock astronomical profiles from a catalog,
either individually or together in one output image. In data analysis,
making a mock image can act like a calibration tool, through which you
can test how successfully your detection technique is able to detect a
known set of objects. There are commonly two aspects to detecting: the
detection of the fainter parts of bright objects (which in the case of
galaxies fade into the noise very slowly) or the complete detection of
an over-all faint object. Making mock galaxies is the most accurate
(and idealistic) way these two aspects of a detection algorithm can be
tested. You also need mock profiles in fitting known functional
profiles with observations.
MakeProfiles was initially built for extra galactic studies, so
currently the only astronomical objects it can produce are stars and
galaxies. We welcome the simulation of any other astronomical object.
The general outline of the steps that MakeProfiles takes are the
following:
1. Build the full profile out to its truncation radius in a possibly
over-sampled array.
2. Multiply all the elements by a fixed constant so its total
magnitude equals the desired total magnitude.
3. If ‘--individual’ is called, save the array for each profile to a
FITS file.
4. If ‘--nomerged’ is not called, add the overlapping pixels of all
the created profiles to the output image and abort.
Using input values, MakeProfiles adds the World Coordinate System
(WCS) headers of the FITS standard to all its outputs (except PSF
images!). For a simple test on a set of mock galaxies in one image,
there is no need for the third step or the WCS information.
However in complicated simulations like weak lensing simulations,
where each galaxy undergoes various types of individual transformations
based on their position, those transformations can be applied to the
different individual images with other programs. After all the
transformations are applied, using the WCS information in each
individual profile image, they can be merged into one output image for
convolution and adding noise.
8.1.1 Modeling basics
---------------------
In the subsections below, first a review of some very basic information
and concepts behind modeling a real astronomical image is given. You
can skip this subsection if you are already sufficiently familiar with
these concepts.
8.1.1.1 Defining an ellipse and ellipsoid
.........................................
The PSF, see *note PSF::, and galaxy radial profiles are generally
defined on an ellipse. Therefore, in this section we will start
defining an ellipse on a pixelated 2D surface. Labeling the major axis
of an ellipse $a$, and its minor axis with $b$, the _axis ratio_ is
defined as: $q\equiv b/a$. The major axis of an ellipse can be aligned
in any direction, therefore the angle of the major axis with respect to
the horizontal axis of the image is defined to be the _position angle_
of the ellipse and in this book, we show it with $\theta$.
Our aim is to put a radial profile of any functional form $f(r)$ over
an ellipse. Hence we need to associate a radius/distance to every point
in space. Let's define the radial distance $r_{el}$ as the distance on
the major axis to the center of an ellipse which is located at $i_c$ and
$j_c$ (in other words $r_{el}\equiv{a}$). We want to find $r_{el}$ of a
point located at $(i,j)$ (in the image coordinate system) from the
center of the ellipse with axis ratio $q$ and position angle $\theta$.
First the coordinate system is rotated(1) by $\theta$ to get the new
rotated coordinates of that point $(i_r,j_r)$:
$$i_r(i,j)=+(i_c-i)\cos\theta+(j_c-j)\sin\theta$$
$$j_r(i,j)=-(i_c-i)\sin\theta+(j_c-j)\cos\theta$$
Recall that an ellipse is defined by $(i_r/a)^2+(j_r/b)^2=1$ and that we
defined $r_{el}\equiv{a}$. Hence, multiplying all elements of the
ellipse definition with $r_{el}^2$ we get the elliptical distance at
this point point located: $r_{el}=\sqrt{i_r^2+(j_r/q)^2}$. To place the
radial profiles explained below over an ellipse, $f(r_{el})$ is
calculated based on the functional radial profile desired.
An ellipse in 3D, or an ellipsoid
(https://en.wikipedia.org/wiki/Ellipsoid), can be defined following
similar principles as before. Labeling the major (largest) axis length
as $a$, the second and third (in a right-handed coordinate system) axis
lengths can be labeled as $b$ and $c$. Hence we have two axis ratios:
$q_1\equiv{b/a}$ and $q_2\equiv{c/a}$. The orientation of the ellipsoid
can be defined from the orientation of its major axis. There are many
ways to define 3D orientation and order matters. So to be clear, here
we use the ZXZ (or $Z_1X_2Z_3$) proper Euler angles
(https://en.wikipedia.org/wiki/Euler_angles) to define the 3D
orientation. In short, when a point is rotated in this order, we first
rotate it around the Z axis (third axis) by $\alpha$, then about the
(rotated) X axis by $\beta$ and finally about the (rotated) Z axis by
$\gamma$.
Following the discussion in *note Merging multiple warpings::, we can
define the full rotation with the following matrix multiplication.
However, here we are rotating the coordinates, not the point.
Therefore, both the rotation angles and rotation order are reversed. We
are also not using homogeneous coordinates (see *note Linear warping
basics::) since we are not concerned with translation in this context:
$$\left[\matrix{i_r\cr j_r\cr k_r}\right] =
\left[\matrix{cos\gamma&sin\gamma&0\cr -sin\gamma&cos\gamma&0\cr 0&0&1}\right]
\left[\matrix{1&0&0\cr 0&cos\beta&sin\beta\cr 0&-sin\beta&cos\beta }\right]
\left[\matrix{cos\alpha&sin\alpha&0\cr -sin\alpha&cos\alpha&0\cr 0&0&1}\right]
\left[\matrix{i_c-i\cr j_c-j\cr k_c-k}\right] $$
Recall that an ellipsoid can be characterized with
$(i_r/a)^2+(j_r/b)^2+(k_r/c)^2=1$, so similar to before
($r_{el}\equiv{a}$), we can find the ellipsoidal radius at pixel
$(i,j,k)$ as: $r_{el}=\sqrt{i_r^2+(j_r/q_1)^2+(k_r/q_2)^2}$.
MakeProfiles builds the profile starting from the nearest element
(pixel in an image) in the dataset to the profile center. The profile
value is calculated for that central pixel using Monte Carlo
integration, see *note Sampling from a function::. The next pixel is
the next nearest neighbor to the central pixel as defined by $r_{el}$.
This process goes on until the profile is fully built upto the
truncation radius. This is done fairly efficiently using a breadth
first parsing strategy(2) which is implemented through an ordered linked
list.
Using this approach, we build the profile by expanding the
circumference. Not one more extra pixel has to be checked (the
calculation of $r_{el}$ from above is not cheap in CPU terms). Another
consequence of this strategy is that extending MakeProfiles to three
dimensions becomes very simple: only the neighbors of each pixel have to
be changed. Everything else after that (when the pixel index and its
radial profile have entered the linked list) is the same, no matter the
number of dimensions we are dealing with.
---------- Footnotes ----------
(1) Do not confuse the signs of $sin$ with the rotation matrix
defined in *note Linear warping basics::. In that equation, the point
is rotated, here the coordinates are rotated and the point is fixed.
(2)
8.1.1.2 Point spread function
.............................
Assume we have a 'point' source, or a source that is far smaller than
the maximum resolution (a pixel). When we take an image of it, it will
'spread' over an area. To quantify that spread, we can define a
'function'. This is how the "point spread function" or the PSF of an
image is defined.
This 'spread' can have various causes, for example, in ground-based
astronomy, due to the atmosphere. In practice we can never surpass the
'spread' due to the diffraction of the telescope aperture (even in
Space!). Various other effects can also be quantified through a PSF.
For example, the simple fact that we are sampling in a discrete space,
namely the pixels, also produces a very small 'spread' in the image.
Convolution is the mathematical process by which we can apply a
'spread' to an image, or in other words blur the image, see *note
Convolution process::. The sum of pixels of an image should remain
unchanged after convolution. Therefore, it is important that the sum of
all the pixels of the PSF be unity. The PSF image also has to have an
odd number of pixels on its sides so one pixel can be defined as the
center.
In MakeProfiles, the PSF can be set by the two methods explained
below:
Parametric functions
A known mathematical function is used to make the PSF. In this
case, only the parameters to define the functions are necessary and
MakeProfiles will make a PSF based on the given parameters for each
function. In both cases, the center of the profile has to be
exactly in the middle of the central pixel of the PSF (which is
automatically done by MakeProfiles). When talking about the PSF,
usually, the full width at half maximum or FWHM is used as a scale
of the width of the PSF.
‘Gaussian’
In the older papers, and to a lesser extent even today, some
researchers use the 2D Gaussian function to approximate the
PSF of ground based images. In its most general form, a
Gaussian function can be written as:
$$f(r)=a \exp \left( -(x-\mu)^2 \over 2\sigma^2 \right)+d$$
Since the center of the profile is pre-defined, $\mu$ and $d$
are constrained. $a$ can also be found because the function
has to be normalized. So the only important parameter for
MakeProfiles is the $\sigma$. In the Gaussian function we
have this relation between the FWHM and $\sigma$:
$$\rm{FWHM}_g=2\sqrt{2\ln{2}}\sigma \approx 2.35482\sigma$$
‘Moffat’
The Gaussian profile is much sharper than the images taken
from stars on photographic plates or CCDs. Therefore in 1969,
Moffat proposed this functional form for the image of stars:
$$f(r)=a \left[ 1+\left( r\over \alpha \right)^2 \right]^{-\beta}$$
Again, $a$ is constrained by the normalization, therefore two
parameters define the shape of the Moffat function: $\alpha$
and $\beta$. The radial parameter is $\alpha$ which is
related to the FWHM by
$$\rm{FWHM}_m=2\alpha\sqrt{2^{1/\beta}-1}$$
Comparing with the PSF predicted from atmospheric turbulence
theory with a Moffat function, Trujillo et al.(1) claim that
$\beta$ should be 4.765. They also show how the Moffat PSF
contains the Gaussian PSF as a limiting case when
$\beta\to\infty$.
An input FITS image
An input image file can also be specified to be used as a PSF. If
the sum of its pixels are not equal to 1, the pixels will be
multiplied by a fraction so the sum does become 1.
Gnuastro has tools to extract the non-parametric (extended) PSF of
any image as a FITS file (assuming there are a sufficient number of
stars in it), see *note Building the extended PSF::. This method
is not perfect (will have noise if you do not have many stars), but
it is the actual PSF of the data that is not forced into any
parametric form.
While the Gaussian is only dependent on the FWHM, the Moffat function
is also dependent on $\beta$. Comparing these two functions with a
fixed FWHM gives the following results:
• Within the FWHM, the functions do not have significant differences.
• For a fixed FWHM, as $\beta$ increases, the Moffat function becomes
sharper.
• The Gaussian function is much sharper than the Moffat functions,
even when $\beta$ is large.
---------- Footnotes ----------
(1) Trujillo, I., J. A. L. Aguerri, J. Cepa, and C. M. Gutierrez
(2001). "The effects of seeing on Sérsic profiles - II. The Moffat
PSF". In: MNRAS 328, pp. 977--985.
8.1.1.3 Stars
.............
In MakeProfiles, stars are generally considered to be a point source.
This is usually the case for extra galactic studies, where nearby stars
are also in the field. Since a star is only a point source, we assume
that it only fills one pixel prior to convolution. In fact, exactly for
this reason, in astronomical images the light profiles of stars are one
of the best methods to understand the shape of the PSF and a very large
fraction of scientific research is preformed by assuming the shapes of
stars to be the PSF of the image.
8.1.1.4 Galaxies
................
Today, most practitioners agree that the flux of galaxies can be modeled
with one or a few generalized de Vaucouleur's (or Sérsic) profiles.
$$I(r) = I_e \exp \left ( -b_n \left[ \left( r \over r_e \right)^{1/n} -1 \right] \right )$$
Gérard de Vaucouleurs (1918-1995) was first to show in 1948 that this
function resembles the galaxy light profiles, with the only difference
that he held $n$ fixed to a value of 4. Twenty years later in 1968, J.
L. Sérsic showed that $n$ can have a variety of values and does not
necessarily need to be 4. This profile depends on the effective radius
($r_e$) which is defined as the radius which contains half of the
profile's 2-dimensional integral to infinity (see *note Profile
magnitude::). $I_e$ is the flux at the effective radius. The Sérsic
index $n$ is used to define the concentration of the profile within
$r_e$ and $b_n$ is a constant dependent on $n$. MacArthur et al.(1)
show that for $n>0.35$, $b_n$ can be accurately approximated using this
equation:
$$b_n=2n - {1\over 3} + {4\over 405n} + {46\over 25515n^2} + {131\over 1148175n^3}-{2194697\over 30690717750n^4}$$
---------- Footnotes ----------
(1) MacArthur, L. A., S. Courteau, and J. A. Holtzman (2003).
"Structure of Disk-dominated Galaxies. I. Bulge/Disk Parameters,
Simulations, and Secular Evolution". In: ApJ 582, pp. 689--722.
8.1.1.5 Sampling from a function
................................
A pixel is the ultimate level of accuracy to gather data, we cannot get
any more accurate in one image, this is known as sampling in signal
processing. However, the mathematical profiles which describe our
models have infinite accuracy. Over a large fraction of the area of
astrophysically interesting profiles (for example, galaxies or PSFs),
the variation of the profile over the area of one pixel is not too
significant. In such cases, the elliptical radius ($r_{el}$) of the
center of the pixel can be assigned as the final value of the pixel,
(see *note Defining an ellipse and ellipsoid::).
As you approach their center, some galaxies become very sharp (their
value significantly changes over one pixel's area). This sharpness
increases with smaller effective radius and larger Sérsic values. Thus
rendering the central value extremely inaccurate. The first method that
comes to mind for solving this problem is integration. The functional
form of the profile can be integrated over the pixel area in a 2D
integration process. However, unfortunately numerical integration
techniques also have their limitations and when such sharp profiles are
needed they can become extremely inaccurate.
The most accurate method of sampling a continuous profile on a
discrete space is by choosing a large number of random points within the
boundaries of the pixel and taking their average value (or Monte Carlo
integration). This is also, generally speaking, what happens in
practice with the photons on the pixel. The number of random points can
be set with ‘--numrandom’.
Unfortunately, repeating this Monte Carlo process would be extremely
time and CPU consuming if it is to be applied to every pixel. In order
to not loose too much accuracy, in MakeProfiles, the profile is built
using both methods explained below. The building of the profile begins
from its central pixel and continues (radially) outwards. Monte Carlo
integration is first applied (which yields $F_r$), then the central
pixel value ($F_c$) is calculated on the same pixel. If the fractional
difference ($|F_r-F_c|/F_r$) is lower than a given tolerance level
(specified with ‘--tolerance’) MakeProfiles will stop using Monte Carlo
integration and only use the central pixel value.
The ordering of the pixels in this inside-out construction is based
on $r=\sqrt{(i_c-i)^2+(j_c-j)^2}$, not $r_{el}$, see *note Defining an
ellipse and ellipsoid::. When the axis ratios are large (near one) this
is fine. But when they are small and the object is highly elliptical,
it might seem more reasonable to follow $r_{el}$ not $r$. The problem
is that the gradient is stronger in pixels with smaller $r$ (and larger
$r_{el}$) than those with smaller $r_{el}$. In other words, the
gradient is strongest along the minor axis. So if the next pixel is
chosen based on $r_{el}$, the tolerance level will be reached sooner and
lots of pixels with large fractional differences will be missed.
Monte Carlo integration uses a random number of points. Thus, every
time you run it, by default, you will get a different distribution of
points to sample within the pixel. In the case of large profiles, this
will result in a slight difference of the pixels which use Monte Carlo
integration each time MakeProfiles is run. To have a deterministic
result, you have to fix the random number generator properties which is
used to build the random distribution. This can be done by setting the
‘GSL_RNG_TYPE’ and ‘GSL_RNG_SEED’ environment variables and calling
MakeProfiles with the ‘--envseed’ option. To learn more about the
process of generating random numbers, see *note Generating random
numbers::.
The seed values are fixed for every profile: with ‘--envseed’, all
the profiles have the same seed and without it, each will get a
different seed using the system clock (which is accurate to within one
microsecond). The same seed will be used to generate a random number
for all the sub-pixel positions of all the profiles. So in the former,
the sub-pixel points checked for all the pixels undergoing Monte carlo
integration in all profiles will be identical. In other words, the
sub-pixel points in the first (closest to the center) pixel of all the
profiles will be identical with each other. All the second pixels
studied for all the profiles will also receive an identical (different
from the first pixel) set of sub-pixel points and so on. As long as the
number of random points used is large enough or the profiles are not
identical, this should not cause any systematic bias.
8.1.1.6 Oversampling
....................
The steps explained in *note Sampling from a function:: do give an
accurate representation of a profile prior to convolution. However, in
an actual observation, the image is first convolved with or blurred by
the atmospheric and instrument PSF in a continuous space and then it is
sampled on the discrete pixels of the camera.
In order to more accurately simulate this process, the unconvolved
image and the PSF are created on a finer pixel grid. In other words,
the output image is a certain odd-integer multiple of the desired size,
we can call this 'oversampling'. The user can specify this multiple as
a command-line option. The reason this has to be an odd number is that
the PSF has to be centered on the center of its image. An image with an
even number of pixels on each side does not have a central pixel.
The image can then be convolved with the PSF (which should also be
oversampled on the same scale). Finally, image can be sub-sampled to
get to the initial desired pixel size of the output image. After this,
mock noise can be added as explained in the next section. This is
because unlike the PSF, the noise occurs in each output pixel, not on a
continuous space like all the prior steps.
8.1.2 If convolving afterwards
------------------------------
In case you want to convolve the image later with a given point spread
function, make sure to use a larger image size. After convolution, the
profiles become larger and a profile that is normally completely outside
of the image might fall within it.
On one axis, if you want your final (convolved) image to be $m$
pixels and your PSF is $2n+1$ pixels wide, then when calling
MakeProfiles, set the axis size to $m+2n$, not $m$. You also have to
shift all the pixel positions of the profile centers on the that axis by
$n$ pixels to the positive.
After convolution, you can crop the outer $n$ pixels with the section
crop box specification of Crop: ‘--section=n+1:*-n,n+1:*-n’ (according
to the FITS standard, counting is from 1 so we use ‘n+1’) assuming your
PSF is a square, see *note Crop section syntax::. This will also remove
all discrete Fourier transform artifacts (blurred sides) from the final
image. To facilitate this shift, MakeProfiles has the options
‘--xshift’, ‘--yshift’ and ‘--prepforconv’, see *note Invoking
astmkprof::.
8.1.3 Profile magnitude
-----------------------
To find the profile's total magnitude, (see *note Brightness flux
magnitude::), it is customary to use the 2D integration of the flux to
infinity. However, in MakeProfiles we do not follow this idealistic
approach and apply a more realistic method to find the total magnitude:
the sum of all the pixels belonging to a profile within its predefined
truncation radius. Note that if the truncation radius is not large
enough, this can be significantly different from the total integrated
light to infinity.
An integration to infinity is not a realistic condition because no
galaxy extends indefinitely (important for high Sérsic index profiles),
pixelation can also cause a significant difference between the actual
total pixel sum value of the profile and that of integration to
infinity, especially in small and high Sérsic index profiles. To be
safe, you can specify a large enough truncation radius for such compact
high Sérsic index profiles.
If oversampling is used then the pixel value is calculated using the
over-sampled image, see *note Oversampling:: which is much more
accurate. The profile is first built in an array completely bounding it
with a normalization constant of unity (see *note Galaxies::). Taking
$V$ to be the desired pixel value and $S$ to be the sum of the pixels in
the created profile, every pixel is then multiplied by $V/S$ so the sum
is exactly $V$.
If the ‘--individual’ option is called, this same array is written to
a FITS file. If not, only the overlapping pixels of this array and the
output image are kept and added to the output array.
8.1.4 Invoking MakeProfiles
---------------------------
MakeProfiles will make any number of profiles specified in a catalog
either individually or in one image. The executable name is ‘astmkprof’
with the following general template
$ astmkprof [OPTION ...] [Catalog]
One line examples:
## Make an image with profiles in catalog.txt (with default size):
$ astmkprof catalog.txt
## Make the profiles in catalog.txt over image.fits:
$ astmkprof --background=image.fits catalog.txt
## Make a Moffat PSF with FWHM 3pix, beta=2.8, truncation=5
$ astmkprof --kernel=moffat,3,2.8,5 --oversample=1
## Make profiles in catalog, using RA and Dec in the given column:
$ astmkprof --ccol=RA_CENTER --ccol=DEC_CENTER --mode=wcs catalog.txt
## Make a 1500x1500 merged image (oversampled 500x500) image along
## with an individual image for all the profiles in catalog:
$ astmkprof --individual --oversample 3 --mergedsize=500,500 cat.txt
The parameters of the mock profiles can either be given through a
catalog (which stores the parameters of many mock profiles, see *note
MakeProfiles catalog::), or the ‘--kernel’ option (see *note
MakeProfiles output dataset::). The catalog can be in the FITS ASCII,
FITS binary format, or plain text formats (see *note Tables::). A plain
text catalog can also be provided using the Standard input (see *note
Standard input::). The columns related to each parameter can be
determined both by number, or by match/search criteria using the column
names, units, or comments, with the options ending in ‘col’, see below.
Without any file given to the ‘--background’ option, MakeProfiles
will make a zero-valued image and build the profiles on that (its size
and main WCS parameters can also be defined through the options
described in *note MakeProfiles output dataset::). Besides the
main/merged image containing all the profiles in the catalog, it is also
possible to build individual images for each profile (only enclosing one
full profile to its truncation radius) with the ‘--individual’ option.
If an image is given to the ‘--background’ option, the pixels of that
image are used as the background value for every pixel hence flux value
of each profile pixel will be added to the pixel in that background
value. You can disable this with the ‘--clearcanvas’ option (which will
initialize the background to zero-valued pixels and build profiles over
that). With the ‘--background’ option, the values to all options
relating to the "canvas" (output size and WCS) will be ignored if
specified: ‘--oversample’, ‘--mergedsize’, ‘--prepforconv’, ‘--crpix’,
‘--crval’, ‘--cdelt’, ‘--cdelt’, ‘--pc’, ‘cunit’ and ‘ctype’.
The sections below discuss the options specific to MakeProfiles based
on context: the input catalog settings which can have many rows for
different profiles are discussed in *note MakeProfiles catalog::, in
*note MakeProfiles profile settings::, we discuss how you can set
general profile settings (that are the same for all the profiles in the
catalog). Finally *note MakeProfiles output dataset:: and *note
MakeProfiles log file:: discuss the outputs of MakeProfiles and how you
can configure them. Besides these, MakeProfiles also supports all the
common Gnuastro program options that are discussed in *note Common
options::, so please flip through them is well for a more comfortable
usage.
When building 3D profiles, there are more degrees of freedom. Hence,
more columns are necessary and all the values related to dimensions (for
example, size of dataset in each dimension and the WCS properties) must
also have 3 values. To allow having an independent set of default
values for creating 3D profiles, MakeProfiles also installs a
‘astmkprof-3d.conf’ configuration file (see *note Configuration
files::). You can use this for default 3D profile values. For example,
if you installed Gnuastro with the prefix ‘/usr/local’ (the default
location, see *note Installation directory::), you can benefit from this
configuration file by running MakeProfiles like the example below. As
with all configuration files, if you want to customize a given option,
call it before the configuration file.
$ astmkprof --config=/usr/local/etc/gnuastro/astmkprof-3d.conf \
catalog.txt
To further simplify the process, you can define a shell alias in any
startup file (for example, ‘~/.bashrc’, see *note Installation
directory::). Assuming that you installed Gnuastro in ‘/usr/local’, you
can add this line to the startup file (you may put it all in one line,
it is broken into two lines here for fitting within page limits).
alias astmkprof-3d="astmkprof \
--config=/usr/local/etc/gnuastro/astmkprof-3d.conf"
Using this alias, you can call MakeProfiles with the name ‘astmkprof-3d’
(instead of ‘astmkprof’). It will automatically load the 3D specific
configuration file first, and then parse any other arguments, options or
configuration files. You can change the default values in this 3D
configuration file by calling them on the command-line as you do with
‘astmkprof’(1).
Please see *note Sufi simulates a detection:: for a very complete
tutorial explaining how one could use MakeProfiles in conjunction with
other Gnuastro's programs to make a complete simulated image of a mock
galaxy.
---------- Footnotes ----------
(1) Recall that for single-invocation options, the last command-line
invocation takes precedence over all previous invocations (including
those in the 3D configuration file). See the description of ‘--config’
in *note Operating mode options::.
8.1.4.1 MakeProfiles catalog
............................
The catalog containing information about each profile can be in the FITS
ASCII, FITS binary, or plain text formats (see *note Tables::). The
latter can also be provided using standard input (see *note Standard
input::). Its columns can be ordered in any desired manner. You can
specify which columns belong to which parameters using the set of
options discussed below. For example, through the ‘--rcol’ and ‘--tcol’
options, you can specify the column that contains the radial parameter
for each profile and its truncation respectively. See *note Selecting
table columns:: for a thorough discussion on the values to these
options.
The value for the profile center in the catalog (the ‘--ccol’ option)
can be a floating point number so the profile center can be on any
sub-pixel position. Note that pixel positions in the FITS standard
start from 1 and an integer is the pixel center. So a 2D image actually
starts from the position (0.5, 0.5), which is the bottom-left corner of
the first pixel. When a ‘--background’ image with WCS information is
provided, or you specify the WCS parameters with the respective
options(1), you may also use RA and Dec to identify the center of each
profile (see the ‘--mode’ option below).
In MakeProfiles, profile centers do not have to be in (overlap with)
the final image. Even if only one pixel of the profile within the
truncation radius overlaps with the final image size, the profile is
built and included in the final image. Profiles that are completely out
of the image will not be created (unless you explicitly ask for it with
the ‘--individual’ option). You can use the output log file (created
with ‘--log’ to see which profiles were within the image, see *note
Common options::.
If PSF profiles (Moffat or Gaussian, see *note PSF::) are in the
catalog and the profiles are to be built in one image (when
‘--individual’ is not used), it is assumed they are the PSF(s) you want
to convolve your created image with. So by default, they will not be
built in the output image but as separate files. The sum of pixels of
these separate files will also be set to unity (1) so you are ready to
convolve, see *note Convolution process::. As a summary, the position
and magnitude of PSF profile will be ignored. This behavior can be
disabled with the ‘--psfinimg’ option. If you want to create all the
profiles separately (with ‘--individual’) and you want the sum of the
PSF profile pixels to be unity, you have to set their magnitudes in the
catalog to the zero point magnitude and be sure that the central
positions of the profiles do not have any fractional part (the PSF
center has to be in the center of the pixel).
The list of options directly related to the input catalog columns is
shown below.
‘--ccol=STR/INT’
Center coordinate column for each dimension. This option must be
called two times to define the center coordinates in an image. For
example, ‘--ccol=RA’ and ‘--ccol=DEC’ (along with ‘--mode=wcs’)
will inform MakeProfiles to look into the catalog columns named
‘RA’ and ‘DEC’ for the Right Ascension and Declination of the
profile centers.
‘--fcol=INT/STR’
The functional form of the profile with one of the values below
depending on the desired profile. The column can contain either
the numeric codes (for example, '‘1’') or string characters (for
example, '‘sersic’'). The numeric codes are easier to use in
scripts which generate catalogs with hundreds or thousands of
profiles.
The string format can be easier when the catalog is to be
written/checked by hand/eye before running MakeProfiles. It is
much more readable and provides a level of documentation. All
Gnuastro's recognized table formats (see *note Recognized table
formats::) accept string type columns. To have string columns in a
plain text table/catalog, see *note Gnuastro text table format::.
• Sérsic profile with '‘sersic’' or '‘1’'.
• Moffat profile with '‘moffat’' or '‘2’'.
• Gaussian profile with '‘gaussian’' or '‘3’'.
• Point source with '‘point’' or '‘4’'.
• Flat profile with '‘flat’' or '‘5’'.
• Circumference profile with '‘circum’' or '‘6’'. A fixed value
will be used for all pixels less than or equal to the
truncation radius ($r_t$) and greater than $r_t-w$ ($w$ is the
value to the ‘--circumwidth’).
• Radial distance profile with '‘distance’' or '‘7’'. At the
lowest level, each pixel only has an elliptical radial
distance given the profile's shape and orientation (see *note
Defining an ellipse and ellipsoid::). When this profile is
chosen, the pixel's elliptical radial distance from the
profile center is written as its value. For this profile, the
value in the magnitude column (‘--mcol’) will be ignored.
You can use this for checks or as a first approximation to
define your own higher-level radial function. In the latter
case, just note that the central values are going to be
incorrect (see *note Sampling from a function::).
• Custom radial profile with '‘custom-prof’' or '‘8’'. The
values to use for each radial interval should be in the table
given to ‘--customtable’. By default, once the profile is
built with the given values, it will be scaled to have a total
magnitude that you have requested in the magnitude column of
the profile (in ‘--mcol’). If you want the raw values in the
2D profile (to ignore the magnitude column), use
‘--mcolnocustprof’. For more, see the description of
‘--customtable’ in *note MakeProfiles profile settings::.
• Azimuthal angle profile with '‘azimuth’' or '‘9’'. Every
pixel within the truncation radius will be given its azimuthal
angle (in degrees, from 0 to 360) from the major axis. In
combination with the radial distance profile, you can now
create complex features in polar coordinates, such as tidal
tails or tidal shocks (using the Arithmetic program to mix the
radius and azimuthal angle through a function to create your
desired features).
• Custom image with '‘custom-img’' or '‘10’'. The image(s) to
use should be given to the ‘--customimg’ option (which can be
called multiple times for multiple images). To identify which
one of the images (given to ‘--customimg’) should be used, you
should specify their counter in the "radius" column below.
For more, see the description of ‘custom-img’ in *note
MakeProfiles profile settings::.
‘--rcol=STR/INT’
The radius parameter of the profiles. Effective radius ($r_e$) if
Sérsic, FWHM if Moffat or Gaussian.
For a custom image profile, this option is not interpreted as a
radius, but as a counter (identifying which one of the images given
to ‘--customimg’ should be used for each row).
‘--ncol=STR/INT’
The Sérsic index ($n$) or Moffat $\beta$.
‘--pcol=STR/INT’
The position angle (in degrees) of the profiles relative to the
first FITS axis (horizontal when viewed in SAO DS9). When building
a 3D profile, this is the first Euler angle: first rotation of the
ellipsoid major axis from the first FITS axis (rotating about the
third axis). See *note Defining an ellipse and ellipsoid::.
‘--p2col=STR/INT’
Second Euler angle (in degrees) when building a 3D ellipsoid. This
is the second rotation of the ellipsoid major axis (following
‘--pcol’) about the (rotated) X axis. See *note Defining an
ellipse and ellipsoid::. This column is ignored when building a 2D
profile.
‘--p3col=STR/INT’
Third Euler angle (in degrees) when building a 3D ellipsoid. This
is the third rotation of the ellipsoid major axis (following
‘--pcol’ and ‘--p2col’) about the (rotated) Z axis. See *note
Defining an ellipse and ellipsoid::. This column is ignored when
building a 2D profile.
‘--qcol=STR/INT’
The axis ratio of the profiles (minor axis divided by the major
axis in a 2D ellipse). When building a 3D ellipse, this is the
ratio of the major axis to the semi-axis length of the second
dimension (in a right-handed coordinate system). See $q1$ in *note
Defining an ellipse and ellipsoid::.
‘--q2col=STR/INT’
The ratio of the ellipsoid major axis to the third semi-axis length
(in a right-handed coordinate system) of a 3D ellipsoid. See $q1$
in *note Defining an ellipse and ellipsoid::. This column is
ignored when building a 2D profile.
‘--mcol=STR/INT’
The total pixelated magnitude of the profile within the truncation
radius, see *note Profile magnitude::.
‘--tcol=STR/INT’
The truncation radius of this profile. By default it is in units
of the radial parameter of the profile (the value in the ‘--rcol’
of the catalog). If ‘--tunitinp’ is given, this value is
interpreted in units of pixels (prior to oversampling) irrespective
of the profile.
---------- Footnotes ----------
(1) The options to set the WCS are the following: ‘--crpix’,
‘--crval’, ‘--cdelt’, ‘--cdelt’, ‘--pc’, ‘cunit’ and ‘ctype’. Just
recall that these options are only used if ‘--background’ is not given:
if the image you give to ‘--background’ does not have WCS, these options
will not be used and you cannot use WCS-mode coordinates like RA or Dec.
8.1.4.2 MakeProfiles profile settings
.....................................
The profile parameters that differ between each created profile are
specified through the columns in the input catalog and described in
*note MakeProfiles catalog::. Besides those there are general settings
for some profiles that do not differ between one profile and another,
they are a property of the general process. For example, how many
random points to use in the monte-carlo integration, this value is fixed
for all the profiles. The options described in this section are for
configuring such properties.
‘--mode=STR’
Interpret the center position columns (‘--ccol’ in *note
MakeProfiles catalog::) in image or WCS coordinates. This option
thus accepts only two values: ‘img’ and ‘wcs’. It is mandatory
when a catalog is being used as input.
‘-r’
‘--numrandom’
The number of random points used in the central regions of the
profile, see *note Sampling from a function::.
‘-e’
‘--envseed’
Use the value to the ‘GSL_RNG_SEED’ environment variable to
generate the random Monte Carlo sampling distribution, see *note
Sampling from a function:: and *note Generating random numbers::.
‘-t FLT’
‘--tolerance=FLT’
The tolerance to switch from Monte Carlo integration to the central
pixel value, see *note Sampling from a function::.
‘-p’
‘--tunitinp’
The truncation column of the catalog is in units of pixels. By
default, the truncation column is considered to be in units of the
radial parameters of the profile (‘--rcol’). Read it as
't-unit-in-p' for 'truncation unit in pixels'.
‘-f’
‘--mforflatpix’
When making fixed value profiles ("flat", "circumference" or
"point" profiles, see '‘--fcol’'), do not use the value in the
column specified by '‘--mcol’' as the magnitude. Instead use it as
the exact value that all the pixels of these profiles should have.
This option is irrelevant for other types of profiles. This option
is very useful for creating masks, or labeled regions in an image.
Any integer, or floating point value can used in this column with
this option, including ‘NaN’ (or '‘nan’', or '‘NAN’', case is
irrelevant), and infinities (‘inf’, ‘-inf’, or ‘+inf’).
For example, with this option if you set the value in the magnitude
column (‘--mcol’) to ‘NaN’, you can create an elliptical or
circular mask over an image (which can be given as the argument),
see *note Blank pixels::. Another useful application of this
option is to create labeled elliptical or circular apertures in an
image. To do this, set the value in the magnitude column to the
label you want for this profile. This labeled image can then be
used in combination with NoiseChisel's output (see *note
NoiseChisel output::) to do aperture photometry with MakeCatalog
(see *note MakeCatalog::).
Alternatively, if you want to mark regions of the image (for
example, with an elliptical circumference) and you do not want to
use NaN values (as explained above) for some technical reason, you
can get the minimum or maximum value in the image (1) using
Arithmetic (see *note Arithmetic::), then use that value in the
magnitude column along with this option for all the profiles.
Please note that when using MakeProfiles on an already existing
image, you have to set '‘--oversample=1’'. Otherwise all the
profiles will be scaled up based on the oversampling scale in your
configuration files (see *note Configuration files::) unless you
have accounted for oversampling in your catalog.
‘--mcolissum’
The value given in the "magnitude" column (specified by ‘--mcol’,
see *note MakeProfiles catalog::) must be interpreted as total sum
of pixel values, not magnitude (which is measured from the total
sum and zero point, see *note Brightness flux magnitude::). When
this option is called, the zero point magnitude (value to the
‘--zeropoint’ option) is ignored and the given value must have the
same units as the input dataset's pixels.
Recall that the total profile magnitude that is specified with in
the ‘--mcol’ column of the input catalog is not an integration to
infinity, but the actual sum of pixels in the profile (until the
desired truncation radius). See *note Profile magnitude:: for more
on this point.
‘--mcolnocustprof’
Do not touch (re-scale) the custom profile that should be inserted
in ‘custom-prof’ profile (see the description of ‘--fcol’ in *note
MakeProfiles catalog:: or the description of ‘--customtable’
below). By default, MakeProfiles will scale (multiply) the custom
image's pixels to have the desired magnitude (or sum of pixels if
‘--mcolissum’ is called) in that row.
‘--mcolnocustimg’
Do not touch (re-scale) the custom image that should be inserted in
‘custom-img’ profile (see the description of ‘--fcol’ in *note
MakeProfiles catalog::). By default, MakeProfiles will scale
(multiply) the custom image's pixels to have the desired magnitude
(or sum of pixels if ‘--mcolissum’ is called) in that row.
‘--magatpeak’
The magnitude column in the catalog (see *note MakeProfiles
catalog::) will be used to set the value only for the profile's
peak (maximum) pixel, not the full profile. Note that this is the
flux of the profile's peak (maximum) pixel in the final output of
MakeProfiles. So beware of the oversampling, see *note
Oversampling::.
This option can be useful if you want to check a mock profile's
total magnitude at various truncation radii. Without this option,
no matter what the truncation radius is, the total magnitude will
be the same as that given in the catalog. But with this option,
the total magnitude will become brighter as you increase the
truncation radius.
In sharper profiles, sometimes the accuracy of measuring the peak
profile flux is more than the overall object sum or magnitude. In
such cases, with this option, the final profile will be built such
that its peak has the given magnitude, not the total profile.
*CAUTION:* If you want to use this option for comparing with
observations, please note that MakeProfiles does not do
convolution. Unless you have deconvolved your data, your images
are convolved with the instrument and atmospheric PSF, see *note
PSF::. Particularly in sharper profiles, the flux in the peak
pixel is strongly decreased after convolution. Also note that in
such cases, besides deconvolution, you will have to set
‘--oversample=1’ otherwise after resampling your profile with Warp
(see *note Warp::), the peak flux will be different.
‘--customtable FITS/TXT’
The filename of the table to use in the custom radial profiles (see
description of ‘--fcol’ in *note MakeProfiles catalog::. This can
be a plain-text table, or FITS table, see *note Recognized table
formats::, if it is a FITS table, you can use ‘--customtablehdu’ to
specify which HDU should be used (described below).
A custom radial profile can have any value you want for a given
radial profile (including NaN/blank values). Each interval is
defined by its minimum (inclusive) and maximum (exclusive) radius,
when a pixel center falls within a radius interval, the value
specified for that interval will be used. If a pixel is not in the
given intervals, a value of 0.0 will be used for that pixel.
The table should have 3 columns as shown below. If the intervals
are contiguous (the maximum value of the previous interval is equal
to the minimum value of an interval) and the intervals all have the
same size (difference between minimum and maximum values) the
creation of these profiles will be fast. However, if the intervals
are not sorted and contiguous, MakeProfiles will parse the
intervals from the top of the table and use the first interval that
contains the pixel center (this may slow it down).
Column 1:
The interval's minimum radius.
Column 2:
The interval's maximum radius.
Column 3:
The value to be used for pixels within the given interval
(including NaN/blank).
Gnuastro's column arithmetic in the Table program has the
‘sorted-to-interval’ operator that will generate the first two
columns from a single column (your radial profile). See the
description of that operator in *note Column arithmetic:: and the
example below.
By default, once a 2D image is constructed for the radial profile,
it will be scaled such that its total magnitude corresponds to the
value in the magnitude column (‘--mcol’) of the main input catalog.
If you want to disable the scaling and use the raw values in your
custom profile (in other words: you want to ignore the magnitude
column) you need to call ‘--mcolnocustprof’ (see above).
In the example below, we'll start with a certain radial profile,
and use this option to build its 2D representation in an image
(recall that you can build radial profiles with *note Generate
radial profile::). But first, we will need to use the
‘sorted-to-interval’ to build the necessary input format (see *note
Column arithmetic::).
$ cat radial.txt
# Column 1: RADIUS [pix ,f32,] Radial distance
# Column 2: MEAN [input-units,f32,] Mean of values.
0.0 1.00000
1.0 0.50184
1.4 0.37121
2.0 0.26414
2.2 0.23427
2.8 0.17868
3.0 0.16627
3.1 0.15567
3.6 0.13132
4.0 0.11404
## Convert the radius in each row to an interval
$ asttable radial.txt --output=interval.fits \
-c'arith RADIUS sorted-to-interval',MEAN
## Inspect the table containing intervals
$ asttable interval.fits -ffixed
-0.500000 0.500000 1.000000
0.500000 1.200000 0.501840
1.200000 1.700000 0.371210
1.700000 2.100000 0.264140
2.100000 2.500000 0.234270
2.500000 2.900000 0.178680
2.900000 3.050000 0.166270
3.050000 3.350000 0.155670
3.350000 3.800000 0.131320
3.800000 4.200000 0.114040
## Build the 2D image of the profile from the interval.
$ echo "1 7 7 8 10 2.5 0 1 1 2" \
| astmkprof --mergedsize=13,13 --oversample=1 \
--customtable=interval.fits \
--output=image.fits
## View the created FITS image.
$ astscript-fits-view image.fits --ds9scale=minmax
Recall that if you want your image pixels to have the same values
as the ‘MEAN’ column in your profile, you should run MakeProfiles
with ‘--mcolnocustprof’.
In case you want to build the profile using *note Generate radial
profile::, be sure to use the ‘--oversample’ option of
‘astscript-radial-profile’. The higher the oversampling, the
better your result will be. For example you can run the following
script to see the effect (also see bug 65106
(https://savannah.gnu.org/bugs/?65106)). But don't take the
oversampling too high: both the radial profile script and
MakeProfiles will become slower and the precision of your results
will decrease.
#!/bin/bash
# Function to avoid repeating code: first generate a radial profile
# with a certain oversampling, then build a 2D profile from it):
# The first argument is the oversampling, the second is the suffix.
gen_rad_make_2dprf () {
# Generate the radial profile
radraw=$bdir/radial-profile-$2.fits
astscript-radial-profile $prof -o$radraw \
--oversample=$1 \
--zeroisnotblank
# Generate the custom table format
custraw=$bdir/customtable-$2.fits
asttable $radraw --output=interval.fits \
-c'arith RADIUS sorted-to-interval',MEAN \
-o$custraw
# Build the 2D profile.
prof2draw=$bdir/prof2d-$2.fits
echo "1 $xc $yc 8 30 0 0 1 0 1" \
| astmkprof --customtable=$custraw \
--mergedsize=$xw,$yw \
--output=$prof2draw \
--mcolnocustprof \
--oversample=1 \
--clearcanvas \
--mode=img
}
# Directory to hold built files
bdir=build
if ! [ -d $bdir ]; then mkdir $bdir; fi
# Build a Gaussian profile in the center of an image to start with.
prof=$bdir/prof.fits
astmkprof --kernel=gaussian,2,5 -o$prof
# Find the center pixel of the image
xw=$(astfits $prof --keyvalue=NAXIS1 --quiet)
yw=$(astfits $prof --keyvalue=NAXIS2 --quiet)
xc=$(echo $xw | awk '{print int($1/2)+1}')
yc=$(echo $yw| awk '{print int($1/2)+1}')
# Generate two 2D radial profiles, one with an oversampling of 1
# and another with an oversampling of 5.
gen_rad_make_2dprf 1 "raw"
gen_rad_make_2dprf 5 "oversample"
# View the two images beside each other:
astscript-fits-view $bdir/prof2d-raw.fits \
$bdir/prof2d-oversample.fits
‘--customtablehdu INT/STR’
The HDU/extension in the FITS file given to ‘--customtable’.
‘--customimg=STR[,STR]’
A custom FITS image that should be used for the ‘custom-img’
profiles (see the description of ‘--fcol’ in *note MakeProfiles
catalog::). Multiple files can be given to this option (separated
by a comma), and this option can be called multiple times itself
(useful when many custom image profiles should be added). If the
HDU of the images are different, you can use ‘--customimghdu’
(described below).
Through the "radius" column, MakeProfiles will know which one of
the images given to this option should be used in each row. For
example, let's assume your input catalog (‘cat.fits’) has the
following contents (output of first command below), and you call
MakeProfiles like the second command below to insert four profiles
into the background ‘back.fits’ image.
The first profile below is Sersic (with an ‘--fcol’, or 4-th
column, code of ‘1’). So MakeProfiles builds the pixels of the
first profile, and all column values are meaningful. However, the
second, third and fourth inserted objects are custom images (with
an ‘--fcol’ code of ‘10’). For the custom image profiles, you see
that the radius column has values of ‘1’ or ‘2’. This tells
MakeProfiles to use the first image given to ‘--customimg’ (or
‘gal-1.fits’) for the second and fourth inserted objects. The
second image given to ‘--customimage’ (or ‘gal-2.fits’) will be
used for the third inserted object. Finally, all three custom
image profiles have different magnitudes, and the values in
‘--ncol’, ‘--pcol’, ‘--qcol’ and ‘--tcol’ are ignored.
$ cat cat.fits
1 53.15506 -27.785165 1 20 1 20 0.6 25 5
2 53.15602 -27.777887 10 1 0 0 0 22 0
3 53.16440 -27.775876 10 2 0 0 0 24 0
4 53.16849 -27.787406 10 1 0 0 0 23 0
$ astmkprof cat.fits --mode=wcs --zeropoint=25.68 \
--background=back.fits --output=out.fits \
--customimg=gal-1.fits --customimg=gal-2.fits
‘--customimghdu=INT/STR’
The HDU(s) of the images given to ‘--customimghdu’. If this option
is only called once, but ‘--customimg’ is called many times,
MakeProfiles will assume that all images given to ‘--customimg’
have the same HDU. Otherwise (if the number of HDUs is equal to the
number of images), then each image will use its corresponding HDU.
‘-X INT,INT’
‘--shift=INT,INT’
Shift all the profiles and enlarge the image along each dimension.
To better understand this option, please see $n$ in *note If
convolving afterwards::. This is useful when you want to convolve
the image afterwards. If you are using an external PSF, be sure to
oversample it to the same scale used for creating the mock images.
If a background image is specified, any possible value to this
option is ignored.
‘-c’
‘--prepforconv’
Shift all the profiles and enlarge the image based on half the
width of the first Moffat or Gaussian profile in the catalog,
considering any possible oversampling see *note If convolving
afterwards::. ‘--prepforconv’ is only checked and possibly
activated if ‘--xshift’ and ‘--yshift’ are both zero (after reading
the command-line and configuration files). If a background image
is specified, any possible value to this option is ignored.
‘-z FLT’
‘--zeropoint=FLT’
The zero point magnitude of the input. For more on the zero point
magnitude, see *note Brightness flux magnitude::.
‘-w FLT’
‘--circumwidth=FLT’
The width of the circumference if the profile is to be an
elliptical circumference or annulus. See the explanations for this
type of profile in ‘--fcol’.
‘-R’
‘--replace’
Do not add the pixels of each profile over the background, or other
profiles. But replace the values.
By default, when two profiles overlap, the final pixel value is the
sum of all the profiles that overlap on that pixel. This is the
expected situation when dealing with physical object profiles like
galaxies or stars/PSF. However, when MakeProfiles is used to build
integer labeled images (for example, in *note Aperture
photometry::), this is not the expected situation: the sum of two
labels will be a new label. With this option, the pixels are not
added but the largest (maximum) value over that pixel is used.
Because the maximum operator is independent of the order of values,
the output is also thread-safe.
---------- Footnotes ----------
(1) The minimum will give a better result, because the maximum can be
too high compared to most pixels in the image, making it harder to
display.
8.1.4.3 MakeProfiles output dataset
...................................
MakeProfiles takes an input catalog uses basic properties that are
defined there to build a dataset, for example, a 2D image containing the
profiles in the catalog. In *note MakeProfiles catalog:: and *note
MakeProfiles profile settings::, the catalog and profile settings were
discussed. The options of this section, allow you to configure the
output dataset (or the canvas that will host the built profiles).
‘-k FITS’
‘--background=FITS’
A background image FITS file to build the profiles on. The
extension that contains the image should be specified with the
‘--backhdu’ option, see below. When a background image is
specified, it will be used to derive all the information about the
output image. Hence, the following options will be ignored:
‘--mergedsize’, ‘--oversample’, ‘--crpix’, ‘--crval’ (generally,
all other WCS related parameters) and the output's data type (see
‘--type’ in *note Input output options::).
The background image will act like a canvas to build the profiles
on: profile pixel values will be summed with the background image
pixel values. With the ‘--replace’ option you can disable this
behavior and replace the profile pixels with the background pixels.
If you want to use all the image information above, except for the
pixel values (you want to have a blank canvas to build the profiles
on, based on an input image), you can call ‘--clearcanvas’, to set
all the input image's pixels to zero before starting to build the
profiles over it (this is done in memory after reading the input,
so nothing will happen to your input file).
‘-B STR/INT’
‘--backhdu=STR/INT’
The header data unit (HDU) of the file given to ‘--background’.
‘-C’
‘--clearcanvas’
When an input image is specified (with the ‘--background’ option,
set all its pixels to 0.0 immediately after reading it into memory.
Effectively, this will allow you to use all its properties
(described under the ‘--background’ option), without having to
worry about the pixel values.
‘--clearcanvas’ can come in handy in many situations, for example,
if you want to create a labeled image (segmentation map) for
creating a catalog (see *note MakeCatalog::). In other cases, you
might have modeled the objects in an image and want to create them
on the same frame, but without the original pixel values.
‘-E STR/INT,FLT[,FLT,[...]]’
‘--kernel=STR/INT,FLT[,FLT,[...]]’
Only build one kernel profile with the parameters given as the
values to this option. The different values must be separated by a
comma (<,>). The first value identifies the radial function of the
profile, either through a string or through a number (see
description of ‘--fcol’ in *note MakeProfiles catalog::). Each
radial profile needs a different total number of parameters: Sérsic
and Moffat functions need 3 parameters: radial, Sérsic index or
Moffat $\beta$, and truncation radius. The Gaussian function needs
two parameters: radial and truncation radius. The point function
does not need any parameters and flat and circumference profiles
just need one parameter (truncation radius).
The PSF or kernel is a unique (and highly constrained) type of
profile: the sum of its pixels must be one, its center must be the
center of the central pixel (in an image with an odd number of
pixels on each side), and commonly it is circular, so its axis
ratio and position angle are one and zero respectively. Kernels
are commonly necessary for various data analysis and data
manipulation steps (for example, see *note Convolve::, and *note
NoiseChisel::. Because of this it is inconvenient to define a
catalog with one row and many zero valued columns (for all the
non-necessary parameters). Hence, with this option, it is possible
to create a kernel with MakeProfiles without the need to create a
catalog. Here are some examples:
‘--kernel=moffat,3,2.8,5’
A Moffat kernel with FWHM of 3 pixels, $\beta=2.8$ which is
truncated at 5 times the FWHM.
‘--kernel=gaussian,2,3’
A circular Gaussian kernel with FWHM of 2 pixels and truncated
at 3 times the FWHM.
This option may also be used to create a 3D kernel. To do that,
two small modifications are necessary: add a ‘-3d’ (or ‘-3D’) to
the profile name (for example, ‘moffat-3d’) and add a number
(axis-ratio along the third dimension) to the end of the parameters
for all profiles except ‘point’. The main reason behind providing
an axis ratio in the third dimension is that in 3D astronomical
datasets, commonly the third dimension does not have the same
nature (units/sampling) as the first and second.
For example, in IFU (optical) or Radio data cubes, the first and
second dimensions are commonly spatial/angular positions (like RA
and Dec) but the third dimension is wavelength or frequency (in
units of Angstroms for Herz). Because of this different nature
(which also affects the processing), it may be necessary for the
kernel to have a different extent in that direction.
If the 3rd dimension axis ratio is equal to $1.0$, then the kernel
will be a spheroid. If it is smaller than $1.0$, the kernel will
be button-shaped: extended less in the third dimension. However,
when it islarger than $1.0$, the kernel will be bullet-shaped:
extended more in the third dimension. In the latter case, the
radial parameter will correspond to the length along the 3rd
dimension. For example, let's have a look at the two examples
above but in 3D:
‘--kernel=moffat-3d,3,2.8,5,0.5’
An ellipsoid Moffat kernel with FWHM of 3 pixels, $\beta=2.8$
which is truncated at 5 times the FWHM. The ellipsoid is
circular in the first two dimensions, but in the third
dimension its extent is half the first two.
‘--kernel=gaussian-3d,2,3,1’
A spherical Gaussian kernel with FWHM of 2 pixels and
truncated at 3 times the FWHM.
Of course, if a specific kernel is needed that does not fit the
constraints imposed by this option, you can always use a catalog to
define any arbitrary kernel. Just call the ‘--individual’ and
‘--nomerged’ options to make sure that it is built as a separate
file (individually) and no "merged" image of the input profiles is
created.
‘-x INT,INT’
‘--mergedsize=INT,INT’
The number of pixels along each axis of the output, in FITS order.
This is before over-sampling. For example, if you call
MakeProfiles with ‘--mergedsize=100,150 --oversample=5’ (assuming
no shift due for later convolution), then the final image size
along the first axis will be 500 by 750 pixels. Fractions are
acceptable as values for each dimension, however, they must reduce
to an integer, so ‘--mergedsize=150/3,300/3’ is acceptable but
‘--mergedsize=150/4,300/4’ is not.
When viewing a FITS image in DS9, the first FITS dimension is in
the horizontal direction and the second is vertical. As an
example, the image created with the example above will have 500
pixels horizontally and 750 pixels vertically.
If a background image is specified, this option is ignored.
‘-s INT’
‘--oversample=INT’
The scale to over-sample the profiles and final image. If not an
odd number, will be added by one, see *note Oversampling::. Note
that this ‘--oversample’ will remain active even if an input image
is specified. If your input catalog is based on the background
image, be sure to set ‘--oversample=1’.
‘--psfinimg’
Build the possibly existing PSF profiles (Moffat or Gaussian) in
the catalog into the final image. By default they are built
separately so you can convolve your images with them, thus their
magnitude and positions are ignored. With this option, they will
be built in the final image like every other galaxy profile. To
have a final PSF in your image, make a point profile where you want
the PSF and after convolution it will be the PSF.
‘-i’
‘--individual’
If this option is called, each profile is created in a separate
FITS file within the same directory as the output and the row
number of the profile (starting from zero) in the name. The file
for each row's profile will be in the same directory as the final
combined image of all the profiles and will have the final image's
name as a suffix. So for example, if the final combined image is
named ‘./out/fromcatalog.fits’, then the first profile that will be
created with this option will be named ‘./out/0_fromcatalog.fits’.
Since each image only has one full profile out to the truncation
radius the profile is centered and so, only the sub-pixel position
of the profile center is important for the outputs of this option.
The output will have an odd number of pixels. If there is no
oversampling, the central pixel will contain the profile center.
If the value to ‘--oversample’ is larger than unity, then the
profile center is on any of the central ‘--oversample’'d pixels
depending on the fractional value of the profile center.
If the fractional value is larger than half, it is on the bottom
half of the central region. This is due to the FITS definition of
a real number position: The center of a pixel has fractional value
$0.00$ so each pixel contains these fractions: .5 - .75 - .00
(pixel center) - .25 - .5.
‘-m’
‘--nomerged’
Do not make a merged image. By default after making the profiles,
they are added to a final image with side lengths specified by
‘--mergedsize’ if they overlap with it.
The options below can be used to define the world coordinate system
(WCS) properties of the MakeProfiles outputs. The option names are
deliberately chosen to be the same as the FITS standard WCS keywords.
See Section 8 of Pence et al [2010]
(https://doi.org/10.1051/0004-6361/201015362) for a short introduction
to WCS in the FITS standard(1).
If you look into the headers of a FITS image with WCS for example,
you will see all these names but in uppercase and with numbers to
represent the dimensions, for example, ‘CRPIX1’ and ‘PC2_1’. You can
see the FITS headers with Gnuastro's *note Fits:: program using a
command like this: ‘$ astfits -p image.fits’.
If the values given to any of these options does not correspond to
the number of dimensions in the output dataset, then no WCS information
will be added. Also recall that if you use the ‘--background’ option,
all of these options are ignored. Such that if the image given to
‘--background’ does not have any WCS, the output of MakeProfiles will
also not have any WCS, even if these options are given(2).
‘--crpix=FLT,FLT’
The pixel coordinates of the WCS reference point. Fractions are
acceptable for the values of this option.
‘--crval=FLT,FLT’
The WCS coordinates of the Reference point. Fractions are
acceptable for the values of this option. The comma-separated
values can either be in degrees (a single number), or sexagesimal
(‘_h_m_’ for RA, ‘_d_m_’ for Dec, or ‘_:_:_’ for both). In any
case, the final value that will be written in the ‘CRVAL’ keyword
will be a floating point number in degrees (according to the FITS
standard).
‘--cdelt=FLT,FLT’
The resolution (size of one data-unit or pixel in WCS units) of the
non-oversampled dataset. Fractions are acceptable for the values
of this option.
‘--pc=FLT,FLT,FLT,FLT’
The PC matrix of the WCS rotation, see the FITS standard (link
above) to better understand the PC matrix.
‘--cunit=STR,STR’
The units of each WCS axis, for example, ‘deg’. Note that these
values are part of the FITS standard (link above). MakeProfiles
will not complain if you use non-standard values, but later usage
of them might cause trouble.
‘--ctype=STR,STR’
The type of each WCS axis, for example, ‘RA---TAN’ and ‘DEC--TAN’.
Note that these values are part of the FITS standard (link above).
MakeProfiles will not complain if you use non-standard values, but
later usage of them might cause trouble.
---------- Footnotes ----------
(1) The world coordinate standard in FITS is a very beautiful and
powerful concept to link/associate datasets with the outside world
(other datasets). The description in the FITS standard (link above)
only touches the tip of the ice-burg. To learn more please see Greisen
and Calabretta [2002] (https://doi.org/10.1051/0004-6361:20021326),
Calabretta and Greisen [2002]
(https://doi.org/10.1051/0004-6361:20021327), Greisen et al. [2006]
(https://doi.org/10.1051/0004-6361:20053818), and Calabretta et al.
(http://www.atnf.csiro.au/people/mcalabre/WCS/dcs_20040422.pdf)
(2) If you want to add profiles _and_ WCS over the background image
(to produce your output), you need more than one command: 1. You should
use ‘--mergedsize’ in MakeProfiles to manually set the output number of
pixels equal to your desired background image (so the background is
zero). In this mode, you can use these WCS-related options to define
the WCS. 2. Then use Arithmetic to add the pixels of your mock image to
the background (see *note Arithmetic::.
8.1.4.4 MakeProfiles log file
.............................
Besides the final merged dataset of all the profiles, or the individual
datasets (see *note MakeProfiles output dataset::), if the ‘--log’
option is called MakeProfiles will also create a log file in the current
directory (where you run MockProfiles). See *note Common options:: for
a full description of ‘--log’ and other options that are shared between
all Gnuastro programs. The values for each column are explained in the
first few commented lines of the log file (starting with ‘#’ character).
Here is a more complete description.
• An ID (row number of profile in input catalog).
• The total magnitude of the profile in the output dataset. When the
profile does not completely overlap with the output dataset, this
will be different from your input magnitude.
• The number of pixels (in the oversampled image) which used Monte
Carlo integration and not the central pixel value, see *note
Sampling from a function::.
• The fraction of flux in the Monte Carlo integrated pixels.
• If an individual image was created, this column will have a value
of ‘1’, otherwise it will have a value of ‘0’.
9 High-level calculations
*************************
After the reduction of raw data (for example, with the programs in *note
Data manipulation::) you will have reduced images/data ready for
processing/analyzing (for example, with the programs in *note Data
analysis::). But the processed/analyzed data (or catalogs) are still
not enough to derive any scientific result. Even higher-level analysis
is still needed to convert the observed magnitudes, sizes or volumes
into physical quantities that we associate with each catalog entry or
detected object which is the purpose of the tools in this section.
9.1 CosmicCalculator
====================
To derive higher-level information regarding our sources in
extra-galactic astronomy, cosmological calculations are necessary. In
Gnuastro, CosmicCalculator is in charge of such calculations. Before
discussing how CosmicCalculator is called and operates (in *note
Invoking astcosmiccal::), it is important to provide a rough but mostly
self sufficient review of the basics and the equations used in the
analysis. In *note Distance on a 2D curved space:: the basic idea of
understanding distances in a curved and expanding 2D universe (which we
can visualize) are reviewed. Having solidified the concepts there, in
*note Extending distance concepts to 3D::, the formalism is extended to
the 3D universe we are trying to study in our research.
The focus here is obtaining a physical insight into these equations
(mainly for the use in real observational studies). There are many
books thoroughly deriving and proving all the equations with all
possible initial conditions and assumptions for any abstract universe,
interested readers can study those books.
9.1.1 Distance on a 2D curved space
-----------------------------------
The observations to date (for example, the Planck 2015 results), have
not measured(1) the presence of significant curvature in the universe.
However to be generic (and allow its measurement if it does in fact
exist), it is very important to create a framework that allows non-zero
uniform curvature. However, this section is not intended to be a fully
thorough and mathematically complete derivation of these concepts.
There are many references available for such reviews that go deep into
the abstract mathematical proofs. The emphasis here is on visualization
of the concepts for a beginner.
As 3D beings, it is difficult for us to mentally create (visualize) a
picture of the curvature of a 3D volume. Hence, here we will assume a
2D surface/space and discuss distances on that 2D surface when it is
flat and when it is curved. Once the concepts have been
created/visualized here, we will extend them, in *note Extending
distance concepts to 3D::, to a real 3D spatial _slice_ of the Universe
we live in and hope to study.
To be more understandable (actively discuss from an observer's point
of view) let's assume there's an imaginary 2D creature living on the 2D
space (which _might_ be curved in 3D). Here, we will be working with
this creature in its efforts to analyze distances in its 2D universe.
The start of the analysis might seem too mundane, but since it is
difficult to imagine a 3D curved space, it is important to review all
the very basic concepts thoroughly for an easy transition to a universe
that is more difficult to visualize (a curved 3D space embedded in 4D).
To start, let's assume a static (not expanding or shrinking), flat 2D
surface similar to *note Figure 9.1: flatplane. and that the 2D creature
is observing its universe from point $A$. One of the most basic ways to
parameterize this space is through the Cartesian coordinates ($x$, $y$).
In *note Figure 9.1: flatplane, the basic axes of these two coordinates
are plotted. An infinitesimal change in the direction of each axis is
written as $dx$ and $dy$. For each point, the infinitesimal changes are
parallel with the respective axes and are not shown for clarity.
Another very useful way of parameterizing this space is through polar
coordinates. For each point, we define a radius ($r$) and angle
($\phi$) from a fixed (but arbitrary) reference axis. In *note Figure
9.1: flatplane. the infinitesimal changes for each polar coordinate are
plotted for a random point and a dashed circle is shown for all points
with the same radius.
../gnuastro-figures//flatplane.eps
Figure 9.1: Two dimensional Cartesian and polar coordinates on a flat
plane.
Assuming an object is placed at a certain position, which can be
parameterized as $(x,y)$, or $(r,\phi)$, a general infinitesimal change
in its position will place it in the coordinates $(x+dx,y+dy)$, or
$(r+dr,\phi+d\phi)$. The distance (on the flat 2D surface) that is
covered by this infinitesimal change in the static universe ($ds_s$, the
subscript signifies the static nature of this universe) can be written
as:
$$ds_s^2=dx^2+dy^2=dr^2+r^2d\phi^2$$
The main question is this: how can the 2D creature incorporate the
(possible) curvature in its universe when it's calculating distances?
The universe that it lives in might equally be a curved surface like
*note Figure 9.2: sphereandplane. The answer to this question but for a
3D being (us) is the whole purpose to this discussion. Here, we want to
give the 2D creature (and later, ourselves) the tools to measure
distances if the space (that hosts the objects) is curved.
*note Figure 9.2: sphereandplane. assumes a spherical shell with
radius $R$ as the curved 2D plane for simplicity. The 2D plane is
tangent to the spherical shell and only touches it at $A$. This idea
will be generalized later. The first step in measuring the distance in
a curved space is to imagine a third dimension along the $z$ axis as
shown in *note Figure 9.2: sphereandplane. For simplicity, the $z$ axis
is assumed to pass through the center of the spherical shell. Our
imaginary 2D creature cannot visualize the third dimension or a curved
2D surface within it, so the remainder of this discussion is purely
abstract for it (similar to us having difficulty in visualizing a 3D
curved space in 4D). But since we are 3D creatures, we have the
advantage of visualizing the following steps. Fortunately the 2D
creature is already familiar with our mathematical constructs, so it can
follow our reasoning.
With the third axis added, a generic infinitesimal change over _the
full_ 3D space corresponds to the distance:
$$ds_s^2=dx^2+dy^2+dz^2=dr^2+r^2d\phi^2+dz^2.$$
../gnuastro-figures//sphereandplane.eps
Figure 9.2: 2D spherical shell (centered on $O$) and flat plane (light
gray) tangent to it at point $A$.
It is very important to recognize that this change of distance is for
_any_ point in the 3D space, not just those changes that occur on the 2D
spherical shell of *note Figure 9.2: sphereandplane. Recall that our 2D
friend can only do measurements on the 2D surfaces, not the full 3D
space. So we have to constrain this general change to any change on the
2D spherical shell. To do that, let's look at the arbitrary point $P$
on the 2D spherical shell. Its image ($P'$) on the flat plain is also
displayed. From the dark gray triangle, we see that
$$\sin\theta={r\over R},\quad\cos\theta={R-z\over R}.$$These relations allow the 2D creature to find the value of $z$ (an abstract dimension for it) as a function of r (distance on a flat 2D plane, which it can visualize) and thus eliminate $z$.
From $\sin^2\theta+\cos^2\theta=1$, we get $z^2-2Rz+r^2=0$ and
solving for $z$, we find:
$$z=R\left(1\pm\sqrt{1-{r^2\over R^2}}\right).$$
The $\pm$ can be understood from *note Figure 9.2: sphereandplane.:
For each $r$, there are two points on the sphere, one in the upper
hemisphere and one in the lower hemisphere. An infinitesimal change in
$r$, will create the following infinitesimal change in $z$:
$$dz={\mp r\over R}\left(1\over
\sqrt{1-{r^2/R^2}}\right)dr.$$Using the positive signed equation instead of $dz$ in the $ds_s^2$ equation above, we get:
$$ds_s^2={dr^2\over 1-r^2/R^2}+r^2d\phi^2.$$
The derivation above was done for a spherical shell of radius $R$ as
a curved 2D surface. To generalize it to any surface, we can define
$K=1/R^2$ as the curvature parameter. Then the general infinitesimal
change in a static universe can be written as:
$$ds_s^2={dr^2\over 1-Kr^2}+r^2d\phi^2.$$
Therefore, when $K>0$ (and curvature is the same everywhere), we have
a finite universe, where $r$ cannot become larger than $R$ as in *note
Figure 9.2: sphereandplane. When $K=0$, we have a flat plane (*note
Figure 9.1: flatplane.) and a negative $K$ will correspond to an
imaginary $R$. The latter two cases may be infinite in area (which is
not a simple concept, but mathematically can be modeled with $r$
extending infinitely), or finite-area (like a cylinder is flat
everywhere with $ds_s^2={dx^2 + dy^2}$, but finite in one direction in
size).
A very important issue that can be discussed now (while we are still
in 2D and can actually visualize things) is that $\overrightarrow{r}$ is
tangent to the curved space at the observer's position. In other words,
it is on the gray flat surface of *note Figure 9.2: sphereandplane, even
when the universe if curved: $\overrightarrow{r}=P'-A$. Therefore for
the point $P$ on a curved space, the raw coordinate $r$ is the distance
to $P'$, not $P$. The distance to the point $P$ (at a specific
coordinate $r$ on the flat plane) over the curved surface (thick line in
*note Figure 9.2: sphereandplane.) is called the _proper distance_ and
is displayed with $l$. For the specific example of *note Figure 9.2:
sphereandplane, the proper distance can be calculated with: $l=R\theta$
($\theta$ is in radians). Using the $\sin\theta$ relation found above,
we can find $l$ as a function of $r$:
$$\theta=\sin^{-1}\left({r\over R}\right)\quad\rightarrow\quad
l(r)=R\sin^{-1}\left({r\over R}\right)$$
$R$ is just an arbitrary constant and can be directly found from $K$,
so for cleaner equations, it is common practice to set $R=1$, which
gives: $l(r)=\sin^{-1}r$. Also note that when $R=1$, then $l=\theta$.
Generally, depending on the curvature, in a _static_ universe the proper
distance can be written as a function of the coordinate $r$ as (from now
on we are assuming $R=1$):
$$l(r)=\sin^{-1}(r)\quad(K>0),\quad\quad
l(r)=r\quad(K=0),\quad\quad l(r)=\sinh^{-1}(r)\quad(K<0).$$With
$l$, the infinitesimal change of distance can be written in a more
simpler and abstract form of
$$ds_s^2=dl^2+r^2d\phi^2.$$
Until now, we had assumed a static universe (not changing with time).
But our observations so far appear to indicate that the universe is
expanding (it is not static). Since there is no reason to expect the
observed expansion is unique to our particular position of the universe,
we expect the universe to be expanding at all points with the same rate
at the same time. Therefore, to add a time dependence to our distance
measurements, we can include a multiplicative scaling factor, which is a
function of time: $a(t)$. The functional form of $a(t)$ comes from the
cosmology, the physics we assume for it: general relativity, and the
choice of whether the universe is uniform ('homogeneous') in density and
curvature or inhomogeneous. In this section, the functional form of
$a(t)$ is irrelevant, so we can avoid these issues.
With this scaling factor, the proper distance will also depend on
time. As the universe expands, the distance between two given points
will shift to larger values. We thus define a distance measure, or
coordinate, that is independent of time and thus does not 'move'. We
call it the _comoving distance_ and display with $\chi$ such that:
$l(r,t)=\chi(r)a(t)$. We have therefore, shifted the $r$ dependence of
the proper distance we derived above for a static universe to the
comoving distance:
$$\chi(r)=\sin^{-1}(r)\quad(K>0),\quad\quad
\chi(r)=r\quad(K=0),\quad\quad \chi(r)=\sinh^{-1}(r)\quad(K<0).$$
Therefore, $\chi(r)$ is the proper distance to an object at a
specific reference time: $t=t_r$ (the $r$ subscript signifies
"reference") when $a(t_r)=1$. At any arbitrary moment ($t\neq{t_r}$)
before or after $t_r$, the proper distance to the object can be scaled
with $a(t)$.
Measuring the change of distance in a time-dependent (expanding)
universe only makes sense if we can add up space and time(2). But we
can only add bits of space and time together if we measure them in the
same units: with a conversion constant (similar to how 1000 is used to
convert a kilometer into meters). Experimentally, we find strong
support for the hypothesis that this conversion constant is the speed of
light (or gravitational waves(3)) in a vacuum. This speed is postulated
to be constant(4) and is almost always written as $c$. We can thus
parameterize the change in distance on an expanding 2D surface as
$$ds^2=c^2dt^2-a^2(t)ds_s^2 = c^2dt^2-a^2(t)(d\chi^2+r^2d\phi^2).$$
---------- Footnotes ----------
(1) The observations are interpreted under the assumption of uniform
curvature. For a relativistic alternative to dark energy (and maybe
also some part of dark matter), non-uniform curvature may be even be
more critical, but that is beyond the scope of this brief explanation.
(2) In other words, making our space-time consistent with Minkowski
space-time geometry. In this geometry, different observers at a given
point (event) in space-time split up space-time into 'space' and 'time'
in different ways, just like people at the same spatial position can
make different choices of splitting up a map into 'left-right' and
'up-down'. This model is well supported by twentieth and twenty-first
century observations.
(3) The speed of gravitational waves was recently found to be very
similar to that of light in vacuum, see LIGO Collaboration 2017
(https://arxiv.org/abs/1710.05834).
(4) In _natural units_, speed is measured in units of the speed of
light in vacuum.
9.1.2 Extending distance concepts to 3D
---------------------------------------
The concepts of *note Distance on a 2D curved space:: are here extended
to a 3D space that _might_ be curved. We can start with the generic
infinitesimal distance in a static 3D universe, but this time in
spherical coordinates instead of polar coordinates. $\theta$ is shown
in *note Figure 9.2: sphereandplane, but here we are 3D beings,
positioned on $O$ (the center of the sphere) and the point $O$ is
tangent to a 4D-sphere. In our 3D space, a generic infinitesimal
displacement will correspond to the following distance in spherical
coordinates:
$$ds_s^2=dx^2+dy^2+dz^2=dr^2+r^2(d\theta^2+\sin^2{\theta}d\phi^2).$$
Like the 2D creature before, we now have to assume an abstract
dimension which we cannot visualize easily. Let's call the fourth
dimension $w$, then the general change in coordinates in the _full_ four
dimensional space will be:
$$ds_s^2=dr^2+r^2(d\theta^2+\sin^2{\theta}d\phi^2)+dw^2.$$
But we can only work on a 3D curved space, so following exactly the same
steps and conventions as our 2D friend, we arrive at:
$$ds_s^2={dr^2\over 1-Kr^2}+r^2(d\theta^2+\sin^2{\theta}d\phi^2).$$
In a non-static universe (with a scale factor a(t)), the distance can be
written as:
$$ds^2=c^2dt^2-a^2(t)[d\chi^2+r^2(d\theta^2+\sin^2{\theta}d\phi^2)].$$
9.1.3 Invoking CosmicCalculator
-------------------------------
CosmicCalculator will calculate cosmological variables based on the
input parameters. The executable name is ‘astcosmiccal’ with the
following general template
$ astcosmiccal [OPTION...] ...
One line examples:
## Print basic cosmological properties at redshift 2.5:
$ astcosmiccal -z2.5
## Only print Comoving volume over 4pi stradian to z (Mpc^3):
$ astcosmiccal --redshift=0.8 --volume
## Print redshift and age of universe when Lyman-alpha line is
## at 6000 angstrom (another way to specify redshift).
$ astcosmiccal --obsline=Ly-alpha,6000 --age
## Print luminosity distance, angular diameter distance and age
## of universe in one row at redshift 0.4
$ astcosmiccal -z0.4 -LAg
## Assume Lambda and matter density of 0.7 and 0.3 and print
## basic cosmological parameters for redshift 2.1:
$ astcosmiccal -l0.7 -m0.3 -z2.1
## Print wavelength of all pre-defined spectral lines when
## Lyman-alpha is observed at 4000 Angstroms.
$ astcosmiccal --obsline=Ly-alpha,4000 --listlinesatz
The input parameters (current matter density, etc.) can be given as
command-line options or in the configuration files, see *note
Configuration files::. For a definition of the different parameters,
please see the sections prior to this. If no redshift is given,
CosmicCalculator will just print its input parameters and abort. For a
full list of the input options, please see *note CosmicCalculator input
options::.
Without any particular output requested (and only a given redshift),
CosmicCalculator will print all basic cosmological calculations (one per
line) with some explanations before each. This can be good when you
want a general feeling of the conditions at a specific redshift.
Alternatively, if any specific calculation(s) are requested (its
possible to call more than one), only the requested value(s) will be
calculated and printed with one character space between them. In this
case, no description or units will be printed. See *note
CosmicCalculator basic cosmology calculations:: for the full list of
these options along with some explanations how when/how they can be
useful.
Another common operation in observational cosmology is dealing with
spectral lines at different redshifts. CosmicCalculator also has
features to help in such situations, please see *note CosmicCalculator
spectral line calculations::.
9.1.3.1 CosmicCalculator input options
......................................
The inputs to CosmicCalculator can be specified with the following
options:
‘-z FLT’
‘--redshift=FLT’
The redshift of interest. There are two other ways that you can
specify the target redshift: 1) Spectral lines and their observed
wavelengths, see ‘--obsline’. 2) Velocity, see ‘--velocity’.
Hence this option cannot be called with ‘--obsline’ or
‘--velocity’.
‘-y FLT’
‘--velocity=FLT’
Input velocity in km/s. The given value will be converted to
redshift internally, and used in any subsequent calculation. This
option is thus an alternative to ‘--redshift’ or ‘--obsline’, it
cannot be used with them. The conversion will be done with the
more general and accurate relativistic equation of
$1+z=\sqrt{(c+v)/(c-v)}$, not the simplified $z\approx v/c$.
‘-H FLT’
‘--H0=FLT’
Current expansion rate (in km sec$^{-1}$ Mpc$^{-1}$).
‘-l FLT’
‘--olambda=FLT’
Cosmological constant density divided by the critical density in
the current Universe ($\Omega_{\Lambda,0}$).
‘-m FLT’
‘--omatter=FLT’
Matter (including massive neutrinos) density divided by the
critical density in the current Universe ($\Omega_{m,0}$).
‘-r FLT’
‘--oradiation=FLT’
Radiation density divided by the critical density in the current
Universe ($\Omega_{r,0}$).
‘-O STR/FLT,FLT’
‘--obsline=STR/FLT,FLT’
Find the redshift to use in next steps based on the rest-frame and
observed wavelengths of a line. This option is thus an alternative
to ‘--redshift’ or ‘--velocity’, it cannot be used with them.
The first argument identifies the line. It can be one of the
standard names, or any rest-frame wavelength in Angstroms. The
second argument is the observed wavelength of that line. For
example, ‘--obsline=Ly-alpha,6000’ is the same as
‘--obsline=1215.64,6000’. Wavelengths are assumed to be in
Angstroms by default (other units can be selected with
‘--lineunit’, see *note CosmicCalculator spectral line
calculations::).
The list of pre-defined names for the lines in Gnuastro's database
is available by running
$ astcosmiccal --listlines
9.1.3.2 CosmicCalculator basic cosmology calculations
.....................................................
By default, when no specific calculations are requested,
CosmicCalculator will print a complete set of all its calculators (one
line for each calculation, see *note Invoking astcosmiccal::). The full
list of calculations can be useful when you do not want any specific
value, but just a general view. In other contexts (for example, in a
batch script or during a discussion), you know exactly what you want and
do not want to be distracted by all the extra information.
You can use any number of the options described below in any order.
When any of these options are requested, CosmicCalculator's output will
just be a single line with a single space between the (possibly)
multiple values. In the example below, only the tangential distance
along one arc-second (in kpc), absolute magnitude conversion, and age of
the universe at redshift 2 are printed (recall that you can merge short
options together, see *note Options::).
$ astcosmiccal -z2 -sag
8.585046 44.819248 3.289979
Here is one example of using this feature in scripts: by adding the
following two lines in a script to keep/use the comoving volume with
varying redshifts:
z=3.12
vol=$(astcosmiccal --redshift=$z --volume)
In a script, this operation might be necessary for a large number of
objects (several of galaxies in a catalog for example). So the fact
that all the other default calculations are ignored will also help you
get to your result faster.
If you are indeed dealing with many (for example, thousands) of
redshifts, using CosmicCalculator is not the best/fastest solution.
Because it has to go through all the configuration files and
preparations for each invocation. To get the best efficiency (least
overhead), we recommend using Gnuastro's cosmology library (see *note
Cosmology library::). CosmicCalculator also calls the library functions
defined there for its calculations, so you get the same result with no
overhead. Gnuastro also has libraries for easily reading tables into a
C program, see *note Table input output::. Afterwards, you can easily
build and run your C program for the particular processing with *note
BuildProgram::.
If you just want to inspect the value of a variable visually, the
description (which comes with units) might be more useful. In such
cases, the following command might be better. The other calculations
will also be done, but they are so fast that you will not notice on
modern computers (the time it takes your eye to focus on the result is
usually longer than the processing: a fraction of a second).
$ astcosmiccal --redshift=0.832 | grep volume
The full list of CosmicCalculator's specific calculations is present
below in two groups: basic cosmology calculations and those related to
spectral lines. In case you have forgot the units, you can use the
‘--help’ option which has the units along with a short description.
‘-e’
‘--usedredshift’
The redshift that was used in this run. In many cases this is the
main input parameter to CosmicCalculator, but it is useful in
others. For example, in combination with ‘--obsline’ (where you
give an observed and rest-frame wavelength and would like to know
the redshift) or with ‘--velocity’ (where you specify the velocity
instead of redshift). Another example is when you run
CosmicCalculator in a loop, while changing the redshift and you
want to keep the redshift value with the resulting calculation.
‘-Y’
‘--usedvelocity’
The velocity (in km/s) that was used in this run. The conversion
from redshift will be done with the more general and accurate
relativistic equation of $1+z=\sqrt{(c+v)/(c-v)}$, not the
simplified $z\approx v/c$.
‘-G’
‘--agenow’
The current age of the universe (given the input parameters) in Ga
(Giga annum, or billion years).
‘-C’
‘--criticaldensitynow’
The current critical density (given the input parameters) in grams
per cubic centimeter ($g/cm^3$).
‘-d’
‘--properdistance’
The proper distance (at the current time, i.e. in comoving units)
to an object at the given redshift in Megaparsecs (Mpc). See *note
Distance on a 2D curved space:: for a description of the proper
distance.
‘-A’
‘--angulardiamdist’
The angular diameter distance to an object at a given redshift in
Megaparsecs (Mpc).
‘-s’
‘--arcsectandist’
The tangential distance covered by 1 arc-second at a given redshift
in physical (not comoving) kilo-parsecs (kpc). This can be useful
when trying to estimate the resolution or pixel scale of an
instrument (usually in units of arc-seconds) required for a galaxy
of a given physical size at a given redshift.
For an arc subtending one degree at a high redshift $z$,
multiplying the result by 3600 will, of course, give the
(tangential) length of an arc subtending one degree, but it will
still be in physical units. However, at one degree, the comoving
separation for redshifts from 1 to 10 is about 100 Mpc give or take
50% or so for nearly $\Lambda{}$CDM models. In other words, this
is the cosmic web scale, where comoving units usually make more
sense than physical units, e.g. for large-scale structure
correlation functions or the baryon acoustic oscillation scale. If
comoving units are appropriate, as will typically be the case for
the degree scale, you must also multiply by $(1+z)$.
‘-L’
‘--luminositydist’
The luminosity distance to object at given redshift in Megaparsecs
(Mpc).
‘-u’
‘--distancemodulus’
The distance modulus at given redshift.
‘-a’
‘--absmagconv’
The conversion factor (addition) to absolute magnitude. Note that
this is practically the distance modulus added with
$-2.5\log{(1+z)}$ for the desired redshift based on the input
parameters. Once the apparent magnitude and redshift of an object
is known, this value may be added with the apparent magnitude to
give the object's absolute magnitude.
‘-g’
‘--age’
Age of the universe at given redshift in Ga (Giga-annum, or billion
years).
‘-b’
‘--lookbacktime’
The look-back time to a given redshift in Ga (Giga annum, or
billion years). The look-back time at a given redshift is defined
as the current age of the universe (‘--agenow’) minus the age of
the universe at the given redshift.
‘-c’
‘--criticaldensity’
The critical density at given redshift in grams per centimeter-cube
($g/cm^3$).
‘-v’
‘--onlyvolume’
The comoving volume in cubic Megaparsecs (Mpc$^3$) through to the
desired redshift, based on the input parameters.
9.1.3.3 CosmicCalculator spectral line calculations
...................................................
At different redshifts, observed spectral lines are shifted compared to
their rest frame wavelengths with this simple relation:
$\lambda_{obs}=\lambda_{rest}(1+z)$. Although this relation is very
simple and can be done for one line in the head (or a simple
calculator!), it slowly becomes tiring when dealing with a lot of lines
or redshifts, or some precision is necessary. The options in this
section are thus provided to greatly simplify usage of this simple
equation, and also helping by storing a list of pre-defined spectral
line wavelengths.
For example, if you want to know the wavelength of the $H\alpha$ line
(at 6562.8 Angstroms in rest frame), when $Ly\alpha$ is at 8000
Angstroms, you can call CosmicCalculator like the first example below.
And if you want the wavelength of all pre-defined spectral lines at this
redshift, you can use the second command.
$ astcosmiccal --obsline=lyalpha,8000 --lineatz=halpha
$ astcosmiccal --obsline=lyalpha,8000 --listlinesatz
Bellow you can see the printed/output calculations of
CosmicCalculator that are related to spectral lines. Note that
‘--obsline’ is an input parameter, so it is discussed (with the full
list of known lines) in *note CosmicCalculator input options::.
‘--listlines’
List the pre-defined rest frame spectral line wavelengths and their
names on standard output, then abort CosmicCalculator. The units
of the displayed wavelengths for each line can be determined with
‘--lineunit’ (see below).
When this option is given, other operations on the command-line
will be ignored. This is convenient when you forget the specific
name of the spectral line used within Gnuastro, or when you forget
the exact wavelength of a certain line.
These names can be used with the options that deal with spectral
lines, for example, ‘--obsline’ and ‘--lineatz’ (*note
CosmicCalculator basic cosmology calculations::).
The format of the output list is a two-column table, with
Gnuastro's text table format (see *note Gnuastro text table
format::). Therefore, if you are only looking for lines in a
specific range, you can pipe the output into Gnuastro's table
program and use its ‘--range’ option on the ‘wavelength’ (first)
column. For example, if you only want to see the lines between
4000 and 6000 Angstroms, you can run this command:
$ astcosmiccal --listlines \
| asttable --range=wavelength,4000,6000
And if you want to use the list later and have it as a table in a
file, you can easily add the ‘--output’ (or ‘-o’) option to the
‘asttable’ command, and specify the filename, for example,
‘--output=lines.fits’ or ‘--output=lines.txt’.
‘--listlinesatz’
Similar to ‘--listlines’ (above), but the printed wavelength is not
in the rest frame, but redshifted to the given redshift. Recall
that the redshift can be specified by ‘--redshift’ directly or by
‘--obsline’, see *note CosmicCalculator input options::. For an
example usage of this option, see *note Viewing spectra and
redshifted lines::.
‘-i STR/FLT’
‘--lineatz=STR/FLT’
The wavelength of the specified line at the redshift given to
CosmicCalculator. The line can be specified either by its name or
directly as a number (its wavelength). The units of the displayed
wavelengths for each line can be determined with ‘--lineunit’ (see
below).
To get the list of pre-defined names for the lines and their
wavelength, you can use the ‘--listlines’ option, see *note
CosmicCalculator input options::. In the former case (when a name
is given), the returned number is in units of Angstroms. In the
latter (when a number is given), the returned value is the same
units of the input number (assuming it is a wavelength).
‘--lineunit=STR’
The units to display line wavelengths above. It can take the
following four values. If you need any other unit, please contact
us at ‘bug-gnuastro@gnu.org’.
‘m’
Meter.
‘micro-m’
Micrometer or $10^{-6}m$.
‘nano-m’
Nanometer, or $10^{-9}m$.
‘angstrom’
Angstrom or $10^{-10}m$; the default unit when this option is
not called.
10 Installed scripts
********************
Gnuastro's programs (introduced in previous chapters) are designed to be
highly modular and thus contain lower-level operations on the data.
However, in many contexts, certain higher-level are also shared between
many contexts. For example, a sequence of calls to multiple Gnuastro
programs, or a special way of running a program and treating the output.
To facilitate such higher-level data analysis, Gnuastro also installs
some scripts on your system with the (‘astscript-’) prefix (in contrast
to the other programs that only have the ‘ast’ prefix).
Like all of Gnuastro's source code, these scripts are also heavily
commented. They are written in portable shell scripts (command-line
environments), which does not need compilation. Therefore, if you open
the installed scripts in a text editor, you can actually read them(1).
For example, with this command (just replace ‘nano’ with your favorite
text editor, like ‘emacs’ or ‘vim’):
$ nano $(which astscript-NAME)
Shell scripting is the same language that you use when typing on the
command-line. Therefore shell scripting is much more widely known and
used compared to C (the language of other Gnuastro programs). Because
Gnuastro's installed scripts do higher-level operations, customizing
these scripts for a special project will be more common than the
programs.
These scripts also accept options and are in many ways similar to the
programs (see *note Common options::) with some minor differences:
• Currently they do not accept configuration files themselves.
However, the configuration files of the Gnuastro programs they call
are indeed parsed and used by those programs.
As a result, they do not have the following options:
‘--checkconfig’, ‘--config’, ‘--lastconfig’, ‘--onlyversion’,
‘--printparams’, ‘--setdirconf’ and ‘--setusrconf’.
• They do not directly allocate any memory, so there is no
‘--minmapsize’.
• They do not have an independent ‘--usage’ option: when called with
‘--usage’, they just recommend running ‘--help’.
• The output of ‘--help’ is not configurable like the programs (see
*note --help::).
• The scripts will commonly use your installed shell and other basic
command-line tools (for example, AWK or SED). Different systems
have different versions and implementations of these basic tools
(for example, GNU/Linux systems use GNU Bash, GNU AWK and GNU SED
which are far more advanced and up to date then the minimalist AWK
and SED of most other systems). Therefore, unexpected errors in
these tools might come up when you run these scripts on
non-GNU/Linux operating systems. If you do confront such strange
errors, please submit a bug report so we fix it as soon as possible
(see *note Report a bug::).
---------- Footnotes ----------
(1) Gnuastro's installed programs (those only starting with ‘ast’)
are not human-readable. They are written in C and need to be compiled
before execution. Compilation optimizes the steps into the low-level
hardware CPU instructions/language to improve efficiency. Because
compiled programs do not need an interpreter like Bash on every run,
they are much faster and more independent than scripts. To read the
source code of the programs, look into the ‘bin/progname’ directory of
Gnuastro's source (*note Downloading the source::). If you would like
to read more about why C was chosen for the programs, please see *note
Why C::.
10.1 Sort FITS files by night
=============================
FITS images usually contain (several) keywords for preserving important
dates. In particular, for lower-level data, this is usually the
observation date and time (for example, stored in the ‘DATE-OBS’ keyword
value). When analyzing observed datasets, many calibration steps (like
the dark, bias or flat-field), are commonly calculated on a
per-observing-night basis.
However, the FITS standard's date format (‘YYYY-MM-DDThh:mm:ss.ddd’)
is based on the western (Gregorian) calendar. Dates that are stored in
this format are complicated for automatic processing: a night starts in
the final hours of one calendar day, and extends to the early hours of
the next calendar day. As a result, to identify datasets from one
night, we commonly need to search for two dates. However calendar
peculiarities can make this identification very difficult. For example,
when an observation is done on the night separating two months (like the
night starting on March 31st and going into April 1st), or two years
(like the night starting on December 31st 2018 and going into January
1st, 2019). To account for such situations, it is necessary to keep
track of how many days are in a month, and leap years, etc.
Gnuastro's ‘astscript-sort-by-night’ script is created to help in
such important scenarios. It uses *note Fits:: to convert the FITS date
format into the Unix epoch time (number of seconds since 00:00:00 of
January 1st, 1970), using the ‘--datetosec’ option. The Unix epoch time
is a single number (integer, if not given in sub-second precision),
enabling easy comparison and sorting of dates after January 1st, 1970.
You can use this script as a basis for making a much more highly
customized sorting script. Here are some examples
• If you need to copy the files, but only need a single extension
(not the whole file), you can add a step just before the making of
the symbolic links, or copies, and change it to only copy a certain
extension of the FITS file using the Fits program's ‘--copy’
option, see *note HDU information and manipulation::.
• If you need to classify the files with finer detail (for example,
the purpose of the dataset), you can add a step just before the
making of the symbolic links, or copies, to specify a file-name
prefix based on other certain keyword values in the files. For
example, when the FITS files have a keyword to specify if the
dataset is a science, bias, or flat-field image. You can read it
and to add a ‘sci-’, ‘bias-’, or ‘flat-’ to the created file (after
the ‘--prefix’) automatically.
For example, let's assume the observing mode is stored in the
hypothetical ‘MODE’ keyword, which can have three values of
‘BIAS-IMAGE’, ‘SCIENCE-IMAGE’ and ‘FLAT-EXP’. With the step below,
you can generate a mode-prefix, and add it to the generated
link/copy names (just correct the filename and extension of the
first line to the script's variables):
modepref=$(astfits infile.fits -h1 \
| sed -e"s/'/ /g" \
| awk '$1=="MODE"{ \
if($3=="BIAS-IMAGE") print "bias-"; \
else if($3=="SCIENCE-IMAGE") print "sci-"; \
else if($3==FLAT-EXP) print "flat-"; \
else print $3, "NOT recognized"; exit 1}')
Here is a description of it. We first use ‘astfits’ to print all
the keywords in extension ‘1’ of ‘infile.fits’. In the FITS
standard, string values (that we are assuming here) are placed in
single quotes (<'>) which are annoying in this context/use-case.
Therefore, we pipe the output of ‘astfits’ into ‘sed’ to remove all
such quotes (substituting them with a blank space). The result is
then piped to AWK for giving us the final mode-prefix: with
‘$1=="MODE"’, we ask AWK to only consider the line where the first
column is ‘MODE’. There is an equal sign between the key name and
value, so the value is the third column (‘$3’ in AWK). We thus use
a simple ‘if-else’ structure to look into this value and print our
custom prefix based on it. The output of AWK is then stored in the
‘modepref’ shell variable which you can add to the link/copy name.
With the solution above, the increment of the file counter for each
night will be independent of the mode. If you want the counter to
be mode-dependent, you can add a different counter for each mode
and use that counter instead of the generic counter for each night
(based on the value of ‘modepref’). But we will leave the
implementation of this step to you as an exercise.
10.1.1 Invoking astscript-sort-by-night
---------------------------------------
This installed script will read a FITS date formatted value from the
given keyword, and classify the input FITS files into individual nights.
For more on installed scripts please see (see *note Installed
scripts::). This script can be used with the following general
template:
$ astscript-sort-by-night [OPTION...] FITS-files
One line examples:
## Use the DATE-OBS keyword
$ astscript-sort-by-night --key=DATE-OBS /path/to/data/*.fits
## Make links to the input files with the `img-' prefix
$ astscript-sort-by-night --link --prefix=img- /path/to/data/*.fits
This script will look into a HDU/extension (‘--hdu’) for a keyword
(‘--key’) in the given FITS files and interpret the value as a date.
The inputs will be separated by "night"s (11:00a.m to next day's
10:59:59a.m, spanning two calendar days, exact hour can be set with
‘--hour’).
The default output is a list of all the input files along with the
following two columns: night number and file number in that night
(sorted by time). With ‘--link’ a symbolic link will be made (one for
each input) that contains the night number, and number of file in that
night (sorted by time), see the description of ‘--link’ for more. When
‘--copy’ is used instead of a link, a copy of the inputs will be made
instead of symbolic link.
Below you can see one example where all the ‘target-*.fits’ files in
the ‘data’ directory should be separated by observing night according to
the ‘DATE-OBS’ keyword value in their second extension (number ‘1’,
recall that HDU counting starts from 0). You can see the output after
the ‘ls’ command.
$ astscript-sort-by-night -pimg- -h1 -kDATE-OBS data/target-*.fits
$ ls
img-n1-1.fits img-n1-2.fits img-n2-1.fits ...
The outputs can be placed in a different (already existing) directory
by including that directory's name in the ‘--prefix’ value, for example,
‘--prefix=sorted/img-’ will put them all under the ‘sorted’ directory.
This script can be configured like all Gnuastro's programs (through
command-line options, see *note Common options::), with some minor
differences that are described in *note Installed scripts::. The
particular options to this script are listed below:
‘-h STR’
‘--hdu=STR’
The HDU/extension to use in all the given FITS files. All of the
given FITS files must have this extension.
‘-k STR’
‘--key=STR’
The keyword name that contains the FITS date format to
classify/sort by.
‘-H FLT’
‘--hour=FLT’
The hour that defines the next "night". By default, all times
before 11:00a.m are considered to belong to the previous calendar
night. If a sub-hour value is necessary, it should be given in
units of hours, for example, ‘--hour=9.5’ corresponds to 9:30a.m.
*Dealing with time zones:* The time that is recorded in ‘--key’ may
be in UTC (Universal Time Coordinate). However, the organization
of the images taken during the night depends on the local time. It
is possible to take this into account by setting the ‘--hour’
option to the local time in UTC.
For example, consider a set of images taken in Auckland (New
Zealand, UTC+12) during different nights. If you want to classify
these images by night, you have to know at which time (in UTC time)
the Sun rises (or any other separator/definition of a different
night). For example, if your observing night finishes before
9:00a.m in Auckland, you can use ‘--hour=21’. Because in Auckland
the local time of 9:00 corresponds to 21:00 UTC.
‘-l’
‘--link’
Create a symbolic link for each input FITS file. This option
cannot be used with ‘--copy’. The link will have a standard name
in the following format (variable parts are written in ‘CAPITAL’
letters and described after it):
PnN-I.fits
‘P’
This is the value given to ‘--prefix’. By default, its value
is ‘./’ (to store the links in the directory this script was
run in). See the description of ‘--prefix’ for more.
‘N’
This is the night-counter: starting from 1. ‘N’ is just
incremented by 1 for the next night, no matter how many nights
(without any dataset) there are between two subsequent
observing nights (its just an identifier for each night which
you can easily map to different calendar nights).
‘I’
File counter in that night, sorted by time.
‘-c’
‘--copy’
Make a copy of each input FITS file with the standard naming
convention described in ‘--link’. With this option, instead of
making a link, a copy is made. This option cannot be used with
‘--link’.
‘-p STR’
‘--prefix=STR’
Prefix to append before the night-identifier of each newly created
link or copy. This option is thus only relevant with the ‘--copy’
or ‘--link’ options. See the description of ‘--link’ for how it is
used. For example, with ‘--prefix=img-’, all the created file
names in the current directory will start with ‘img-’, making
outputs like ‘img-n1-1.fits’ or ‘img-n3-42.fits’.
‘--prefix’ can also be used to store the links/copies in another
directory relative to the directory this script is being run (it
must already exist). For example,
‘--prefix=/path/to/processing/img-’ will put all the links/copies
in the ‘/path/to/processing’ directory, and the files (in that
directory) will all start with ‘img-’.
‘--stdintimeout=INT’
Number of micro-seconds to wait for standard input within this
script. This does not correspond to general inputs into the
script, inputs to the script should always be given as a file.
However, within the script, pipes are often used to pass the output
of one program to another. The value given to this option will be
passed to those internal pipes. When running this script, if you
confront an error, saying "No input!", you should be able to fix it
by giving a larger number to this option (the default value is
10000000 micro-seconds or 10 seconds).
10.2 Generate radial profile
============================
The 1 dimensional radial profile of an object is an important parameter
in many aspects of astronomical image processing. For example, you want
to study how the light of a galaxy is distributed as a function of the
radial distance from the center. In other cases, the radial profile of
a star can show the PSF (see *note PSF::). Gnuastro's
‘astscript-radial-profile’ script is created to obtain such radial
profiles for one object within an image. This script uses *note
MakeProfiles:: to generate elliptical apertures with the values equal to
the distance from the center of the object and *note MakeCatalog:: for
measuring the values over the apertures.
10.2.1 Invoking astscript-radial-profile
----------------------------------------
This installed script will measure the radial profile of an object
within an image. A general overview of this script has been published
in Infante-Sainz et al. 2024) (https://arxiv.org/abs/2401.05303);
please cite it if this script proves useful in your research. For more
on installed scripts please see (see *note Installed scripts::). This
script can be used with the following general template:
$ astscript-radial-profile [OPTION...] FITS-file
Examples:
## Generate the radial profile with default options (assuming the
## object is in the center of the image, and using the mean).
$ astscript-radial-profile image.fits
## Generate the radial profile centered at x=44 and y=37 (in pixels),
## up to a radial distance of 19 pixels, use the mean value.
$ astscript-radial-profile image.fits --center=44,37 --rmax=19
## Generate a 2D polar plot with the same properties as above:
$ astscript-radial-profile image.fits --center=44,37 --rmax=19 \
--polar
## Generate the radial profile centered at x=44 and y=37 (in pixels),
## up to a radial distance of 100 pixels, compute sigma clipped
## mean and standard deviation (sigclip-mean and sigclip-std) using
## 5 sigma and 0.1 tolerance (default is 3 sigma and 0.2 tolerance).
$ astscript-radial-profile image.fits --center=44,37 --rmax=100 \
--sigmaclip=5,0.1 \
--measure=sigclip-mean,sigclip-std
## Generate the radial profile centered at RA=20.53751695,
## DEC=0.9454292263, up to a radial distance of 88 pixels,
## axis ratio equal to 0.32, and position angle of 148 deg.
## Name the output table as `radial-profile.fits'
$ astscript-radial-profile image.fits --mode=wcs \
--center=20.53751695,0.9454292263 \
--rmax=88 --axis-ratio=0.32 \
--position-angle=148 -oradial-profile.fits
## Generate the radial profile centered at RA=40.062675270971,
## DEC=-8.1511992735126, up to a radial distance of 20 pixels,
## and calculate the SNR using the INPUT-NO-SKY and SKY-STD
## extensions of the NoiseChisel output file.
$ astscript-radial-profile image_detected.fits -hINPUT-NO-SKY \
--mode=wcs --measure=sn \
--center=40.062675270971,-8.1511992735126 \
--rmax=20 --stdhdu=SKY_STD
## Generate the radial profile centered at RA=40.062675270971,
## DEC=-8.1511992735126, up to a radial distance of 20 pixels,
## and compute the SNR with a fixed value for std, std=10.
$ astscript-radial-profile image.fits -h1 --mode=wcs --rmax=20 \
--center=40.062675270971,-8.1511992735126 \
--measure=sn --instd=10
## Generate the radial profile centered at X=1201, Y=1201 pixels, up
## to a radial distance of 20 pixels and compute the median and the
## SNR using the first extension of sky-std.fits as the dataset for std
## values.
$ astscript-radial-profile image.fits -h1 --mode=img --rmax=20 \
--center=1201,1201 --measure=median,sn \
--instd=sky-std.fits
This installed script will read a FITS image and will use it as the
basis for constructing the radial profile. The output radial profile is
a table (FITS or plain-text) containing the radial distance from the
center in the first row and the specified measurements in the other
columns (mean, median, sigclip-mean, sigclip-median, etc.).
To measure the radial profile, this script needs to generate
temporary files. All these temporary files will be created within the
directory given to the ‘--tmpdir’ option. When ‘--tmpdir’ is not
called, a temporary directory (with a name based on the inputs) will be
created in the running directory. If the directory does not exist at
run-time, this script will create it. After the output is created, this
script will delete the directory by default, unless you call the
‘--keeptmp’ option.
With the default options, the script will generate a circular radial
profile using the mean value and centered at the center of the image.
In order to have more flexibility, several options are available to
configure for the desired radial profile. In this sense, you can change
the center position, the maximum radius, the axis ratio and the position
angle (elliptical apertures are considered), the operator for obtaining
the profiles, and others (described below).
*Debug your profile:* to debug your results, especially close to the
center of your object, you can see the radial distance associated to
every pixel in your input. To do this, use ‘--keeptmp’ to keep the
temporary files, and compare ‘crop.fits’ (crop of your input image
centered on your desired coordinate) with ‘apertures.fits’ (radial
distance of each pixel).
*Finding properties of your elliptical target: * you want to measure the
radial profile of a galaxy, but do not know its exact location, position
angle or axis ratio. To obtain these values, you can use *note
NoiseChisel:: to detect signal in the image, feed it to *note Segment::
to do basic segmentation, then use *note MakeCatalog:: to measure the
center (‘--x’ and ‘--y’ in MakeCatalog), axis ratio (‘--axis-ratio’) and
position angle (‘--position-angle’).
*Masking other sources:* The image of an astronomical object will
usually have many other sources with your main target. A crude solution
is to use sigma-clipped measurements for the profile. However,
sigma-clipped measurements can easily be biased when the number of
sources at each radial distance increases at larger distances.
Therefore a robust solution is to mask all other detections within the
image. You can use *note NoiseChisel:: and *note Segment:: to detect
and segment the sources, then set all pixels that do not belong to your
target to blank using *note Arithmetic:: (in particular, its ‘where’
operator).
‘-h STR’
‘--hdu=STR’
The HDU/extension of the input image to use.
‘-o STR’
‘--output=STR’
Filename of measured radial profile. It can be either a FITS
table, or plain-text table (determined from your given file name
suffix).
‘-c FLT[,FLT[,...]]’
‘--center=FLT[,FLT[,...]]’
The central position of the radial profile. This option is used
for placing the center of the profiles. This parameter is used in
*note Crop:: to center and crop the region. The positions along
each dimension must be separated by a comma (<,>) and fractions are
also acceptable. The number of values given to this option must be
the same as the dimensions of the input dataset. The units of the
coordinates are read based on the value to the ‘--mode’ option, see
below.
‘-O STR’
‘--mode=STR’
Interpret the center position of the object (values given to
‘--center’) in image or WCS coordinates. This option thus accepts
only two values: ‘img’ or ‘wcs’. By default, it is ‘--mode=img’.
‘-R FLT’
‘--rmax=FLT’
Maximum radius for the radial profile (in pixels). By default, the
radial profile will be computed up to a radial distance equal to
the maximum radius that fits into the image (assuming circular
shape).
‘-P INT’
‘--precision=INT’
The precision (number of digits after the decimal point) in
resolving the radius. The default value is ‘--precision=0’ (or
‘-P0’), and the value cannot be larger than ‘6’. A higher
precision is primarily useful when the very central few pixels are
important for you. A larger precision will over-resolve larger
radial regions, causing scatter to significantly affect the
measurements.
For example, in the command below, we will generate the radial
profile of an imaginary source (at RA,DEC of 1.23,4.567) and check
the output without setting a precision:
$ astscript-radial-profile image.fits --center=1.23,4.567 \
--mode=wcs --measure=mean,area --rmax=10 \
--output=radial.fits --quiet
$ asttable radial.fits --head=10 -ffixed -p4
0.0000 0.0139 1
1.0000 0.0048 8
2.0000 0.0023 16
3.0000 0.0015 20
4.0000 0.0011 24
5.0000 0.0008 40
6.0000 0.0006 36
7.0000 0.0005 48
8.0000 0.0004 56
9.0000 0.0003 56
Let's repeat the command above, but use a precision of 3 to resolve
more finer details of the radial profile, while only printing the
top 10 rows of the profile:
$ astscript-radial-profile image.fits --center=1.23,4.567 \
--mode=wcs --measure=mean,area --rmax=10 \
--precision=3 --output=radial.fits --quiet
$ asttable radial.fits --head=10 -ffixed -p4
0.0000 0.0139 1
1.0000 0.0056 4
1.4140 0.0040 4
2.0000 0.0027 4
2.2360 0.0024 8
2.8280 0.0018 4
3.0000 0.0017 4
3.1620 0.0016 8
3.6050 0.0013 8
4.0000 0.0011 4
Do you see how many more radii have been added? Between 1.0 and
2.0, we now have one extra radius, between 2.0 to 3.0, we have two
new radii and so on. If you go to larger and larger radii, you
will notice that they get resolved into many sub-components and the
number of pixels used in each measurement will not be significant
(you can already see that in the comparison above). This has two
problems: 1. statistically, the scatter in larger radii (where the
signal-to-noise ratio is usually low will make it hard to interpret
the profile. 2. technically, the output table will have many more
rows!
*Use higher precision only for small radii:* If you want to look at
the whole profile (or the outer parts!), don't set the precision,
the default mode is usually more than enough! But when you are
targeting the very central few pixels (usually less than a pixel
radius of 5), use a higher precision.
‘-v INT’
‘--oversample=INT’
Oversample the input dataset to the fraction given to this option.
Therefore if you set ‘--rmax=20’ for example, and ‘--oversample=5’,
your output will have 100 rows (without ‘--oversample’ it will only
have 20 rows). Unless the object is heavily undersampled (the
pixels are larger than the actual object), this method provides a
much more accurate result and there are sufficient number of pixels
to get the profile accurately.
Due to the discrete nature of pixels, if you use this option to
oversample your profile, set ‘--precision=0’. Otherwise, your
profile will become step-like (with several radii having a single
value).
‘-u INT’
‘--undersample=INT’
Undersample the input dataset by the number given to this option.
This option is for considering larger apertures than the original
pixel size (aperture size is equal to 1 pixel). For example, if a
radial profile computed by default has 100 different radii
(apertures of 1 pixel width), by considering ‘--undersample=2’ the
radial profile will be computed over apertures of 2 pixels, so the
final radial profile will have 50 different radii. This option is
good to measure over a larger number of pixels to improve the
measurement.
‘-Q FLT’
‘--axis-ratio=FLT’
The axis ratio of the apertures (minor axis divided by the major
axis in a 2D ellipse). By default (when this option is not given),
the radial profile will be circular (axis ratio of 1). This
parameter is used as the option ‘--qcol’ in the generation of the
apertures with ‘astmkprof’.
‘-p FLT’
‘--position-angle=FLT’
The position angle (in degrees) of the profiles relative to the
first FITS axis (horizontal when viewed in SAO DS9). By default,
it is ‘--position-angle=0’, which means that the semi-major axis of
the profiles will be parallel to the first FITS axis.
‘-a FLT,FLT’
‘--azimuth=FLT,FLT’
Limit the profile to the given azimuthal angle range (two numbers
given to this option, in degrees, from 0 to 360) from the major
axis (defined by ‘--position-angle’). The radial profile will
therefore be created on a wedge-like shape, not the full
circle/ellipse. The pixel containing the center of the profile
will always be included in the profile (because it contains all
azimuthal angles!).
If the first angle is _smaller_ than the second (for example,
‘--azimuth=10,80’), the region between, or _inside_, the two angles
will be used. Otherwise (for example, ‘--azimuth=80,10’), the
region _outside_ the two angles will be used. The latter case can
be useful when you want to ignore part of the 2D shape (for
example, due to a bright star that can be contaminating it).
You can visually see the shape of the region used by running this
script with ‘--keeptmp’ and viewing the ‘values.fits’ and
‘apertures.fits’ files of the temporary directory with a FITS image
viewer like *note SAO DS9::. You can use *note Viewing FITS file
contents with DS9 or TOPCAT:: to open them together in one instance
of DS9, with both frames matched and locked (for easy comparison in
case you want to zoom-in or out). For example, see the commands
below (based on your target object, just change the image name,
center, position angle, etc.):
## Generate the radial profile
$ astscript-radial-profile image.fits --center=1.234,6.789 \
--mode=wcs --rmax=50 --position-angle=20 \
--axis-ratio=0.8 --azimuth=95,150 --keeptmp \
--tmpdir=radial-tmp
## Visually check the values and apertures used.
$ astscript-fits-view radial-tmp/values.fits \
radial-tmp/apertures.fits
‘-g’
‘--polar’
Generate a 2D polar-plot image in the second HDU of the output
(called ‘POLAR-PLOT’). A polar plot is a projection of the
original pixel grid into polar coordinates (where the horizontal
axis is the azimuthal angle and the vertical axis is the radius).
By default it assumes the full azimuthal range (from 0 to 360
degrees); if a narrower azimuthal range is desired, use ‘--azimuth’
(for example ‘--azimuth=30,50’ to only generate the polar plot
between 30 and 50 degrees of azimuth).
The output image contains WCS information to map the pixel
coordinates into the polar coordinates. This is especially useful
when the azimuthal range is not the full range: the first pixel in
the horizontal axis is not 0 degrees.
Currently, the polar plot cannot to be used with ‘--oversample’ and
‘--undersample’ options (please get in touch with us if you need
it). Until it is implemented, you can use the ‘--scale’ option of
*note Warp:: to do the oversampling of the input image yourself and
generate the polar plot from that. A comprehensive overview of
this option has been published in Eskandarlou et al. 2024
(https://arxiv.org/abs/2406.14619). If this script yields useful
results in your research, please be sure to cite it.
‘-m STR’
‘--measure=STR’
The operator for measuring the values over each radial distance.
The values given to this option will be directly passed to *note
MakeCatalog::. As a consequence, all MakeCatalog measurements like
the magnitude, magnitude error, median, mean, signal-to-noise ratio
(S/N), std, surface brightness, sigclip-mean, and sigclip-number
can be used here. For a full list of MakeCatalog's measurements,
please run ‘astmkcatalog --help’ or see *note MakeCatalog
measurements::. Multiple values can be given to this option, each
separated by a comma. This option can also be called multiple
times.
*Masking background/foreground objects:* For crude rejection of
outliers, you can use sigma-clipping using MakeCatalog measurements
like ‘--sigclip-mean’ or ‘--sigclip-mean-sb’ (see *note MakeCatalog
measurements::). To properly mask the effect of
background/foreground objects from your target object's radial
profile, you can use ‘astscript-psf-stamp’ script, see *note
Invoking astscript-psf-stamp::, and feed it the output of *note
Segment::. This script will mask unwanted objects from the image
that is later used to measure the radial profile.
Some measurements by MakeCatalog require a per-pixel sky standard
deviation (for example, magnitude error or S/N). Therefore when
asking for such measurements, use the ‘--instd’ option (described
below) to specify the per-pixel sky standard deviation over each
pixel. For other measurements like the magnitude or surface
brightness, MakeCatalog will need a Zero point, which you can set
with the ‘--zeropoint’ option.
For example, by setting ‘--measure=mean,sigclip-mean
--measure=median’, the mean, sigma-clipped mean and median values
will be computed. The output radial profile will have 4 columns in
this order: radial distance, mean, sigma-clipped and median. By
default (when this option is not given), the mean of all pixels at
each radial position will be computed.
‘-s FLT,FLT’
‘--sigmaclip=FLT,FLT’
Sigma clipping parameters: only relevant if sigma-clipping
operators are requested by ‘--measure’. For more on
sigma-clipping, see *note Sigma clipping::. If given, the value to
this option is directly passed to the ‘--sigmaclip’ option of *note
MakeCatalog::, see *note MakeCatalog inputs and basic settings::.
By default (when this option is not given), the default values
within MakeCatalog will be used. To see the default value of this
option in MakeCatalog, you can run this command:
$ astmkcatalog -P | grep " sigmaclip "
‘-z FLT’
‘--zeropoint=FLT’
The Zero point of the input dataset. This is necessary when you
request measurements like magnitude, or surface brightness.
‘-Z’
‘--zeroisnotblank’
Account for zero-valued pixels in the profile. By default, such
pixels are not considered (when this script crops the necessary
region of the image before generating the profile). The long
format of this option is identical to a similarly named option in
Crop (see *note Invoking astcrop::). When this option is called,
it is passed directly to Crop, therefore the zero-valued pixels are
not considered as blank and used in the profile creation.
‘-i FLT/STR’
‘--instd=FLT/STR’
Sky standard deviation as a single number (FLT) or as the filename
(STR) containing the image with the std value for each pixel (the
HDU within the file should be given to the ‘--stdhdu’ option
mentioned below). This is only necessary when the requested
measurement (value given to ‘--measure’) by MakeCatalog needs the
Standard deviation (for example, the signal-to-noise ratio or
magnitude error). If your measurements do not require a standard
deviation, it is best to ignore this option (because it will slow
down the script).
‘-d INT/STR’
‘--stdhdu=INT/STR’
HDU/extension of the sky standard deviation image specified with
‘--instd’.
‘-t STR’
‘--tmpdir=STR’
Several intermediate files are necessary to obtain the radial
profile. All of these temporal files are saved into a temporal
directory. With this option, you can directly specify this
directory. By default (when this option is not called), it will be
built in the running directory and given an input-based name. If
the directory does not exist at run-time, this script will create
it. Once the radial profile has been obtained, this directory is
removed. You can disable the deletion of the temporary directory
with the ‘--keeptmp’ option.
‘-k’
‘--keeptmp’
Do not delete the temporary directory (see description of
‘--tmpdir’ above). This option is useful for debugging. For
example, to check that the profiles generated for obtaining the
radial profile have the desired center, shape and orientation.
‘--cite’
Give BibTeX and acknowledgment information for citing this script
within your paper. For more, see ‘Operating mode options’.
10.3 SAO DS9 region files from table
====================================
Once your desired catalog (containing the positions of some objects) is
created (for example, with *note MakeCatalog::, *note Match::, or *note
Table::) it often happens that you want to see your selected objects on
an image for a feeling of the spatial properties of your objects. For
example, you want to see their positions relative to each other.
In this section we describe a simple installed script that is
provided within Gnuastro for converting your given columns to an SAO DS9
region file to help in this process. SAO DS9(1) is one of the most
common FITS image visualization tools in astronomy and is free software.
---------- Footnotes ----------
(1)
10.3.1 Invoking astscript-ds9-region
------------------------------------
This installed script will read two positional columns within an input
table and generate an SAO DS9 region file to visualize the position of
the given objects over an image. For more on installed scripts please
see (see *note Installed scripts::). This script can be used with the
following general template:
## Use the RA and DEC columns of 'table.fits' for the region file.
$ astscript-ds9-region table.fits --column=RA,DEC \
--output=ds9.reg
## Select objects with a magnitude between 18 to 20, and generate the
## region file directly (through a pipe), each region with radius of
## 0.5 arcseconds.
$ asttable table.fits --range=MAG,18:20 --column=RA,DEC \
| astscript-ds9-region --column=1,2 --radius=0.5
## With the first command, select objects with a magnitude of 25 to 26
## as red regions in 'bright.reg'. With the second command, select
## objects with a magnitude between 28 to 29 as a green region and
## show both.
$ asttable cat.fits --range=MAG_F160W,25:26 -cRA,DEC \
| astscript-ds9-region -c1,2 --color=red -obright.reg
$ asttable cat.fits --range=MAG_F160W,28:29 -cRA,DEC \
| astscript-ds9-region -c1,2 --color=green \
--command="ds9 image.fits -regions bright.reg"
The input can either be passed as a named file, or from standard
input (a pipe). Only the ‘--column’ option is mandatory (to specify the
input table columns): two columns from the input table must be
specified, either by name (recommended) or number. You can optionally
also specify the region's radius, width and color of the regions with
the ‘--radius’, ‘--width’ and ‘--color’ options, otherwise default
values will be used for these (described under each option).
The created region file will be written into the file name given to
‘--output’. When ‘--output’ is not called, the default name of
‘ds9.reg’ will be used (in the running directory). If the file exists
before calling this script, it will be overwritten, unless you pass the
‘--dontdelete’ option. Optionally you can also use the ‘--command’
option to give the full command that should be run to execute SAO DS9
(see example above and description below). In this mode, the created
region file will be deleted once DS9 is closed (unless you pass the
‘--dontdelete’ option). A full description of each option is given
below.
‘-h INT/STR’
‘--hdu INT/STR’
The HDU of the input table when a named FITS file is given as
input. The HDU (or extension) can be either a name or number
(counting from zero). For more on this option, see *note Input
output options::.
‘-c STR,STR’
‘--column=STR,STR’
Identifiers of the two positional columns to use in the DS9 region
file from the table. They can either be in WCS (RA and Dec) or
image (pixel) coordinates. The mode can be specified with the
‘--mode’ option, described below.
‘-n STR’
‘--namecol=STR’
The column containing the name (or label) of each region. The type
of the column (numeric or a character-based string) is irrelevant:
you can use both types of columns as a name or label for the
region. This feature is useful when you need to recognize each
region with a certain ID or property (for example, magnitude or
redshift).
‘-m wcs|img’
‘--mode=wcs|org’
The coordinate system of the positional columns (can be either
‘--mode=wcs’ and ‘--mode=img’). In the WCS mode, the values within
the columns are interpreted to be RA and Dec. In the image mode,
they are interpreted to be pixel X and Y positions. This option
also affects the interpretation of the value given to ‘--radius’.
When this option is not explicitly given, the columns are assumed
to be in WCS mode.
‘-C STR’
‘--color=STR’
The color to use for created regions. These will be directly
interpreted by SAO DS9 when it wants to open the region file so it
must be recognizable by SAO DS9. As of SAO DS9 8.2, the recognized
color names are ‘black’, ‘white’, ‘red’, ‘green’, ‘blue’, ‘cyan’,
‘magenta’ and ‘yellow’. The default color (when this option is not
called) is ‘green’
‘-w INT’
‘--width=INT’
The line width of the regions. These will be directly interpreted
by SAO DS9 when it wants to open the region file so it must be
recognizable by SAO DS9. The default value is ‘1’.
‘-r FLT’
‘--radius=FLT’
The radius of all the regions. In WCS mode, the radius is assumed
to be in arc-seconds, in image mode, it is in pixel units. If this
option is not explicitly given, in WCS mode the default radius is 1
arc-seconds and in image mode it is 3 pixels.
‘--dontdelete’
If the output file name exists, abort the program and do not
over-write the contents of the file. This option is thus good if
you want to avoid accidentally writing over an important file.
Also, do not delete the created region file when ‘--command’ is
given (by default, when ‘--command’ is given, the created region
file will be deleted after SAO DS9 closes).
‘-o STR’
‘--output=STR’
Write the created SAO DS9 region file into the name given to this
option. If not explicitly given on the command-line, a default
name of ‘ds9.reg’ will be used. If the file already exists, it
will be over-written, you can avoid the deletion (or over-writing)
of an existing file with the ‘--dontdelete’.
‘--command="STR"’
After creating the region file, run the string given to this option
as a command-line command. The SAO DS9 region command will be
appended to the end of the given command. Because the command will
mostly likely contain white-space characters it is recommended to
put the given string in double quotations.
For example, let's assume ‘--command="ds9 image.fits -zscale"’.
After making the region file (assuming it is called ‘ds9.reg’), the
following command will be executed:
ds9 image.fits -zscale -regions ds9.reg
You can customize all aspects of SAO DS9 with its command-line
options, therefore the value of this option can be as long and
complicated as you like. For example, if you also want the image
to fit into the window, this option will be: ‘--command="ds9
image.fits -zscale -zoom to fit"’. You can see the SAO DS9
command-line descriptions by clicking on the "Help" menu and
selecting "Reference Manual". In the opened window, click on
"Command Line Options".
10.4 Viewing FITS file contents with DS9 or TOPCAT
==================================================
The FITS definition allows for multiple extensions (or HDUs) inside one
FITS file. Each HDU can have a completely independent dataset inside of
it. One HDU can be a table, another can be an image and another can be
another independent image. For example, each image HDU can be one CCD
of a multi-CCD camera, or in processed images one can be the deep
science image and the next can be its weight map, alternatively, one HDU
can be an image, and another can be the catalog/table of objects within
it.
The most common software for viewing FITS images is SAO DS9 (see
*note SAO DS9::) and for plotting tables, TOPCAT is the most commonly
used tool in astronomy (see *note TOPCAT::). After installing them (as
described in the respective appendix linked in the previous sentence),
you can open any number of FITS images or tables with DS9 or TOPCAT with
the commands below:
$ ds9 image-a.fits image-b.fits
$ topcat table-a.fits table-b.fits
But usually the default mode is not enough. For example, in DS9, the
window can be too small (not covering the height of your monitor), you
probably want to match and lock multiple images, you have a favorite
color map that you prefer to use, or you may want to open a
multi-extension FITS file as a cube.
Using the simple commands above, you need to manually do all these in
the DS9 window once it opens and this can take several tens of seconds
(which is enough to distract you from what you wanted to inspect). For
example, if you have a multi-extension file containing 2D images, one
way to load and switch between each 2D extension is to take the
following steps in the SAO DS9 window: "File"→"Open Other"→"Open Multi
Ext Cube" and then choose the Multi extension FITS file in your
computer's file structure.
The method above is a little tedious to do every time you want view a
multi-extension FITS file. A different series of steps is also
necessary if you the extensions are 3D data cubes (since they are
already cubes, and should be opened as multi-frame). Furthermore, if
you have multiple images and want to "match" and "lock" them (so when
you zoom-in to one, all get zoomed-in) you will need several other
sequence of menus and clicks.
Fortunately SAO DS9 also provides command-line options that you can
use to specify a particular behavior before/after opening a file. One
of those options is ‘-mecube’ which opens a FITS image as a
multi-extension data cube (treating each 2D extension as a slice in a 3D
cube). This allows you to flip through the extensions easily while
keeping all the settings similar. Just to avoid confusion, note that
SAO DS9 does not follow the GNU style of separating long and short
options as explained in *note Arguments and options::. In the GNU
style, this 'long' (multi-character) option should have been called like
‘--mecube’, but SAO DS9 follows its own conventions.
For example, try running ‘$ds9 -mecube foo.fits’ to see the effect
(for example, on the output of *note NoiseChisel::). If the file has
multiple extensions, a small window will also be opened along with the
main DS9 window. This small window allows you to slide through the
image extensions of ‘foo.fits’. If ‘foo.fits’ only consists of one
extension, then SAO DS9 will open as usual.
On the other hand, for visualizing the contents of tables (that are
also commonly stored in the FITS format), you need to call a different
software (most commonly, people use TOPCAT, see *note TOPCAT::). And to
make things more inconvenient, by default both of these are only
installed as command-line software, so while you are navigating in your
GUI, you need to open a terminal there, and run these commands. All of
the issues above are the founding purpose of the installed script that
is introduced in *note Invoking astscript-fits-view::.
10.4.1 Invoking astscript-fits-view
-----------------------------------
Given any number of FITS files, this script will either open SAO DS9
(for images or cubes) or TOPCAT (for tables) to visualize their contents
in a graphic user interface (GUI). For more on installed scripts please
see (see *note Installed scripts::). This script can be used with the
following general template:
$ astscript-fits-view [OPTION] input.fits [input-b.fits ...]
One line examples
## Call TOPCAT to load all the input FITS tables.
$ astscript-fits-view table-*.fits
## Call SAO DS9 to open all the input FITS images.
$ astscript-fits-view image-*.fits
This script will use Gnuastro's *note Fits:: program to see if the
file is a table or image. If the first input file contains an image
HDU, then the sequence of files will be given to *note SAO DS9::.
Otherwise, the input(s) will be given to *note TOPCAT:: to visualize
(plot) as tables. When opening DS9 it will also inspect the
dimensionality of the first image HDU of the first input and open it
slightly differently when the input is 2D or 3D:
2D
DS9's ‘-mecube’ will be used to open all the 2D extensions of each
input file as a "Multi-extension cube". A "Cube" window will also
be opened with DS9 that can be used to slide/flip through each
extensions. When multiple files are given, each file will be in
one "frame".
3D
DS9's ‘-multiframe’ option will be used to open all the extensions
in a separate "frame" (since each input is already a 3D cube, the
‘-mecube’ option can be confusing). To flip through the extensions
(while keeping the slice fixed), click the "frame" button on the
top row of buttons, then use the last four buttons of the bottom
row ("first", "previous", "next" and "last") to change between the
extensions. If multiple files are given, there will be a separate
frame for each HDU of each input (each HDU's name or number will be
put in square brackets after its name).
*Double-clicking on FITS file to open DS9 or TOPCAT:* for those graphic
user interface (GUI) that follow the freedesktop.org standards
(including GNOME, KDS Plasma, or Xfce) Gnuastro installs a
‘fits-view.desktop’ file to instruct your GUI to call this script for
opening FITS files when you click on them. To activate this feature
take the following steps:
1. Run the following command, while replacing ‘PREFIX’. If you do not
know what to put in ‘PREFIX’, run ‘which astfits’ on the
command-line, and extract ‘PREFIX’ from the output (the string
before ‘/bin/astfits’). For more, see *note Installation
directory::.
ln -sf PREFIX/share/gnuastro/astscript-fits-view.desktop \
~/.local/share/applications/
2. Right-click on a FITS file, and choose these items in order (based
on GNOME, may be different in KDE or Xfce): "Open with other
application"→"View all applications"→"astscript-fits-view".
This script takes the following options
‘-h STR’
‘--hdu=STR’
The HDU(s), or extension(s), of the input dataset(s) to display.
The value can be the HDU name (a string) or number (the first HDU
is counted from 0). If there are multiple inputs, this option
needs to be called multiple times: the first input will be opened
with the first call to this option, the second input with the
second call and etc. If you want to open the same HDU of all your
inputs, you don't need to repeat this option, use ‘--globalhdu’
instead.
‘-g STR’
‘--globalhdu=STR’
The HDU/extension name or number to use for all inputs. Note that
HDU counting starts from 0. If ‘--hdu’ called, it takes precedence
over this.
‘-p STR’
‘--prefix=STR’
Directory to search for SAO DS9 or TOPCAT's executables (assumed to
be ‘ds9’ and ‘topcat’). If not called they will be assumed to be
present in your ‘PATH’ (see *note Installation directory::). If
you do not have them already installed, their installation
directories are available in *note SAO DS9:: and *note TOPCAT::
(they can be installed in non-system-wide locations that do not
require administrator/root permissions).
‘-s STR’
‘--ds9scale=STR’
The string to give to DS9's ‘-scale’ option. You can use this
option to use a different scaling. The Fits-view script will place
‘-scale’ before your given string when calling DS9. If you do not
call this option, the default behavior is to cal DS9 with: ‘-scale
mode zscale’ or ‘--ds9scale="mode zscale"’ when using this script.
The Fits-view script has the following aliases to simplify the
calling of this option (and avoid the double-quotations and ‘mode’
in the example above):
‘zscale’
or ‘--ds9scale=zscale’ equivalent to ‘--ds9scale="mode
zscale"’.
‘minmax’
or ‘--ds9scale=minmax’ equivalent to ‘--ds9scale="mode
minmax"’.
‘-c=FLT,FLT’
‘--ds9center=FLT,FLT’
The central coordinate for DS9's view of the FITS image after it
opens. This is equivalent to the "Pan" button in DS9. The nature
of the coordinates will be determined by the ‘--ds9mode’ option
that is described below.
‘-r=STR[,STR[,STR]]’
‘--ds9region=STR[,STR[,STR]]’
Name of DS9 region file(s) to load within the DS9 window once it
opens. Any number of region files can be loaded with a single call
to this command (separated by comas).
‘-O img/wcs’
‘--ds9mode=img/wcs’
The coordinate system (or mode) to interpret the values given to
‘--ds9center’. This can either be ‘img’ (or DS9's "Image"
coordinates) or ‘wcs’ (or DS9's "wcs fk5" coordinates).
‘-g INTxINT’
‘--ds9geometry=INTxINT’
The initial DS9 window geometry (value to DS9's ‘-geometry’
option).
‘-m’
‘--ds9colorbarmulti’
Do not show a single color bar for all the loaded images. By
default this script will call DS9 in a way that a single color bar
is shown for any number of images. A single color bar is preferred
for two reasons: 1) when there are a lot of images, they consume a
large fraction of the display area. 2) the color-bars are locked
by this script, so there is no difference between! With this
option, you can have separate color bars under each image.
‘-e STR’
‘--ds9extra=STR’
Any other custom options you would like to directly pass to DS9.
This option can be called multiple times. The string(s) given to
this option will be the final component of the DS9 command that
this script generates. This allows you to over-ride/customize
previously set options (for example in large scripts).
Besides enabling all DS9 options that don't have a high-level
wrapper in this script, you can also use this option to re-order
the way the options of this script call DS9. This is because this
script ignores the order that you have requested options and will
always put the high-level functions in a fixed order. Before
calling DS9, this script will print the command that it executes so
you can see the order explicitly before DS9 opens.
Since DS9 does not recognize a <=> between the option name and
value, a is usually necessary. Therefore, be sure to put
them in a single string within double quotes (so the spaces are
preserved and passed to DS9 correctly). For example usage of this
option within this manual, see *note Column statistics
color-magnitude diagram:: or *note Subtracting the PSF::.
10.5 Zero point estimation
==========================
Through the "zero point", we are able to give physical units to the
pixel values of an image (often in units of "counts" or ADUs) and thus
compare them with other images (as well as measurements that are done on
them). The zero point is therefore an important calibration of pixel
values (as astromerty is a calibration of the pixel positions). The
fundamental concepts behind the zero point are described in *note
Brightness flux magnitude::. We will therefore not go deeper into the
basics here and stick to the practical aspects of it.
The purpose of Gnuastro’s ‘astscript-zeropoint’ script is to obtain
the zero point of an image by considering another image (where the zero
point is already known), or a catalog. In the The operation involves
multiple lower-level programs in a standard series of steps. For
example, when using another image, the script will take the following
steps:
1. Download the Gaia catalog that overlaps with the input image using
Gnuastro’s Query program (see *note Query::). This is done to
determine the stars within the image(1).
2. Perform aperture photometry(2) with *note MakeProfiles:: *note
MakeCatalog::. We will assume a zero point of 0 for the input
image. If the reference is an image, then we should perform
aperture photometry also in that image.
3. Match the two catalogs(3) with *note Match::.
4. The difference between the input and reference magnitudes should be
independent of the magnitude of the stars. This does not hold when
the stars are saturated in one/both the images (giving us a
bright-limit for the magnitude range to use) or for stars fainter
than a certain magnitude, where the signal-to-noise ratio drops
significantly in one/both images (giving us a faint limit for the
magnitude range to use).
5. Since a zero point of 0 was used for the input image, the magnitude
difference above (in the reliable magnitude range) is the zero
point of the input image.
In the "Tutorials" chapter of this Gnuastro book, there are two
tutorials dedicated to the usage of this script. The first uses an
image as a reference (*note Zero point tutorial with reference image::)
and the second uses a catalog (*note Zero point tutorial with reference
catalog::). For the full set of options an a detailed description of
each, see *note Invoking astscript-zeropoint::.
---------- Footnotes ----------
(1) Stars have an almost identical shape in the image (as opposed to
galaxies for example), using confirmed stars will produce a more
reliable result.
(2) For a complete tutorial on aperture photometry, see *note
Aperture photometry::.
(3) For a tutorial on matching catalogs, see *note Matching
catalogs::).
10.5.1 Invoking astscript-zeropoint
-----------------------------------
This installed script will calculate the zero point of an input image to
calibrate it. A general overview of this script has been published in
Eskandarlou et al. 2023 (https://arxiv.org/abs/2312.04263); please cite
it if this script proves useful in your research. The reference can be
an image or catalog (which have been previously calibrated) The
executable name is ‘astscript-zeropoint’, with the following general
template:
## Using a reference image in four apertures.
$ astscript-zeropoint image.fits --hdu=1 \
--refimgs=ref-img1.fits,ref-img2.fits \
--refimgshdu=1,1 \
--refimgszp=22.5,22.5 \
--aperarcsec=1.5,2,2.5,3 \
--magnituderange=16,18 \
--output=output.fits
## Using a reference catalog
$ astscript-zeropoint image.fits --hdu=1 \
--refcat=cat.fits \
--refcathdu=1 \
--aperarcsec=1.5,2,2.5,3 \
--magnituderange=16,18 \
--output=output.fits
To learn more about the core concepts behind the zero point, please
see *note Brightness flux magnitude::. For a practical review of how to
optimally use this script and ways to interpret its results, we have two
tutorials: *note Zero point tutorial with reference image:: and *note
Zero point tutorial with reference catalog::.
To find the zero point of your input image, this script can use a
reference image (that already has a zero point) or a reference catalog
(that just has magnitudes). In any case, it is mandatory to identify at
least one aperture for aperture photometry over the image (using
‘--aperarcsec’). If reference image(s) is(are) given, it is mandatory
to specify its(their) zero point(s) using the ‘--refimgszp’ option (it
can take a separate value for each reference image). When a catalog is
given, it should already contain the magnitudes of the object (you can
specify which column to use).
This script will not estimate the zero point based on all the objects
in the reference image or catalog. It will first query Gaia database
and only select objects have a significant parallax (because Gaia's
algorithms sometimes confuse galaxies and stars based on pure
morphology). You can bypass this step (which needs internet connection
and can only be used on real data, not simulations) using the
‘--starcat’ option described in *note zero point options::. This script
will then match the catalog of stars (either through Gaia or
‘--starcat’) with the reference catalog and only use them. If the
reference is an image, it will simply use the stars catalog to do
aperture photometry.
By default, this script will estimate the number of available threads
and run all independent steps in parallel on those threads. To control
this behavior (and allow it to only run on a certain number of threads),
you can use the ‘--numthreads’ option.
During its operation, this script will build a temporary file in the
running directory that will be deleted once it is finished. The
‘--tmpdir’ option can be used to manually set the temporary directory's
location at any location in your file system. The ‘--keeptmp’ option
can be used to stop the deletion of that directory (useful for when you
want to debug the script or better understand what it does).
10.5.1.1 astscript-zeropoint output
...................................
The output will be a multi-extension FITS table. The first table in the
output gives the zero point and its standard deviation for all the
requested apertures. This gives you the ability to inspect them and
select the best. The other table(s) give the exact measurements for
each star that was used (if you use ‘--keepzpap’, it will be for all
your apertures, if not, only for the aperture with the smallest standard
deviation). For a full tutorial on how to interpret the output of this
script, see *note Zero point tutorial with reference image::
If you just want the estimated zero point with the least standard
deviation, this script will write it as a FITS keyword in the first
table of the output.
‘ZPAPER’
Read as "Zero Point APERture". This shows the aperture radius (in
arcseconds) that had the smallest standard deviation in the
estimated zero points.
‘ZPVALUE’
The zero point estimation for the aperture of ‘ZPAPER’.
‘ZPSTD’
The standard deviation of the zero point (for all the stars used,
within the aperture of ‘ZPAPER’).
‘ZPMAGMIN’
The minimum (brightest) magnitude used to estimate the zero point.
‘ZPMAGMAX’
The maximum (faintest) magnitude used to estimate the zero point.
A simple way to see these keywords, or read the value of one is shown
below. For more on viewing and manipulating FITS keywords, see *note
Keyword inspection and manipulation::.
## See all the keywords written by this script (that start with 'ZP')
$ astfits out.fits -h1 | grep ^ZP
## If you just want the zero point
$ astfits jplus-zeropoint.fits -h1 --keyvalue=ZPVALUE
10.5.1.2 astscript-zeropoint options
....................................
All the operating phases of the this script can be customized through
the options below.
‘-h STR/INT’
‘--hdu=STR/INT’
The HDU/extension of the input image to use.
‘-o STR’
‘--output=STR’
The name of the output file produced by this script. See *note
zero point output:: for the format of its contents.
‘-N INT’
‘--numthreads=INT’
The number of threads to use. By default this script will attempt
to find the number of available threads at run-time and will use
them.
‘-a FLT,[FLT]’
‘--aperarcsec=FLT,[FLT]’
The radius/radii (in arc seconds) of aperture(s) used in aperture
photometry of the input image. This option can take many values
(to check different apertures and find the best for a more accurate
zero point estimation). If a reference image is used, the same
aperture radii will be used for aperture photometry there.
‘-M FLT,FLT’
‘--magnituderange=FLT,FLT’
Range of the magnitude for finding the best aperture and zero
point. Very bright stars get saturated and fainter stars are
affected too much by noise. Therefore, it is important to limit
the range of magnitudes used in estimating the zero point. A full
tutorial is given in *note Zero point tutorial with reference
image::.
‘-S STR’
‘--starcat=STR’
Name of catalog containing the RA and Dec of positions for aperture
photometry in the input image and reference (catalog or image). If
not given, the Gaia database will be queried for all stars that
overlap with the input image (see *note Available databases::).
This option is therefore useful in the following scenarios (among
others):
• No internet connection.
• Many images having a major overlap in the sky, making it
inefficient to query Gaia for every image separately: you can
query the larger area (containing all the images) once, and
directly give the downloaded table to all the separate runs of
this script. Especially if the field is wide, the download
time can be the slowest part of this script.
• In simulations (where you have a pre-defined list of stars).
Through the ‘--starcathdu’, ‘--starcatra’ and ‘--starcatdec’
options described below, you can specify the HDU, RA column and Dec
Column within this file.
The reference image or catalog probably have many objects that are
not stars. But it is only stars that have the same shape (the PSF)
across the image(1). Therefore
‘--starcathdu=STR/INT’
The HDU name or number in file given to ‘--starcat’ (described
above) that contains the table of RA and Dec positions for aperture
photometry. If not given, it is assumed that the table is in HDU
number 1 (counting from 0).
‘--starcatra=STR/INT’
The column name or number (in the table given to ‘--starcat’) that
contains the Right Ascension.
‘--starcatdec=STR/INT’
The column name or number (in the table given to ‘--starcat’) that
contains the Declination.
‘-c STR’
‘--refcat=STR’
Reference catalog used to estimate the zero point of the input
image. This option is mutually exclusive with (cannot be given at
the same time as) ‘--refimgs’. This catalog should have RA, Dec
and Magnitude of the stars (that match with Gaia or ‘--starcat’).
‘-C STR/INT’
‘--refcathdu=STR/INT’
The HDU/extension of the reference catalog will be calculated.
‘-r STR’
‘--refcatra=STR’
Right Ascension column name of the reference catalog.
‘-d STR’
‘--refcatdec=STR’
Declination column name of the reference catalog.
‘-m STR’
‘--refcatmag=STR’
Magnitude column name of the reference catalog.
‘-s FLT’
‘--matchradius=FLT’
Matching radius of stars (in arc seconds) and reference catalog in
arc-seconds. By default it is 0.2 arc seconds.
‘-R STR,[STR]’
‘--refimgs=STR,[STR]’
Reference image(s) for estimating the zero point. This option can
take any number of separate file names, separated by a comma. The
HDUs of each reference image should be given to the ‘refimgshdu’
option. In case the images are in separate HDUs of the same file,
you need to repeat the file name here. This option is mutually
exclusive with (cannot be given at the same time as) ‘--refimgs’.
‘-H STR/INT’
‘--refimgshdu=STR/INT’
HDU/Extension name of number of the reference files. The number of
values given to this option should be the same as the number of
reference image(s).
‘-z FLT,[FLT]’
‘--refimgszp=FLT,[FLT]’
Zero point of the reference image(s). The number of values given
to this should be the same as the number of names given to
‘--refimgs’.
‘-K’
‘--keepzpap’
Keep the table of separate zero points found for each star for all
apertures. By default, this table is only present for the aperture
that had the least standard deviation in the estimated zero point.
‘-t’
‘--tmpdir’
Directory to keep temporary files during the execution of the
script. If the directory does not exist at run$-$time, this script
will create it. By default, upon completion of the script, this
directory will be deleted. However, if you would like to keep the
intermediate files, you can use the ‘--keeptmp’ option.
‘-k’
‘--keeptmp’
Its recommended to not remove the temporary directory (see
description of ‘--keeptmp’). This option is useful for debugging
and checking the outputs of internal steps.
‘--mksrc=STR’
Use a non-standard Makefile for the Makefile to call. This option
is primarily useful during the development of this script and its
Makefile, not for normal/regular user. So if you are not
developing this script, you can safely ignore this option. When
this option is given, the default installed Makefile will not be
used: the file given to this option will be read by ‘make’ (within
the script) instead.
‘--cite’
Give BibTeX and acknowledgment information for citing this script
within your paper. For more, see ‘Operating mode options’.
---------- Footnotes ----------
(1) The PSF itself can vary across the field of view; but that is
second-order for this analysis.
10.6 Pointing pattern simulation
================================
Astronomical images are often composed of many single exposures. When
the science topic does not depend on the time of observation (for
example galaxy evolution), after completing the observations, we stack
those single exposures into one "deep" image. Designing the strategy to
take those single exposures is therefore a very important aspect of
planning your astronomical observation. There are many reasons for
taking many short exposures instead of one long exposure:
• Modern astronomical telescopes have very high precision (with
pixels that are often much smaller than an arc-second or 1/3600
degrees. However, the Earth is orbiting the Sun at a very high
speed of roughly 15 degrees every hour! Keeping the (often very
large!) telescopes in track with this fast moving sky is not easy;
such that most cannot continue accurate tracking more than 10
minutes.
• For ground-based observations, the turbulence of the atmosphere
changes very fast (on the scale of minutes!). So if you plan to
observe at 10 minutes and at the start of your observations the
seeing is good, it may happen that on the 8th minute, it becomes
bad. This will affect the quality of your final exposure!
• When an exposure is taken, the instrument/environment imprint a lot
of artifacts on it. One common example that we also see in normal
cameras is vignetting (https://en.wikipedia.org/wiki/Vignetting);
where the center receives a larger fraction of the incoming light
than the periphery). In order to characterize and remove such
artifacts (which depend on many factors at the precision that we
need in astronomy!), we need to take many exposures of our science
target.
• By taking many exposures we can build a stack that has a higher
resolution; this is often done in under-sampled data, like those in
the Hubble Space Telescope (HST) or James Webb Space Telescope
(JWST).
• The scientific target can be larger than the field of view of your
telescope and camera.
In the jargon of observational astronomers, each exposure is also
known as a "dither" (literally/generally meaning "trembling" or
"vibration"). This name was chosen because two exposures are not
usually taken on exactly the same position of the sky (known as
"pointing"). In order to improve all the item above, we often move the
center of the field of view from one exposure to the next. In most
cases this movement is small compared to the field of view, so most of
the central part of the final stack has a fixed depth, but the edges are
shallower (conveying a sense of vibration). When the spacing between
pointings is large, they are known as an "offset". A "pointing" is used
to refer to either a dither or an offset.
For example see Figures 3 and 4 of Illingworth et al. 2013
(https://arxiv.org/pdf/1305.1931.pdf) which show the exposures that went
into the XDF survey. The pointing pattern can also be large compared to
the field of view, for example see Figure 1 of Trujillo et al. 2021
(https://arxiv.org/pdf/2109.07478.pdf), which show the pointing strategy
for the LIGHTS survey. These types of images (where each pixel contains
the number of exposures, or time, that were used in it) are known as
exposure maps.
The pointing pattern therefore is strongly defined by the science
case (high-level purpose of the observation) and your telescope's field
of view. For example in the XDF survey is focused on very high redshift
(most distant!) galaxies. These are very small objects and within that
small footprint (of just 1 arcmin) we have thousands of them. However,
the LIGHTS survey is focused on the halos of large nearby galaxies (that
can be more than 10 arcminutes wide!).
In *note Invoking astscript-pointing-simulate:: of Gnuastro's *note
Installed scripts:: is described in detail. This script is designed to
simplify the process of selecting the best pointing pattern for your
observation strategy. For a practical tutorial on using this script,
see *note Pointing pattern design::.
10.6.1 Invoking astscript-pointing-simulate
-------------------------------------------
This installed script will simulate a final stacked image from a certain
pointing pattern (given as a table). A general overview of this script
has been published in Akhlaghi (2023)
(https://ui.adsabs.harvard.edu/abs/2023RNAAS...7..211A); please cite it
if this script proves useful in your research. The executable name is
‘astscript-pointing-simulate’, with the following general template:
$ astscript-pointing-simulate [OPTION...] pointings.fits
Examples (for a tutorial, see *note Pointing pattern design::):
$ astscript-pointing-simulate pointing.fits --output=stack.fits \
--img=image.fits --center=10,10 --width=1,1
The default output of this script is a stacked image that results
from placing the given image (given to ‘--img’) in the pointings of a
pointing pattern. The Right Ascension (RA) and Declination (Dec) of
each pointing is given in the main input catalog (‘pointing.fits’ in the
example above). The center and width of the final stack (both in
degrees by default) should be specified using the ‘--width’ option.
Therefore, in order to successfully run, this script at least needs the
following four inputs:
Pointing positions
A table containing the RA and Dec of each pointing (the only input
argument). The actual column names that contain them can be set
with the ‘--racol’ and ‘--deccol’ options (see below).
An image
This is used for its distortion and rotation, its pixel values and
position on the sky will be ignored. The file containing the image
should be given to the ‘--img’ option.
Stack's central coordinate
The central RA and Dec of the finally produced stack (given to the
‘--center’ option).
Stack's width
The width (in degrees) of the final stack (given to the ‘--width’
option).
This script will ignore the pixel values of the reference image
(given to ‘--img’) and the Reference coordinates (values to ‘CRVAL1’ and
‘CRVAL2’ in its WCS keywords). For each pointing pointing, this script
will put the given RA and Dec into the ‘CRVAL1’ and ‘CRVAL2’ keywords of
a copy of the input (not changing the input in anyway), and reset that
input's pixel values to 1. The script will then warp the modified copy
into the final pixel grid (correcting any rotation and distortions that
are used from the original input). This process is done for all the
pointing points in parallel. Finally, all the exposures in the pointing
list are stacked together to produce an exposure map (showing how many
exposures go into each pixel of the final stack.
Except for the table of pointing positions, the rest of the inputs
and settings are configured through *note Options::, just note the
restrictions in *note Installed scripts::.
‘-o STR’
‘--output=STR’
Name of the output. The output is an image of the requested size
(‘--width’) and position (‘--center’) in the sky, but each pixel
will contain the number of exposures that go into it after the
pointing has been done. See description above for more.
‘-h STR/INT’
‘--hdu=STR/INT’
The name or counter (counting from zero; from the FITS standard) of
HDU containing the table of pointing pointing positions (the file
name of this table is the main input argument to this script). For
more, see the description of this option in *note Input output
options::.
‘-i STR’
‘--img=STR’
The references image. The pixel values and central location in
this image will be ignored by the script. The only relevant
information within this script are the WCS properties (except for
‘CRVAL1’ and ‘CRVAL2’, which connect it to a certain position on
the sky) and image size. See the description above for more.
‘-H STR/INT’
‘--imghdu=STR/INT’
The name or counter (counting from zero; from the FITS standard) of
the HDU containing the reference image (file name should be given
to the ‘--img’ option). If not given, a default value of ‘1’ is
assumed; so this is not a mandatory option.
‘-r STR/INT’
‘--racol=STR/INT’
The name or counter (counting from 1; from the FITS standard) of
the column containing the Right Ascension (RA) of each pointing to
be used in the pointing pattern. The file containing the table is
given to this script as its only argument.
‘-d STR/INT’
‘--deccol=STR/INT’
The name or counter (counting from 1; from the FITS standard) of
the column containing the Declination (Dec) of each pointing to be
used in the pointing pattern. The file containing the table is
given to this script as its only argument.
‘-C FLT,FLT’
‘--center=FLT,FLT’
The central RA and Declination of the final stack in degrees.
‘-w FLT,FLT’
‘--width=FLT,FLT’
The width of the final stack in degrees. If ‘--widthinpix’ is
given, the two values given to this option will be interpreted as
degrees.
‘--widthinpix’
Interpret the values given to ‘--width’ as number of pixels along
each dimension), and not as degrees.
‘--ctype=STR,STR’
The projection of the output stack (‘CTYPEi’ keyword in the FITS
WCS standard). For more, see the description of the same option in
*note Align pixels with WCS considering distortions::.
‘--hook-warp-before='STR'’
Command to run before warping each exposure into the output pixel
grid. By default, the exposure is immediately warped to the final
pixel grid, but in some scenarios it is necessary to do some
operations on the exposure before warping (for example account for
vignetting; see *note Accounting for non-exposed pixels::). The
warping of each exposure is done in parallel by default; therefore
there are pre-defined variables that you should use for the input
and output file names of your command:
‘$EXPOSURE’
Input: name of file with the same size as the reference image
with all pixels having a fixed value of 1. The WCS has also
been corrected based on the pointing pattern.
‘$TOWARP’
Output: name of the expected output of your hook. If it is
not created by your script, the script will complain and
abort. This file will be given to Warp to be warped into the
output pixel grid.
For an example of using hooks with an extended discussion, see
*note Pointing pattern design:: and *note Accounting for
non-exposed pixels::.
To develop your command, you can use ‘--hook-warp-before='...; echo
GOOD; exit 1'’ (where ‘...’ can be replaced by any command) and run
the script on a single thread (with ‘--numthreads=1’) to produce a
single file and simplify the checking that your desired operation
works as expected. All the files will be within the temporary
directory (see ‘--tmpdir’).
‘--hook-warp-after='STR'’
Command to run after the warp of each exposure into the output
pixel grid, but before the stacking of all exposures. For more on
hooks, see the description of ‘--hook-warp-before’, *note Pointing
pattern design:: and *note Accounting for non-exposed pixels::.
‘$WARPED’
Input: name of file containing the warped exposure in the
output pixel grid.
‘$TOWARP’
Output: name of the expected output of your hook. If it is
not created by your script, the script will complain and
abort. This file will be stacked from the same file for all
exposures into the final output.
‘--stack-operator="STR"’
The operator to use for stacking the warped individual exposures
into the final output of this script. For the full list, see *note
Stacking operators::. By default it is the ‘sum’ operator (to
produce an output exposure map). For an example usage, see the
tutorial in *note Pointing pattern design::.
‘--mksrc=STR’
Use a non-standard Makefile for the Makefile to call. This option
is primarily useful during the development of this script and its
Makefile, not for normal/regular usage. So if you are not
developing this script, you can safely ignore this option. When
this option is given, the default installed Makefile will not be
used: the file given to this option will be read by ‘make’ (within
the script) instead.
‘-t STR’
‘--tmpdir=STR’
Name of directory containing temporary files. If not given, a
temporary directory will be created in the running directory with a
long name using some of the input options. By default, this
temporary directory will be deleted after the output is created.
You can disable the deletion of the temporary directory (useful for
debugging!) with the ‘--keeptmp’ option.
Using this option has multiple benefits in larger pipelines:
• You can avoid conflicts in case the used inputs in the default
name are the same.
• You can put this directory somewhere else in the running file
system to avoid mixing output files with your source, or to
use other storage hardware that are mounted on the running
file system.
‘-k’
‘--keeptmp’
Keep the temporary directory (and do not delete it).
‘-?’
‘--help’
Print a list of all the options, along with a short description and
context for the program. For more, see ‘Operating mode options’.
‘-N INT’
‘--numthreads=INT’
The number of threads to use for parallel operations (warping the
input into the different pointing points). If not given (by
default), the script will try to find the number of available
threads on the running system and use that. For more, see
‘Operating mode options’.
‘--cite’
Give BibTeX and acknowledgment information for citing this script
within your paper. For more, see ‘Operating mode options’.
‘-q’
‘--quiet’
Do not print the series of commands or their outputs in the
terminal. For more, see ‘Operating mode options’.
‘-V’
‘--version’
Print the version of the running Gnuastro along with a copyright
notice and list of authors that contributed to this script. For
more, see ‘Operating mode options’.
10.7 Color images with gray faint regions
=========================================
Typical astronomical images have a very wide range of pixel values and
generally, it is difficult to show the entire dynamical range in a color
image. For example, by using *note ConvertType::, it is possible to
obtain a color image with three FITS images as each of the
Red-Green-Blue (or RGB) color channels. However, depending on the pixel
distribution, it could be very difficult to see the different regions
together (faint and bright objects at the same time). In something like
DS9, you end up changing the color map parameters to see the regions you
are most interested in.
The reason is that images usually have a lot of faint pixels (near to
the sky background or noise values), and few bright pixels
(corresponding to the center of stars, galaxies, etc.) that can be
millions of times brighter! As a consequence, by considering the images
without any modification, it is extremely hard to visualize the entire
range of values in a color image. This is because standard color
formats like JPEG, TIFF or PDF are defined as 8-bit integer precision,
while astronomical data are usually 32-bit floating point! To solve
this issue, it is possible to perform some transformations of the images
and then obtain the color image.
This is actually what the current script does: it makes some
non-linear transformations and then uses Gnuastro's ConvertType to
generate the color image. There are several parameters and options in
order to change the final output that are described in *note Invoking
astscript-color-faint-gray::. A full tutorial describing this script
with actual data is available in *note Color images with full dynamic
range::. A general overview of this script is published in
Infante-Sainz et al. 2024 (https://arxiv.org/abs/2401.03814); please
cite it if this script proves useful in your research.
10.7.1 Invoking astscript-color-faint-gray
------------------------------------------
This installed script will consider several images to combine them into
a single color image to visualize the full dynamic range. The
executable name is ‘astscript-color-faint-gray’, with the following
general template:
$ astscript-color-faint-gray [OPTION...] r.fits g.fits b.fits
Examples (for a tutorial, see *note Color images with full dynamic
range::):
## Generate a color image from three images with default options.
$ astscript-color-faint-gray r.fits g.fits b.fits -g1 --output color.pdf
## Generate a color image, consider the minimum value to be zero.
$ astscript-color-faint-gray r.fits g.fits b.fits -g1 \
--minimum=0.0 --output=color.jpg
## Generate a color image considering different zero points, minimum
## values, weights, and also increasing the contrast.
$ astscript-color-faint-gray r.fits g.fits b.fits -g1 \
-z=22.4 -z=25.5 -z=24.6 \
-m=-0.1 -m=0.0 -m=0.1 \
-w=1 -w=2 -w=3 \
--contrast=3 \
--output=color.tiff
This script takes three inputs images to generate a RGB color image
as the output. The order of the images matters, reddest (longest
wavelength) filter (R), green (an intermediate wavelength) filter (G)
and bluest (shortest wavelength). In astronomy, these can be any filter
(for example from infra-red, radio, optical or x-ray); the "RGB"
designation is from the general definition of colors (see
) These images are
internally manipulated by a series of non-linear transformations and
normalized to homogenize and finally combine them into a color image.
In general, for typical astronomical images, the default output is an
image with bright pixels in color and noise pixels in black.
The option ‘--minimum’ sets the minimum value to be shown and it is a
key parameter, it uses to be a value close to the sky background level.
The current non-linear transformation is from Lupton et al. 2004
(https://ui.adsabs.harvard.edu/abs/2004PASP..116..133L), which we call
the "asinh" transformation. The two important parameters that control
this transformation are ‘--qthresh’ and ‘--stretch’. With the option
‘--coloronly’, it is possible to generate a color image with the
background in black: bright pixels in color and the sky background (or
noise) values in black. It is possible to provide a fourth image (K)
that will be used for showing the gray region: R, G, B, K
The generation of a good color image is something that requires
several trials, so we encourage the user to play with the different
parameters cleverly. After some testing, we find it useful to follow
the steps. For a more complete description of the logic of the process,
see the dedicated tutorial in *note Color images with full dynamic
range::.
1. Use the default options to estimate the parameters. By running the
script with no options at all, it will estimate the parameters and
they will be printed on the command-line.
2. Select a good sky background value of the images. If the sky
background has been subtracted, a minimum value of zero could be a
good option: ‘--minimum=0.0’.
3. Focus on the bright regions to tweak ‘--qbright’ and ‘--stretch’.
First, try low values of ‘--qbright’ to show the bright parts.
Then, adjust ‘--stretch’ to show the fainter regions around bright
parts. Overall, play with these two parameters to show the color
regions appropriately.
4. Change ‘--colorval’ to separate the color and black regions. This
is the lowest value of the threshold image that is shown in color.
5. Change ‘--grayval’ to separate the black and gray regions. This is
highest value of the threshold image that is shown in gray.
6. Use ‘--checkparams’ to check the pixel value distributions.
7. Use ‘--keeptmp’ to not remove the threshold image and check it.
A full description of each option is given below:
‘-h’
‘--hdu=STR/INT’
Input HDU name or counter (counting from 0) for each input FITS
file. If the same HDU should be used from all the FITS files, you
can use the ‘--globalhdu’ option described below to avoid repeating
this option.
‘-g’
‘--globalhdu=STR/INT’
Use the value given to this option (a HDU name or a counter,
starting from 0) for the HDU identifier of all the input FITS
files. This is useful when all the inputs are distributed in
different files, but have the same HDU in those files.
‘-o’
‘--output’
Output color image name. The output can be in any of the
recognized output formats of ConvertType (including PDF, JPEG and
TIFF).
‘-m’
‘--minimum=FLT’
Minimum value to be mapped for each R, G, B, and K FITS images. If
a single value is given to this option it will be used for all the
input images.
This parameter controls the smallest visualized pixel value. In
general, it is a good decision to set this value close to the sky
background level. This value can dramatically change the output
color image (especially when there are large negative values in the
image that you do not intend to visualize).
‘-Z’
‘--zeropoint=FLT’
Zero point value for each R, G, B, and K FITS images. If a single
value is given, it is used for all the input images.
Internally, the zero point values are used to transform the pixel
values in units of Janskys. The units are not important for a
color image, but the fact that the images are photometrically
calibrated is important for obtaining an output color image whose
color distribution is realistic.
‘-w’
‘--weight=FLT’
Relative weight for each R, G, B channel. With this parameter, it
is possible to change the importance of each channel to modify the
color balance of the image.
For example, ‘-w=1 -w=2 -w=5’ indicates that the B band will be 5
times more important than the R band, and that the G band is 2
times more important than the R channel. In this particular
example, the combination will be done as
$\rm{colored}=(1{\times}\rm{R}+2{\times}\rm{G}+5{\times}\rm{B})/(1
+ 2 + 5)=0.125{\times}\rm{R} + 0.250{\times}\rm{G} +
0.625{\times}\rm{B}$.
In principle, a color image should recreate "real" colors, but
"real" is a very subjective matter and with this option, it is
possible to change the color balance and make it more aesthetically
interesting. However, be careful to avoid confusing the viewers of
your image and report the weights with the filters you used for
each channel. It is up to the user to use this parameter
carefully.
‘-Q’
‘--qbright=FLT’
It is one of the parameters that control the asinh transformation.
It should be used in combination with ‘--stretch’. In general, it
has to be set to low values to better show the brightest regions.
Afterwards, adjust ‘--stretch’ to set the linear stretch (show the
intermediate/faint structures).
‘-s’
‘--stretch=FLT’
It is one of the parameters that control the asinh transformation.
It should be used in combination with ‘--qbright’. It is used for
bringing out the faint/intermediate bright structures of the image
that are shown linearly. In general, this parameter is chosen
after setting ‘--qbright’ to a low value.
*The asinh transformation.* The asinh transformation is done on
the stacked R, G, B image. It consists in the modification of the
stacked image (I) in order to show the entire dynamical range
appropriately following the expression: $f(I) = asinh($ ‘qbright’
$\cdot$ ‘stretch’ $\cdot I) /$ ‘qbright’. See *note Color images
with full dynamic range:: for a complete tutorial that shows the
intricacies of this transformation with step-by-step examples.
‘--coloronly’
By default, the fainter parts of the image are shown in grayscale
(not color, since colored noise is not too informative). With this
option, the output image will be fully in color with the background
(noise pixels) in black.
‘--colorval=FLT’
The value that separates the color and black regions. By default,
it ranges from 100 (all pixels becoming in color) to 0 (all pixels
becoming black). Check the histogram ‘FOR COLOR and GRAY
THRESHOLDS’ with the option ‘--checkparams’ for selecting a good
value.
‘--grayval=FLT’
This parameter defines the value that separates the black and gray
regions. It ranges from 100 (all pixels becoming black) to 0 (all
pixels becoming white). Check the histogram ‘FOR COLOR and GRAY
THRESHOLDS’ with the option ‘--checkparams’ to select the value.
‘--regions=STR’
Labeled image, identifying the pixels to use for color (value 2),
those to use for black (value 1) and those to use for gray (value
0). When this option is given the ‘--colorval’ and ‘--grayval’
options will be ignored. This gives you the freedom to select the
pixels to show in color, black or gray based on any criteria that
is relevant for your purpose. For an example of using this option
to get a physically motivated threshold, see *note Manually setting
color-black-gray regions::.
*IMPORTANT NOTE.* The options ‘--colorval’ and ‘--grayval’ are
related one to each other. They are defined from the threshold
image (an image generated in the temporary directory) named
‘colorgray_threshold.fits’. By default, this image is computed
from the stack and later asinh-transformation of the three R, G, B
channels. Its pixel values range between 100 (brightest) to 0
(faintest). The ‘--colorval’ value computed by default is the
median of this image. Pixels above this value are shown in color.
Pixels below this value are shown in gray. Regions of pure black
color can be defined with the ‘--grayval’ option if this value is
between 0 and ‘--colorval’. In other words. Color region are
defined by those pixels between 100 and ‘--colorval’. Pure black
region are defined by those pixels between ‘--colorval’ to
‘grayval’. Gray region are defined by those pixels between
‘--grayval’ to 0.
If a fourth image is provided as the "K" channel, then this image
is used as the threshold image. See *note Color images with full
dynamic range:: for a complete tutorial.
‘--colorkernelfwhm=FLT’
Gaussian kernel FWHM (in pixels) for convolving the color regions.
Sometimes, a convolution of the color regions (bright pixels) is
desired to further increase their signal-to-noise ratio (but make
them look smoother). With this option, the kernel will be created
internally and convolved with the colored regions.
‘--graykernelfwhm=FLT’
Gaussian kernel FWHM (in pixels) for convolving the background
image. Sometimes, a convolution of the background image is
necessary to smooth the noisier regions and increase their
signal-to-noise ratios. With this option, the kernel will be
created internally and convolved with the colored regions.
‘-b’
‘--bias=FLT’
Change the brightness of the final image. By increasing this
value, a pedestal value will be added to the color image. This
option is rarely useful, it is most common to use ‘--contrast’, see
below.
‘-c’
‘--contrast=FLT’
Change the contrast of the final image. The transformation is:
$\rm{output}=\rm{contrast}\times{image}+brightness$.
‘-G’
‘--gamma=FLT’
Gamma exponent value for a gamma transformation. This
transformation is not linear: $\rm{output}=\rm{image}^{gamma}$.
This option overrides ‘--bias’ or ‘--contrast’.
‘--markoptions=STR’
Options to draw marks on the final output image. Anything given to
this option is passed directly to ConvertType in order to draw
marks on the output image. For example, if you construct a table
named ‘marks.txt’ that contains the column names: x, y, shape,
size, axis ratio, angle, color; you will execute the script with
the following option: ‘--markoptions="--marks=markers.txt
--markcoords=x,y --markshape=shape --marksize=size,axisratio
--markrotate=angle --markcolor=color"’. See *note Drawing with
vector graphics:: for more information on how to draw markers and
*note Weights contrast markers and other customizations:: for a
tutorial.
‘--checkparams’
Print the statistics of intermediate images that are used for
estimating the parameters. This option is useful to decide the
optimum set of parameters.
‘--keeptmp’
Do not remove the temporary directory. This is useful for
debugging and checking the outputs of internal steps.
‘--cite’
Give BibTeX and acknowledgment information for citing this script
within your paper. For more, see ‘Operating mode options’.
‘-q’
‘--quiet’
Do not print the series of commands or their outputs in the
terminal. For more, see ‘Operating mode options’.
‘-V’
‘--version’
Print the version of the running Gnuastro along with a copyright
notice and list of authors that contributed to this script. For
more, see ‘Operating mode options’.
10.8 PSF construction and subtraction
=====================================
The point spread function (PSF) describes how the light of a point-like
source is affected by several optical scattering effects (atmosphere,
telescope, instrument, etc.). Since the light of all astrophysical
sources undergoes all these effects, characterizing the PSF is key in
astronomical analysis (for small and large objects). Consequently,
having a good characterization of the PSF is fundamental to any
analysis.
In some situations(1) a parametric (analytical) model is sufficient
for the PSF (such as Gaussian or Moffat, see *note PSF::). However,
once you are interested in objects that are larger than a handful of
pixels, it is almost impossible to find an analytic function to
adequately characterize the PSF. Therefore, it is necessary to obtain an
empirical (non-parametric) and extended PSF. In this section we describe
a set of installed scrips in Gnuastro that will let you construct the
non-parametric PSF using point-like sources. They allow you to derive
the PSF from the same astronomical images that the science is derived
from (without assuming any analytical function).
The scripts are based on the concepts described in Infante-Sainz et
al. 2020 (https://arxiv.org/abs/1911.01430). But to be complete, we
first give a summary of the logic and overview of their combined usage
in *note Overview of the PSF scripts::. Furthermore, before going into
the technical details of each script, we encourage you to go through the
tutorial that is devoted to this at *note Building the extended PSF::.
The tutorial uses a real dataset and includes all the logic and
reasoning behind every step of the usage in every installed script.
---------- Footnotes ----------
(1) An example scenario where a parametric PSF may be enough: you are
only interested in very small, high redshift objects that only extended
a handful of pixels.
10.8.1 Overview of the PSF scripts
----------------------------------
To obtain an extended and non-parametric PSF, several steps are
necessary and we will go through them here. The fundamental ideas of
the following methodology are thoroughly described in Infante-Sainz et
al. 2020 (https://arxiv.org/abs/1911.01430). A full tutorial is also
available in *note Building the extended PSF::. The tutorial will go
through the full process on a pre-selected dataset, but will describe
the logic behind every step in away that can easily be
modified/generalized to other datasets.
This section is basically just a summary of that tutorial. We could
have put all these steps into one large program (installed script),
however this would introduce several problems. The most prominent of
these problems are:
• The command would require _many_ options, making it very complex to
run every time.
• You usually have many stars in an image, and many of the steps can
be optimized or parallelized depending on the particular analysis
scenario. Predicting all the possible optimizations for all the
possible usage scenarios would make the code extremely complex
(filled with many unforeseen bugs!).
Therefore, following the modularity principle of software
engineering, after several years of working on this, we have broken the
full job into the smallest number of independent steps as separate
scripts. All scripts are independent of each other, meaning this that
you are free to use all of them as you wish (for example, only some of
them, using another program for a certain step, using them for other
purposes, or running independent parts in parallel).
For constructing the PSF from your dataset, the first step is to
obtain a catalog of stars within it (you cannot use galaxies to build
the PSF!). But you cannot blindly use all the stars either! For
example, we do not want contamination from other bright, and nearby
objects. The first script below is therefore designed for selecting
only good star candidates in your image. It will use different
criteria, for example, good parallax (where available, to avoid
confusion with galaxies), not being near to bright stars, axis ratio,
etc. For more on this script, see *note Invoking
astscript-psf-select-stars::.
Once the catalog of stars is constructed, another script is in charge
of making appropriate stamps of the stars. Each stamp is a cropped
image of the star with the desired size, normalization of the flux, and
mask of the contaminant objects. For more on this script, see *note
Invoking astscript-psf-stamp:: After obtaining a set of star stamps,
they can be stacked for obtaining the combined PSF from many stars (for
example, with *note Stacking operators::).
In the combined PSF, the masked background objects of each star's
image will be covered and the signal-to-noise ratio will increase,
giving a very nice view of the "clean" PSF. However, it is usually
necessary to obtain different regions of the same PSF from different
stars. For example, to construct the far outer wings of the PSF, it is
necessary to consider very bright stars. However, these stars will be
saturated in the most inner part, and immediately outside of the
saturation level, they will be deformed due to non-linearity effects.
Consequently, fainter stars are necessary for the inner regions.
Therefore, you need to repeat the steps above for certain stars (in a
certain magnitude range) to obtain the PSF in certain radial ranges.
For example, in Infante-Sainz et al. 2020
(https://arxiv.org/abs/1911.01430), the final PSF was constructed from
three regions (and thus, using stars from three ranges in magnitude).
In other cases, we even needed four groups of stars! But in the example
dataset from the tutorial, only two groups are necessary (see *note
Building the extended PSF::).
Once clean stacks of different parts of the PSF have been constructed
through the steps above, it is therefore necessary to blend them all
into one. This is done by finding a common radial region in both, and
scaling the inner region by a factor to add with the outer region. This
is not trivial, therefore, a third script is in charge of it, see *note
Invoking astscript-psf-unite::.
Having constructed the PSF as described above (or by any other
procedure), it can be scaled to the magnitude of the various stars in
the image to get subtracted (and thus remove the extended/bright wings;
better showing the background objects of interest). Note that the
absolute flux of a PSF is meaningless (and in fact, it is usually
normalized to have a total sum of unity!), so it should be scaled. We
therefore have another script that will calculate the scale
(multiplication) factor of the PSF for each star. For more on the
scaling script, see *note Invoking astscript-psf-scale-factor::.
Once the flux factor has been computed, a final script is in charge
of placing the scaled PSF over the proper location in the image, and
subtracting it. It is also possible to only obtain the modeled star by
the PSF. For more on the scaling and positioning script, see *note
Invoking astscript-psf-subtract::.
As mentioned above, in the following sections, each script has its
own documentation and list of options for very detailed customization
(if necessary). But if you are new to these scripts, before continuing,
we recommend that you do the tutorial *note Building the extended PSF::.
Just do not forget to run every command, and try to tweak its steps
based on the logic to nicely understand it.
10.8.2 Invoking astscript-psf-select-stars
------------------------------------------
This installed script will select good star candidates for constructing
a PSF. It will consider stars within a given range of magnitudes without
nearby contaminant objects. To do that, it allows to the user to
specify different options described here. A complete tutorial is
available to show the operation of this script as a modular component to
extract the PSF of a dataset: *note Building the extended PSF::. The
executable name is ‘astscript-psf-select-stars’, with the following
general template:
$ astscript-psf-select-stars [OPTION...] FITS-file
Examples:
## Select all stars within 'image.fits' with magnitude in range
## of 6 to 10; only keeping those that are less than 0.02 degrees
## from other nearby stars.
$ astscript-psf-select-stars image.fits \
--magnituderange=6,10 --mindistdeg=0.02
The input of this script is an image, and the output is a catalog of
stars with magnitude in the requested range of magnitudes (provided with
‘--magnituderange’). The output catalog will also only contain stars
that are sufficiently distant (‘--mindistdeg’) from all other brighter,
and some fainter stars. It is possible to consider different datasets
with the option ‘--dataset’ (by default, Gaia DR3 dataset is considered)
All stars that are ‘--faintmagdiff’ fainter than the faintest limit will
also be accounted for, when selecting good stars. The
‘--magnituderange’, and ‘--mindistdeg’ are mandatory: if not specified
the code will abort.
The output of this script is a file whose name can be specified with
the (optional) ‘--output’ option. If not given, an automatically
generated name will be used for the output. A full description of each
option is given below.
‘-h STR/INT’
‘--hdu=STR/INT’
The HDU/extension of the input image to use.
‘-S STR’
‘--segmented=STR’
Optional segmentation file obtained by *note Segment::. It should
have two extensions (‘CLUMPS’ and ‘OBJECTS’). If given, a catalog
of ‘CLUMPS’ will be computed and matched with the Gaia catalog to
reject those objects that are too elliptical (see
‘--minaxisratio’). The matching will occur on an aperture (in
degrees) specified by ‘--matchaperturedeg’.
‘-a FLT’
‘--matchaperturedeg=FLT’
This option determines the aperture (in degrees) for matching the
catalog from gaia with the clumps catalog that is produced by the
segmentation image given to ‘--segmented’. The default value is 10
arc-seconds.
‘-c STR’
‘--catalog=STR’
Optional reference catalog to use for selecting stars (instead of
querying an external catalog like Gaia). When this option is
given, ‘--dataset’ (described below) will be ignored and no
internet connection will be necessary.
‘-d STR’
‘--dataset=STR’
Optional dataset to query (see *note Query::). It should contain
the database and dataset entries to Query. Its value will be
immediately given to ‘astquery’. By default, its value is ‘gaia
--dataset=dr3’ (so it connects to the Gaia database and requests
the data release 3). For example, if you want to use VizieR's Gaia
DR3 instead (for example due to a maintenance on ESA's Gaia
servers), you should use ‘--dataset="vizier --dataset=gaiadr3"’.
It is possible to specify a different dataset from which the
catalog is downloaded. In that case, the necessary column names
may also differ, so you also have to set ‘--refcatra’,
‘--refcatdec’ and ‘--field’. See their description for more.
‘-r STR’
‘--refcatra=STR’
The name of the column containing the Right Ascension (RA) in the
requested dataset (‘--dataset’). If the user does not determine
this option, the default value is assumed to be ‘ra’.
‘-d STR’
‘--refcatdec=STR’
The name of the column containing the Declination (Dec) in the
requested dataset (‘--dataset’). If the user does not determine
this option, the default value is assumed to be ‘dec’.
‘-f STR’
‘--field=STR’
The name of the column containing the magnitude in the requested
dataset (‘--dataset’). The output will only contain stars that
have a value in this column, between the values given to
‘--magnituderange’ (see below). By default, the value of this
option is ‘phot_g_mean_mag’ (that corresponds to the name of the
magnitude of the G-band in the Gaia catalog).
‘-m FLT,FLT’
‘--magnituderange=FLT,FLT’
The acceptable range of values for the column in ‘--field’. This
option is mandatory and no default value is assumed.
‘-p STR,STR’
‘--parallaxanderrorcolumn=STR,STR’
With this option the user can provide the parallax and parallax
error column names in the requested dataset. When given, the
output will only contain stars for which the parallax value is
smaller than three times the parallax error. If the user does not
provide this option, the script will not use parallax information
for selecting the stars. In the case of Gaia, if you want to use
parallax to further limit the good stars, you can pass
‘parallax,parallax_error’.
‘-D FLT’
‘--mindistdeg=FLT’
Stars with nearby bright stars closer than this distance are
rejected. The default value is 1 arc minute. For fainter stars
(when constructing the center of the PSF), you should decrease the
value.
‘-b INT’
‘--brightmag=INT’
The brightest star magnitude to avoid (should be brighter than the
brightest of ‘--magnituderange’). The basic idea is this: if a
user asks for stars with magnitude 6 to 10 and one of those stars
is near a magnitude 3 star, that star (with a magnitude of 6 to 10)
should be rejected because it is contaminated. But since the
catalog is constrained to stars of magnitudes 6-10, the star with
magnitude 3 is not present and cannot be compared with! Therefore,
when considering proximity to nearby stars, it is important to use
a larger magnitude range than the user's requested magnitude range
for good stars. The acceptable proximity is defined by
‘--mindistdeg’.
With this option, you specify the brightest limit for the proximity
check. The default value is a magnitude of $-10$, so you'll rarely
need to change or customize this option!
The faint limit of the proximity check is specified by
‘--faintmagdiff’. As the name suggests, this is a "diff" or
relative value. The default value is 4. Therefore if the user
wants to build the PSF with stars in the magnitude range of 6 to
10, the faintest stars used for the proximity check will have a
magnitude of 14: $10+4$. In summary, by default, the proximity
check will be done with stars in the magnitude range $-10$ to $14$.
‘-F INT’
‘--faintmagdiff’
The magnitude difference of the faintest star used for proximity
checks to the faintest limit of ‘--magnituderange’. For more, see
the description of ‘--brightmag’.
‘-Q FLT’
‘--minaxisratio=FLT’
Minimum acceptable axis ratio for the selected stars. In other
words, only stars with axis ratio between ‘--minaxisratio’ to 1.0
will be selected. Default value is ‘--minaxisratio=0.9’. Recall
that the axis ratio is only used when you also give a segmented
image with ‘--segmented’.
‘-t’
‘--tmpdir’
Directory to keep temporary files during the execution of the
script. If the directory does not exist at run-time, this script
will create it. By default, upon completion of the script, this
directory will be deleted. However, if you would like to keep the
intermediate files, you can use the ‘--keeptmp’ option.
‘-k’
‘--keeptmp’
Do not remove the temporary directory (see description of
‘--keeptmp’). This option is useful for debugging and checking the
outputs of internal steps.
‘-o STR’
‘--output=STR’
The output name of the final catalog containing good stars.
10.8.3 Invoking astscript-psf-stamp
-----------------------------------
This installed script will generate a stamp of fixed size, centered at
the provided coordinates (performing sub-pixel re-gridding if necessary)
and normalized at a certain normalization radius. Optionally, it will
also mask all the other background sources. A complete tutorial is
available to show the operation of this script as a modular component to
extract the PSF of a dataset: *note Building the extended PSF::. The
executable name is ‘astscript-psf-stamp’, with the following general
template:
$ astscript-psf-stamp [OPTION...] FITS-file
Examples:
## Make a stamp around (x,y)=(53,69) of width=151 pixels.
## Normalize the stamp within the radii 20 and 30 pixels.
$ astscript-psf-stamp image.fits --mode=img \
--center=53,69 --widthinpix=151,151 --normradii=20,30 \
--output=stamp.fits
## Iterate over a catalog with positions of stars that are
## in the input image. Use WCS coordinates.
$ asttable catalog.fits | while read -r ra dec mag; do \
astscript-psf-stamp image.fits \
--mode=wcs \
--center=$ra,$dec \
--normradii=20,30 \
--widthinpix=150,150 \
--output=stamp-"$ra"-"$dec".fits; done
The input is an image from which the stamp of the stars are
constructed. The output image will have the following properties:
• A certain width (specified by ‘--widthinpix’ in pixels).
• Centered at the coordinate specified by the option ‘--center’ (it
can be in image/pixel or WCS coordinates, see ‘--mode’). If no
center is specified, then it is assumed that the object of interest
is already in the center of the image.
• If the given coordinate has sub-pixel elements (for example, pixel
coordinates 1.234,4.567), the pixel grid of the output will be
warped so your given coordinate falls in the center of the central
pixel of the final output. This is very important for building the
central parts of the PSF, but not too effective for the middle or
outer parts (to speed up the program in such cases, you can disable
it with the ‘--nocentering’ option).
• Normalized "normalized" by the value computed within the ring
around the center (at a radial distance between the two radii
specified by the option ‘--normradii’). If no normalization ring
is considered, the output will not be normalized.
In the following cases, this script will produce a fully NaN-valued
stamp (of the size given to ‘--widthinpix’). A fully NaN image can
safely be used with the stacking operators of Arithmetic (see *note
Stacking operators::) because they will be ignored. In case you do not
want to waste storage with fully NaN images, you can compress them with
‘gzip --best output.fits’, and give the resulting ‘.fits.gz’ file to
Arithmetic.
• The requested box (center coordinate with desired width) is not
within the input image at all.
• If a normalization radius is requested, and all the pixels within
the normalization radii are NaN. Here are some scenarios that this
can happen: 1) You have a saturated star (where the saturated
pixels are NaN), and your normalization radius falls within the
saturated region. 2) The star is outside the image by more than
your larger normalization radius (so there are no pixels for doing
normalization), but the full stamp width still overlaps part of the
image.
The full set of options are listed below for optimal customization in
different scenarios:
‘-h STR’
‘--hdu=STR’
The HDU/extension of the input image to use.
‘-O STR’
‘--mode=STR’
Interpret the center position of the object (values given to
‘--center’) in image or WCS coordinates. This option thus accepts
only two values: ‘img’ or ‘wcs’.
‘-c FLT,FLT’
‘--center=FLT,FLT’
The central position of the object. This option is used for
placing the center of the stamp. This parameter is used in *note
Crop:: to center and crop the image. The positions along each
dimension must be separated by a comma (<,>). The units of the
coordinates are read based on the value to the ‘--mode’ option, see
the examples above.
The given coordinate for the central value can have sub-pixel
elements (for example, it falls on coordinate 123.4,567.8 of the
input image pixel grid). In such cases, after cropping, this
script will use Gnuastro's *note Warp:: to shift (or translate) the
pixel grid by $-0.4$ pixels along the horizontal and $1-0.8=0.2$
pixels along the vertical. Finally the newly added pixels (due to
the warping) will be trimmed to have your desired coordinate
exactly in the center of the central pixel of the output. This is
very important (critical!) when you are constructing the central
part of the PSF. But for the outer parts it is not too effective,
so to avoid wasting time for the warping, you can simply use
‘--nocentering’ to disable it.
‘-d’
‘--nocentering’
Do not do the sub-pixel centering to a new pixel grid. See the
description of the ‘--center’ option for more.
‘-W INT,INT’
‘--widthinpix=INT,INT’
Size (width) of the output image stamp in pixels. The size of the
output image will be always an odd number of pixels. As a
consequence, if the user specify an even number, the final size
will be the specified size plus 1 pixel. This is necessary to
place the specified coordinate (given to ‘--center’) in the center
of the central pixel. This is very important (and necessary) in
the case of the centers of stars, therefore a sub-pixel translation
will be performed internally to ensure this.
‘-n FLT,FLT’
‘--normradii=FLT,FLT’
Minimum and maximum radius of ring to normalize the image. This
option takes two values, separated by a comma (<,>). The first
value is the inner radius, the second is the outer radius.
‘-S STR’
‘--segment=STR’
Optional filename of a segmentation image from Segment's output
(must contain the ‘CLUMPS’ and ‘OBJECT’ HDUs). For more on the
definition of "objects" and "clumps", see *note Segment::. If
given, Segment's output is used to mask all background sources from
the large foreground object (a bright star):
• Objects that are not the central object.
• Clumps (within the central object) that are not the central
clump.
The result is that all objects and clumps that contaminate the
central source are masked, while the diffuse flux of the central
object remains. The non masked object and clump labels are kept
into the header of the output image. The keywords are ‘CLABEL’ and
‘OLABEL’. If no segmentation image is used, then their values are
set to ‘none’.
‘-T FLT’
‘--snthresh=FLT’
Mask all the pixels below the given signal-to-noise ratio (S/N)
threshold. This option is only valid with the ‘--segment’ option
(it will use the ‘SKY_STD’ extension of the *note Segment output::.
This threshold is applied prior to the possible normalization or
centering of the stamp. After all pixels below the given threshold
are masked, the mask is also dilated by one level to avoid single
pixels above the threshold (which are mainly due to noise when the
threshold is lower).
After applying the signal-to-noise threshold (if it is requested),
any extra pixels that are not connected to the central target are
also masked. Such pixels can remain in rivers between bright
clumps and will cause problem in the final stack, if they are not
masked.
This is useful for increasing the S/N of inner parts of each region
of the finally stacked PSF. As the stars (that are to be stacked)
become fainter, the S/N of their outer parts (at a fixed radius)
decreases. The stack of a higher-S/N image with a lower-S/N image
will have an S/N that is lower than the higher one. But we can
still use the inner parts of those fainter stars (that have
sufficiently high S/N).
‘-N STR’
‘--normop=STR’
The operator for measuring the values within the ring defined by
the option ‘--normradii’. The operator given to this option will
be directly passed to the radial profile script
‘astscript-radial-profile’, see *note Generate radial profile::.
As a consequence, all MakeCatalog measurements (median, mean,
sigclip-mean, sigclip-number, etc.) can be used here. For a full
list of MakeCatalog's measurements, please run ‘astmkcatalog
--help’. The final normalization value is saved into the header of
the output image with the keyword ‘NORMVAL’. If no normalization
is done, then the value is set to ‘1.0’.
‘-Q FLT’
‘--axis-ratio=FLT’
The axis ratio of the radial profiles for computing the
normalization value. By default (when this option is not given),
the radial profile will be circular (axis ratio of 1). This
parameter is used directly in the ‘astscript-radial-profile’
script.
‘-p FLT’
‘--position-angle=FLT’
The position angle (in degrees) of the profiles relative to the
first FITS axis (horizontal when viewed in SAO DS9). By default,
it is ‘--position-angle=0’, which means that the semi-major axis of
the profiles will be parallel to the first FITS axis. This
parameter is used directly in the ‘astscript-radial-profile’
script.
‘-s FLT,FLT’
‘--sigmaclip=FLT,FLT’
Sigma clipping parameters: only relevant if sigma-clipping
operators are requested by ‘--normop’. For more on sigma-clipping,
see *note Sigma clipping::.
‘-t’
‘--tmpdir’
Directory to keep temporary files during the execution of the
script. If the directory does not exist at run-time, this script
will create it. By default, upon completion of the script, this
directory will be deleted. However, if you would like to keep the
intermediate files, you can use the ‘--keeptmp’ option.
‘-k’
‘--keeptmp’
Do not remove the temporary directory (see description of
‘--keeptmp’). This option is useful for debugging and checking the
outputs of internal steps.
‘-o STR’
‘--output=STR’
Filename of stamp image. By default the name of the stamp will be
a combination of the input image name, the name of the script, and
the coordinates of the center. For example, if the input image is
named image.fits and the center is ‘--center=33,78’, then the
output name wil be: image_stamp_33_78.fits The main reason of
setting this name is to have an unique name for each stamp by
default.
10.8.4 Invoking astscript-psf-unite
-----------------------------------
This installed script will join two PSF images at a given radius. This
operation is commonly used when merging (uniting) the inner and outer
parts of the PSF. A complete tutorial is available to show the operation
of this script as a modular component to extract the PSF of a dataset:
*note Building the extended PSF::. The executable name is
‘astscript-psf-unite’, with the following general template:
$ astscript-psf-unite [OPTION...] FITS-file
Examples:
## Multiply inner.fits by 3 and put it in the center (within a radius of
## 25 pixels) of outer.fits. The core goes up to a radius of 25 pixels.
$ astscript-psf-unite outer.fits \
--core=inner.fits --scale=3 \
--radius=25 --output=joined.fits
## Same example than the above, but considering an
## ellipse (instead of a circle).
$ astscript-psf-unite outer.fits \
--core=inner.fits --scale=3 \
--radius=25 --axis-ratio=0.5 \
--position-angle=40 --output=joined.fits
The junction is done by considering the input image as the outer
part. The central part is specified by FITS image given to ‘--inner’
and it is multiplied by the factor ‘--scale’. All pixels within
‘--radius’ (in pixels) of the center of the outer part are then replaced
with the inner image.
The scale factor to multiply with the inner part has to be explicitly
provided (see the description of ‘--scale’ below). Note that this
script assumes that PSF is centered in both images. More options are
available with the goal of obtaining a good junction. A full
description of each option is given below.
‘-h STR’
‘--hdu=STR’
The HDU/extension of the input image to use.
‘-i STR’
‘--inner=STR’
Filename of the inner PSF. This image is considered to be the
central part of the PSF. It will be cropped at the radius specified
by the option ‘--radius’, and multiplied by the factor specified by
‘--scale’. After that, it will be appended to the outer part
(input image).
‘-I STR’
‘--innerhdu=STR’
The HDU/extension of the inner PSF (option ‘--inner’).
‘-f FLT’
‘--scale=FLT’
Factor by which the inner part (‘--inner’) is multiplied. This
factor is necessary to put the two different parts of the PSF at
the same flux level. A convenient way of obtaining this value is
by using the script ‘astscript-model-scale-factor’, see *note
Invoking astscript-psf-scale-factor::. There is also a full
tutorial on using all the ‘astscript-psf-*’ installed scripts
together, see *note Building the extended PSF::. We recommend
doing that tutorial before starting to work on your own datasets.
‘-r FLT’
‘--radius=FLT’
Radius (in pixels) at which the junction of the images is done.
All pixels in the outer image within this radius (from its center)
will be replaced with the pixels of the inner image (that has been
scaled). By default, a circle is assumed for the shape of the
inner region, but this can be tweaked with ‘--axis-ratio’ and
‘--position-angle’ (see below).
‘-Q FLT’
‘--axisratio=FLT’
Axis ratio of ellipse to define the inner region. By default this
option has a value of 1.0, so all central pixels (of the outer
image) within a circle of radius ‘--radius’ are replaced with the
scaled inner image pixels. With this option, you can customize the
shape of pixels to take from the inner and outer profiles.
For a PSF, it will usually not be necessary to change this option:
even if the PSF is non-circular, the inner and outer parts will
both have the same ellipticity. So if the scale factor is chosen
accurately, using a circle to select which pixels from the inner
image to use in the outer image will be irrelevant.
‘-p FLT’
‘--position-angle=FLT’
Position angle of the ellipse (in degrees) to define which central
pixels of the outer image to replace with the scaled inner image.
Similar to ‘--axis-ratio’ (see above).
‘-t’
‘--tmpdir’
Directory to keep temporary files during the execution of the
script. If the directory does not exist at run-time, this script
will create it. By default, upon completion of the script, this
directory will be deleted. However, if you would like to keep the
intermediate files, you can use the ‘--keeptmp’ option.
‘-k’
‘--keeptmp’
Do not remove the temporary directory (see description of
‘--keeptmp’). This option is useful for debugging and checking the
outputs of internal steps.
10.8.5 Invoking astscript-psf-scale-factor
------------------------------------------
This installed script will compute the multiplicative factor (scale)
that is necessary to match the PSF to a given star. The match in flux
is done within a ring of pixels. It can also be used to compute the
scale factor to multiply the inner part of the PSF with the outer part
during the creation of a PSF. A complete tutorial is available to show
the operation of this script as a modular component to extract the PSF
of a dataset: *note Building the extended PSF::. The executable name is
‘astscript-psf-scale-factor’, with the following general template:
$ astscript-psf-scale-factor [OPTION...] FITS-file
Examples:
## Compute the scale factor for the object at (x,y)=(53,69) for
## the PSF (psf.fits). Compute it in the ring 20-30 pixels.
$ astscript-psf-scale-factor image.fits --mode=img \
--center=53,69 --normradii=20,30 --psf=psf.fits
## Iterate over a catalog with RA,Dec positions of stars that are in
## the input image to compute their scale factors.
$ asttable catalog.fits | while read -r ra dec mag; do \
astscript-psf-scale-factor image.fits \
--mode=wcs \
--psf=psf.fits \
--center=$ra,$dec --quiet \
--normradii=20,30 > scale-"$ra"-"$dec".txt; done
The input should be an image containing the star that you want to
match in flux with the PSF. The output will be a single number that is
printed on the command-line. That number is the multiplicative factor
to scale the PSF image (given to ‘--psf’) to match in flux with the
given star (which is located in ‘--center’ coordinate of the input
image). The scale factor will be calculated within the ring of pixels
specified by the option ‘--normradii’.
All the pixels within this ring will be separated from both the PSF
and input images. For the input image, around the selected coordinate;
while masking all other sources (see ‘--segment’). The finally selected
pixels of the input image will then be divided by those of the PSF
image. This gives us an image containing one scale factor per pixel.
The finally reported value is the sigma-clipped median of all the scale
factors in the finally-used pixels. To fully understand the process on
first usage, we recommend that you run this script with ‘--keeptmp’ and
inspect the files inside of the temporary directory.
The most common use-cases of this scale factor are:
1. To find the factor for joining two different parts of the same PSF,
see *note Invoking astscript-psf-unite::.
2. When modeling a star in order to subtract it using the PSF, see
*note Invoking astscript-psf-subtract::.
For a full tutorial on how to use this script along with the other
‘astscript-psf-*’ scripts in Gnuastro, please see *note Building the
extended PSF::. To allow full customizability, the following options
are available with this script.
‘-h STR’
‘--hdu=STR’
The HDU/extension of the input image to use.
‘-p STR’
‘--psf=STR’
Filename of the PSF image. The PSF is assumed to be centered in
this image.
‘-P STR’
‘--psfhdu=STR’
The HDU/extension of the PSF image.
‘-c FLT,FLT’
‘--center=FLT,FLT’
The central position of the object to scale with the PSF. This
parameter is passed to Gnuastro's Crop program make a crop for
further processing (see *note Crop::). The positions along each
dimension must be separated by a comma (<,>). The units of the
coordinates are interpreted based on the value to the ‘--mode’
option (see below).
The given coordinate for the central value can have sub-pixel
elements (for example, it falls on coordinate 123.4,567.8 of the
input image pixel grid). In such cases, after cropping, this
script will use Gnuastro's *note Warp:: to shift (or translate) the
pixel grid by $-0.4$ pixels along the horizontal and $1-0.8=0.2$
pixels along the vertical. Finally the newly added pixels (due to
the warping) will be trimmed to have your desired coordinate
exactly in the center of the central pixel of the output. This is
very important (critical!) when you are constructing the central
part of the PSF. But for the very far outer parts it may not too
effective (should be checked), or the target object may have
already been centered at the requested coordinate. In such cases,
to avoid wasting time for the warping, you can simply use
‘--nocentering’ to disable sub-pixel centering.
‘-d’
‘--nocentering’
Do not do the sub-pixel centering to a new pixel grid. See the
description of the ‘--center’ option for more.
‘-O STR’
‘--mode=STR’
Interpret the center position of the object (values given to
‘--center’) in image or WCS coordinates. This option thus accepts
only two values: ‘img’ or ‘wcs’.
‘-n INT,INT’
‘--normradii=INT,INT’
Inner (inclusive) and outer (exclusive) radii (in units of pixels)
around the central position in which the scale factor is computed.
The option takes two values separated by a comma (<,>). The first
value is the inner radius, the second is the outer radius. These
two radii define a ring of pixels around the center that is used
for obtaining the scale factor value.
‘-W INT,INT’
‘--widthinpix=INT,INT’
Size (width) of the image stamp in pixels. This is an intermediate
product computed internally by the script. By default, the size of
the stamp is automatically set to be as small as possible (i.e.,
two times the external radius of the ring specified by
‘--normradii’) to make the computation fast. As a consequence,
this option is only relevant for checking and testing that
everything is fine (debugging; it will usually not be necessary).
‘-S STR’
‘--segment=STR’
Optional filename of a segmentation image from Segment's output
(must contain the ‘CLUMPS’ and ‘OBJECT’ HDUs). For more on the
definition of "objects" and "clumps", see *note Segment::. If
given, Segment's output is used to mask all background sources from
the large foreground object (a bright star):
• Objects that are not the central object.
• Clumps (within the central object) that are not the central
clump.
The result is that all objects and clumps that contaminate the
central source are masked, while the diffuse flux of the central
object remains.
‘-s FLT,FLT’
‘--sigmaclip=FLT,FLT’
Sigma clipping parameters used in the end to find the final scale
factor from the distribution of all pixels used. For more on
sigma-clipping, see *note Sigma clipping::.
‘-t’
‘--tmpdir’
Directory to keep temporary files during the execution of the
script. If the directory does not exist at run-time, this script
will create it. By default, upon completion of the script, this
directory will be deleted. However, if you would like to keep the
intermediate files, you can use the ‘--keeptmp’ option.
‘-k’
‘--keeptmp’
Do not remove the temporary directory (see description of
‘--keeptmp’). This option is useful for debugging and checking the
outputs of internal steps.
10.8.6 Invoking astscript-psf-subtract
--------------------------------------
This installed script will put the provided PSF into a given position
within the input image (implementing sub-pixel adjustments where
necessary), and then it will subtract it. It is aimed at modeling and
subtracting the scattered light field of an input image. It is also
possible to obtain the modeled star with the PSF (and not make the
subtraction of it from the original image).
A complete tutorial is available to show the operation of this script
as a modular component to extract the PSF of a dataset: *note Building
the extended PSF::. The executable name is ‘astscript-psf-subtract’,
with the following general template:
$ astscript-psf-subtract [OPTION...] FITS-file
Examples:
## Multiply the PSF (psf.fits) by 3 and subtract it from the
## input image (image.fits) at the pixel position (x,y)=(53,69).
$ astscript-psf-subtract image.fits \
--psf=psf.fits \
--mode=img \
--scale=3 \
--center=53,69 \
--output=star-53_69.fits
## Iterate over a catalog with positions of stars that are
## in the input image. Use WCS coordinates.
$ asttable catalog.fits | while read -r ra dec mag; do
scale=$(cat scale-"$ra"_"$dec".txt)
astscript-psf-subtract image.fits \
--mode=wcs \
--psf=psf.fits \
--scale=$scale \
--center=$ra,$dec; done
The input is an image from which the star is considered. The result
is the same image but with the star subtracted (modeled by the PSF). The
modeling of the star is done with the PSF image specified with the
option ‘--psf’, and flux-scaled with the option ‘--scale’ at the
position defined by ‘--center’. Instead of obtaining the PSF-subtracted
image, it is also possible to obtain the modeled star by the PSF. To do
that, use the option ‘--modelonly’. With this option, the output will
be an image with the same size as the original one with the PSF situated
in the star coordinates and flux-scaled. In this case, the region not
covered by the PSF are set to zero values.
Note that this script works over individual objects. As a
consequence, to generate a scattered light field of many stars, it is
necessary to make multiple calls. A full description of each option is
given below.
‘-h STR’
‘--hdu=STR’
The HDU/extension of the input image to use.
‘-p STR’
‘--psf=STR’
Filename of the PSF image. The PSF is assumed to be centered in
this image.
‘-P STR’
‘--psfhdu=STR’
The HDU/extension of the PSF image.
‘-O STR’
‘--mode=STR’
Interpret the center position of the object (values given to
‘--center’) in image or WCS coordinates. This option thus accepts
only two values: ‘img’ or ‘wcs’.
‘-c FLT,FLT’
‘--center=FLT,FLT’
The central position of the object. This parameter is used in
*note Crop:: to center and crop the image. The positions along
each dimension must be separated by a comma (<,>). The number of
values given to this option must be the same as the dimensions of
the input dataset. The units of the coordinates are read based on
the value to the ‘--mode’ option, see above.
If the central position does not fall in the center of a pixel in
the input image, the PSF is resampled with sub-pixel change in the
pixel grid before subtraction.
‘-s FLT’
‘--scale=FLT’
Factor by which the PSF (‘--psf’) is multiplied. This factor is
necessary to put the PSF with the desired flux level. A convenient
way of obtaining this value is by using the script
‘astscript-scale-factor’, see *note Invoking
astscript-psf-scale-factor::. For a full tutorial on using the
‘astscript-psf-*’ scripts together, see *note Building the extended
PSF::.
‘-t’
‘--tmpdir’
Directory to keep temporary files during the execution of the
script. If the directory does not exist at run-time, this script
will create it. By default, upon completion of the script, this
directory will be deleted. However, if you would like to keep the
intermediate files, you can use the ‘--keeptmp’ option.
‘-k’
‘--keeptmp’
Do not remove the temporary directory (see description of
‘--keeptmp’). This option is useful for debugging and checking the
outputs of internal steps.
‘-m’
‘--modelonly’
Do not make the subtraction of the modeled star by the PSF. This
option is useful when the user wants to obtain the scattered light
field given by the PSF modeled star.
11 Makefile extensions (for GNU Make)
*************************************
Make (https://en.wikipedia.org/wiki/Make_(software)) is a build
automation tool. It can greatly help manage your analysis workflow,
even very complex projects with thousands of files and hundreds of
processing steps. In this book, we have discussed Make previously in
the context of parallelization (see *note How to run simultaneous
operations::). For example, Maneage (http://maneage.org) uses Make to
organize complex and reproducible workflows, see Akhlaghi et al. 2021
(https://arxiv.org/abs/2006.03018).
GNU Make is the most common and powerful implementation of Make, with
many unique additions to the core POSIX standard of Make. One of those
features is the ability to add extensions using a dynamic library (that
Gnuastro provides). For the details of this feature from GNU Make's own
manual, see its Loading dynamic objects
(https://www.gnu.org/software/make/manual/html_node/Loading-Objects.html)
section. Through this feature, Gnuastro provides additional Make
functions that are useful in the context of data analysis.
To use this feature, Gnuastro has to be built in shared library more.
Gnuastro's Make extensions will not work if you build Gnuastro without
shared libraries (for example, when you configure Gnuastro with
‘--disable-shared’ or ‘--debug’).
11.1 Loading the Gnuastro Make functions
========================================
To load Gnuastro's Make functions in your Makefile, you should use the
‘load’ command of GNU Make in your Makefile. The load command should be
given Gnuastro's ‘libgnuastro_make.so’ dynamic library, which has been
specifically written for being called by GNU Make. The generic command
looks like this (the ‘/PATH/TO’ part should be changed):
load /PATH/TO/lib/libgnuastro_make.so
Here are the possible replacements of the ‘/PATH/TO’ component:
‘/usr/local’
If you installed Gnuastro from source and did not use the
‘--prefix’ option at configuration time, you should use this base
directory.
‘/usr/’
If you installed Gnuastro through your operating system's package
manager, it is highly likely that Gnuastro's library is here.
‘~/.local’
If you installed Gnuastro from source, but used ‘--prefix’ to
install Gnuastro in your home directory (as described in *note
Installation directory::).
If you cannot find ‘libgnuastro_make.so’ in the locations above, the
command below should give you its location. It assumes that the
libraries are in the same base directory as the programs (which is
usually the case).
$ which astfits | sed -e's|bin/astfits|lib/libgnuastro_make.so|'
11.2 Makefile functions of Gnuastro
===================================
All Gnuastro Make functions start with the ‘ast-’ prefix (similar to the
programs on the command-line, but with a dash). After you have loaded
Gnuastro's shared library for Makefiles within your Makefile, you can
call these functions just like any Make function. For instructions on
how to load Gnuastro's Make functions, see *note Loading the Gnuastro
Make functions::.
There are two types of Make functions in Gnuastro's Make extensions:
1) Basic operations on text which is more general than astronomy or
Gnuastro, see *note Text functions for Makefiles::). 2) Operations that
are directly related to astronomy (mostly FITS files) and Gnuastro, see
*note Astronomy functions for Makefiles::).
*Difference between '‘=’' or '‘:=’' for variable definition* When you
define a variable with '‘=’', its value is expanded only when used, not
when defined. However, when you use '‘:=’', it is immediately expanded
when defined. Therefore the location of a '‘:=’' variable in the
Makefile matters: if used before its definition, it will be empty!
Those defined by '‘=’' can be used even before they are defined! On the
other hand, if your variable invokes functions (like ‘foreach’ or
‘wildcard’), it is better to use '‘:=’'. Otherwise, each time the value
is used, the function will be expanded (possibly may times) and this
will reduce the speed of your pipeline. For more, see the The two
flavors of variables
(https://www.gnu.org/software/make/manual/html_node/Flavors.html) in the
GNU Make manual.
11.2.1 Text functions for Makefiles
-----------------------------------
The functions described below operate on simple strings (plain text).
They are therefore generic (not limited to astronomy/FITS), but because
they are commonly necessary in astronomical data analysis pipelines and
are not available anywhere else, we have included them in Gnuastro. The
names of these functions start with ‘ast-text-*’ and each has a fully
working example to demonstrate its usage.
‘$(ast-text-to-upper STRING)’
Returns the input string but with all characters in UPPER-CASE. For
example, the following minimal Makefile will print ‘FOOO BAAR UGGH’
word of the list.
load /usr/local/lib/libgnuastro_make.so
list = fOOo bAar UggH
ulist := $(ast-text-to-upper $(list))
all:; echo $(ulist)
‘$(ast-text-to-lower STRING)’
Returns the input string but with all characters in lower-case.
For example, the following minimal Makefile will print ‘fooo baar
uggh’ word of the list.
load /usr/local/lib/libgnuastro_make.so
list = fOOo bAar UggH
llist := $(ast-text-to-lower $(list))
all:; echo $(llist)
‘$(ast-text-contains STRING, TEXT)’
Returns all white-space-separated words in ‘TEXT’ that contain the
‘STRING’, removing any words that _do not_ match. For example, the
following minimal Makefile will only print the ‘bAaz Aah’ word of
the list.
load /usr/local/lib/libgnuastro_make.so
list = fooo baar bAaz uggh Aah
all:
echo $(ast-text-contains Aa, $(list))
This can be thought of as Make's own ‘filter’ function, but if it
would accept two patterns in a format like this ‘$(filter
%Aa%,$(list))’ (for the example above). In fact, the first
sentence describing this function is taken from the Make manual's
first sentence that describes the ‘filter’ function! However,
unfortunately Make's ‘filter’ function only accepts a single ‘%’,
not two!
‘$(ast-text-not-contains STRING, TEXT)’
Returns all white-space-separated words in ‘TEXT’ that _do not_
contain the ‘STRING’, removing any words that _do not_ match. This
is the inverse of the ‘ast-text-contains’ function. For example,
the following minimal Makefile will print ‘fooo baar uggh’ word of
the list.
load /usr/local/lib/libgnuastro_make.so
list = fooo baar bAaz uggh Aah
all:
echo $(ast-text-not-contains Aa, $(list))
‘$(ast-text-prev TARGET, LIST)’
Returns the word in ‘LIST’ that is previous to ‘TARGET’. If
‘TARGET’ is the first word of the list, or is not within it at all,
this function will return an empty string (nothing). If any of the
arguments are an empty string (or only contain space characters
like '', '', new-line and etc), this function will
return an empty string (having no effect in Make).
One scenario when this function can be useful is when you want a
list of higher-level targets to always be executed in sequence
(even when Make is run in parallel). But you want their
lower-level prerequisites to be executed in parallel.
The fully working example below shows this in practice: the "final"
target depends on the sub-components ‘a.fits’, ‘b.fits’, ‘c.fits’
and ‘d.fits’. But each one of these has seven dependencies (for
example ‘a.fits’ depends on the sub-sub-components ‘a-1.fits’,
‘a-2.fits’, ‘a-3.fits’, ...). Without this function, Make will
first build all the sub-sub-components first, then the
sub-components and ultimately the final target.
When the files are small and there aren't too many of them, this is
not a problem. But when you hundreds/thousands of
sub-sub-components, your computer may not have the capacity to hold
them all in storage or RAM (during processing). In such cases, you
want to build the sub-components to built in series, but the
sub-sub-components of each sub-component to be built in parallel.
This function allows just this in an easy manner as below: the
sub-sub-components of each sub-component depend on the previous
sub-component.
To see the effect of this function put the example below in a
‘Makefile’ and run ‘make -j12’ (to simultaneously execute 12 jobs);
then comment/remove this function (so there is no prerequisite in
‘$(subsubs)’) and re-run ‘make -j12’.
# Basic settings
all: final
.SECONDEXPANSION:
load /usr/local/lib/libgnuastro_make.so
# 4 sub-components (alphabetic), each with 7
# sub-sub-components (numeric).
subids = a b c d
subsubids = 1 2 3 4 5 6 7
subs := $(foreach s, $(subids), $(s).fits)
subsubs := $(foreach s, $(subids), \
$(foreach ss, $(subsubids), \
$(s)-$(ss).fits))
# Build the sub-components:
$(subsubs): %.fits: $$(ast-text-prev \
$$(word 1, $$(subst -, ,%)).fits, \
$(subs))
@echo "$@: $^"
# Build the final components
$(subs): %.fits: $$(foreach s, $(subsubids), %-$$(s).fits)
@echo "$@: $^"
# Final
final: $(subs)
@echo "$@: $^"
As you see, when this function is present, the sub-sub-components
of each sub-component are executed in parallel, while at each
moment, only a single sub-component's prerequisites are being made.
Without this function, make first builds all the
sub-sub-components, then goes to the sub-components. There can be
any level of components between these, allowing this operation to
be as complex as necessary in your data analysis pipeline.
Unfortunately the ‘.NOTPARALLEL’ target of GNU Make doesn't allow
this level of customization.
‘$(ast-text-prev-batch TARGET, NUM, LIST)’
Returns the previous batch of ‘NUM’ words in ‘LIST’ (in relation to
the batch containing ‘TARGET’). ‘NUM’ will be interpreted as an
unsigned integer and cannot be zero. If any of the arguments are
an empty string (or only contain space characters like '',
'', new-line and etc), this function will return an empty
string (having no effect in Make). In the special case that
‘NUM=1’, this is equivalent to the ‘ast-text-prev’ function that is
described above.
Here is one scenario where this function is useful: in astronomy
datasets are can easily be very large. Therefore, some Make
recipes in your pipeline may require a lot of memory; such that
executing them on all the available threads (for example 12 threads
with ‘-j12’) will immediately occupy all your RAM, causing a crash
in your pipeline. However, let's assume that you have sufficient
RAM to execute 4 targets of those recipes in parallel. Therefore
while you want all the other steps of your pipeline to be using all
12 threads, you want one rule to only build 4 targets at any time.
But before starting to use this function, also see
‘ast-text-prev-batch-by-ram’.
The example below demonstrates the usage of this function in a
minimal working example of the scenario above: we want to build 15
targets, but in batches of 4 target at a time, irrespective of how
many threads Make was executed with.
load /usr/local/lib/libgnuastro_make.so
.SECONDEXPANSION:
targets := $(foreach i,$(shell seq 15),a-$(i).fits)
all: $(targets)
$(targets): $$(ast-text-prev-batch $$@,4,$(targets))
@echo "$@: $^"
If you place the example above in a plain-text file called
‘Makefile’ (correcting for the TAB at the start of the recipe), and
run Make on 12 threads like below, you will see the following
output. The targets in each batch are not ordered (and the order
may change in different runs) because they have been run in
parallel.
$ make -j12
a-1.fits:
a-3.fits:
a-2.fits:
a-4.fits:
a-5.fits: a-1.fits a-2.fits a-3.fits a-4.fits
a-6.fits: a-1.fits a-2.fits a-3.fits a-4.fits
a-8.fits: a-1.fits a-2.fits a-3.fits a-4.fits
a-7.fits: a-1.fits a-2.fits a-3.fits a-4.fits
a-9.fits: a-5.fits a-6.fits a-7.fits a-8.fits
a-11.fits: a-5.fits a-6.fits a-7.fits a-8.fits
a-12.fits: a-5.fits a-6.fits a-7.fits a-8.fits
a-10.fits: a-5.fits a-6.fits a-7.fits a-8.fits
a-13.fits: a-9.fits a-10.fits a-11.fits a-12.fits
a-15.fits: a-9.fits a-10.fits a-11.fits a-12.fits
a-14.fits: a-9.fits a-10.fits a-11.fits a-12.fits
Any other rule that is later added to this make file (as a
prerequisite/parent of ‘targets’ or as a child of ‘targets’) will
be run on 12 threads.
‘$(ast-text-prev-batch-by-ram TARGET, NEEDED_RAM_GB, LIST)’
Similar to ‘ast-text-prev-batch’, but instead of taking the number
of words/files in each batch, this function takes the maximum
amount of RAM that is needed by one instance of the recipe.
Through the ‘NEEDED_RAM_GB’ argument, you should specify the amount
of ram that a _single_ instance of the recipe in this rule needs.
If any of the arguments are an empty string (or only contain space
characters like '', '', new-line and etc), this
function will return an empty string (having no effect in Make).
When the needed RAM is larger than the available RAM only one job
will be done at a time (similar to ‘ast-text-prev’).
The number of files in each batch is calculated internally by
reading the available RAM on the system at the moment Make calls
this function. Therefore this function is more generalizable to
different computers (with very different RAM and/or CPU threads).
But to avoid overlapping with other rules that may consume a lot of
RAM, it is better to design your Makefile such that other rules are
only executed once all instances of this rule have been completed.
For example, assume every instance of one rule in your Makefile
requires a maximum of 5.2 GB of RAM during its execution, and your
computer has 32 GB of RAM and 2 threads. In this case, you do not
need to manage the targets at all: at the worst moment your
pipeline will consume 10.4GB of RAM (much smaller than the 32GB of
RAM that you have). However, you later run the same pipeline on
another machine with identical RAM, but 12 threads! In this case,
you will need $5.2\times12=62.4$GB of RAM; but the new system
doesn't have that much RAM, causing your pipeline to crash. If you
used ‘ast-text-prev-batch’ function (described above) to manage
these hardware limitations, you would have to manually change the
number on every new system; this is inconvenient, can cause many
bugs, and requires manual intervention (not making your pipeline
automatic).
The ‘ast-text-prev-batch-by-ram’ function was designed as a
solution to the problem above: it will read the amount of available
RAM at the time that Make starts (before the recipes in your
pipeline are actually executed). From the value to
‘NEEDED_RAM_GB’, it will then estimate how many instances of that
recipe can be executed in parallel without breaching the available
RAM of the system. Therefore it is important to not run another
heavy RAM consumer on the system while your pipeline is being
executed. Note that this function reads the available RAM, not
total RAM; it therefore accounts for the background operations of
the operating system or graphic user environment that are running
in parallel to your pipeline; and assumes they will remain at the
same level.
The fully working example below shows the usage of this function in
a scenario where we assume the recipe requires 4.2GB of RAM for
each target.
load /usr/local/lib/libgnuastro_make.so
.SECONDEXPANSION:
targets := $(foreach i,$(shell seq 13),$(i).fits)
all: $(targets)
$(targets): $$(ast-text-prev-batch-by-ram $$@,4.2,$(targets))
@echo "$@: $^"
Once the contents above are placed in a ‘Makefile’ and you execute
the command below in a system with about 27GB of available RAM
(total RAM is 32GB; the 5GB difference is used by the operating
system and other background programs), you will get an output like
below.
$ make -j12
1.fits:
2.fits:
3.fits:
4.fits:
5.fits:
6.fits:
7.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
8.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
11.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
10.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
9.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
12.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
13.fits: 7.fits 8.fits 9.fits 10.fits 11.fits 12.fits
Depending on the amount of available RAM on your system, you will
get a different output. To see the effect, you can decrease or
increase the amount of required RAM (‘4.2’ in the example above).
*What is the maximum RAM required by my command?* Put a
'‘/usr/bin/time --format=%M’' prefix behind your full command
(including any options and arguments). For example like this for a
call to Gnuastro's Warp program:
/usr/bin/time --format=%M astwarp image.fits
After the regular outputs of the program, you will see a number on
the last line. This number is the maximum used RAM (in
_kilobytes_) during the execution of the program. Later, you can
convert this to Gigabytes (to feed into this function) by dividing
it to $10^6$.
11.2.2 Astronomy functions for Makefiles
----------------------------------------
FITS files (the standard data format in astronomy) have unique features
(header keywords and HDUs) that can greatly help designing workflows in
Makefiles. The Makefile extension functions of this section allow you
to optimally use those features within your pipelines. Besides FITS,
when designing your workflow/pipeline with Gnuastro, there are also
special features like version checking that simplify your design.
‘$(ast-version-is STRING)’
Returns ‘1’ if the version of the used Gnuastro is equal to
‘STRING’, and ‘0’ otherwise. This is useful/critical for obtaining
reproducible results on different systems. It can be used in
combination with Conditionals in Make
(https://www.gnu.org/software/make/manual/html_node/Conditionals.html)
to ensure the required version of Gnuastro is going to be used in
your workflow.
For example, in the minimal working Makefile below, we are using it
to specify if the default (first) target (‘all’) should have any
prerequisites (and let the workflow start), or if it should simply
print a message (that the required version of Gnuastro isn't
installed) and abort (without any prerequisites).
load /usr/local/lib/libgnuastro_make.so
gnuastro-version = 0.19
ifeq ($(ast-version-is $(gnuastro-version)),1)
all: paper.pdf
else
all:; @echo "Please use Gnuastro $(gnuastro-version)"
endif
result.fits: input.fits
astnoisechisel $< --output=$@
paper.pdf: result.fits
pdflatex --halt-on-error paper.tex
‘$(ast-fits-with-keyvalue KEYNAME, KEYVALUES, HDU, FITS_FILES)’
Will select only the FITS files (from a list of many in
‘FITS_FILES’, non-FITS files are ignored), where the ‘KEYNAME’
keyword has the value(s) given in ‘KEYVALUES’. Only the HDU given
in the ‘HDU’ argument will be checked. According to the FITS
standard, the keyword name is not case sensitive, but the keyword
value is.
For example, if you have many FITS files in the ‘/datasets/images’
directory, the minimal Makefile below will put those with a value
of ‘BAR’ or ‘BAZ’ for the ‘FOO’ keyword in HDU number ‘1’ in the
‘selected’ Make variable. Notice how there is no comma between
‘BAR’ and ‘BAZ’: you can specify any series of values.
load /usr/local/lib/libgnuastro_make.so
files := $(wildcard /datasets/images/*.fits)
selected := $(ast-fits-with-keyvalue FOO, BAR BAZ, 1, $(files))
all:
echo "Full: $(words $(files)) files";
echo "Selected: $(words $(selected)) files"
‘$(ast-fits-unique-keyvalues KEYNAME, HDU, FITS_FILES)’
Will return the unique values given to the given FITS keyword
(‘KEYNAME’) in the given HDU of all the input FITS files (non-FITS
files are ignored). For example, after the commands below, the
‘keyvalues’ variable will contain the unique values given to the
‘FOO’ keyword in HDU number 1 of all the FITS files in
‘/datasets/images/*.fits’.
files := $(wildcard /datasets/images/*.fits)
keyvalues := $(ast-fits-unique-keyvalues FOO, 1, $(files))
This is useful when you do not know the full range of values
a-priori. For example, let's assume that you are looking at a
night's observations with a telescope and the purpose of the FITS
image is written in the ‘OBJECT’ keyword of the image (which we can
assume is in HDU number 1). This keyword can have the name of the
various science targets (for example, ‘NGC123’ and ‘M31’) and
calibration targets (for example, ‘BIAS’ and ‘FLAT’). The list of
science targets is different from project to project, such that in
one night, you can observe multiple projects. But the calibration
frames have unique names. Knowing the calibration keyword values,
you can extract the science keyword values of the night with the
command below (feeding the output of this function to Make's
‘filter-out’ function).
calib = BIAS FLAT
files := $(wildcard /datasets/images/*.fits)
science := $(filter-out $(calib), \
$(ast-fits-unique-keyvalues OBJECT, 1, $(files)))
The ‘science’ variable will now contain the unique science targets
that were observed in your selected FITS images. You can use it to
group the various exposures together in the next stages to make
separate stacks of deep images for each science target (you can
select FITS files based on their keyword values using the
‘ast-fits-with-keyvalue’ function, which is described separately in
this section).
12 Library
**********
Each program in Gnuastro that was discussed in the prior chapters (or
any program in general) is a collection of functions that is compiled
into one executable file which can communicate directly with the outside
world. The outside world in this context is the operating system. By
communication, we mean that control is directly passed to a program from
the operating system with a (possible) set of inputs and after it is
finished, the program will pass control back to the operating system.
For programs written in C and C++, the unique ‘main’ function is in
charge of this communication.
Similar to a program, a library is also a collection of functions
that is compiled into one executable file. However, unlike programs,
libraries do not have a ‘main’ function. Therefore they cannot
communicate directly with the outside world. This gives you the chance
to write your own ‘main’ function and call library functions from within
it. After compiling your program into a binary executable, you just
have to _link_ it to the library and you are ready to run (execute) your
program. In this way, you can use Gnuastro at a much lower-level, and
in combination with other libraries on your system, you can
significantly boost your creativity.
This chapter starts with a basic introduction to libraries and how
you can use them in *note Review of library fundamentals::. The
separate functions in the Gnuastro library are then introduced
(classified by context) in *note Gnuastro library::. If you end up
routinely using a fixed set of library functions, with a well-defined
input and output, it will be much more beneficial if you define a
program for the job. Therefore, in its *note Version controlled
source::, Gnuastro comes with the *note The TEMPLATE program:: to easily
define your own programs(s).
12.1 Review of library fundamentals
===================================
Gnuastro's libraries are written in the C programming language. In
*note Why C::, we have thoroughly discussed the reasons behind this
choice. C was actually created to write Unix, thus understanding the
way C works can greatly help in effectively using programs and libraries
in all Unix-like operating systems. Therefore, in the following
subsections some important aspects of C, as it relates to libraries (and
thus programs that depend on them) on Unix are reviewed. First we will
discuss header files in *note Headers:: and then go onto *note
Linking::. This section finishes with *note Summary and example on
libraries::. If you are already familiar with these concepts, please
skip this section and go directly to *note Gnuastro library::.
In theory, a full operating system (or any software) can be written
as one function. Such a software would not need any headers or linking
(that are discussed in the subsections below). However, writing that
single function and maintaining it (adding new features, fixing bugs,
documentation, etc.) would be a programmer or scientist's worst
nightmare! Furthermore, all the hard work that went into creating it
cannot be reused in other software: every other programmer or scientist
would have to re-invent the wheel. The ultimate purpose behind
libraries (which come with headers and have to be linked) is to address
this problem and increase modularity: "the degree to which a system's
components may be separated and recombined" (from Wikipedia). The more
modular the source code of a program or library, the easier maintaining
it will be, and all the hard work that went into creating it can be
reused for a wider range of problems.
12.1.1 Headers
--------------
C source code is read from top to bottom in the source file, therefore
program components (for example, variables, data structures and
functions) should all be _defined_ or _declared_ closer to the top of
the source file: before they are used. _Defining_ something in C or C++
is jargon for providing its full details. _Declaring_ it, on the
other-hand, is jargon for only providing the minimum information needed
for the compiler to pass it temporarily and fill in the detailed
definition later.
For a function, the _declaration_ only contains the inputs and their
data-types along with the output's type(1). The _definition_ adds to
the declaration by including the exact details of what operations are
done to the inputs to generate the output. As an example, take this
simple summation function:
double
sum(double a, double b)
{
return a + b;
}
What you see above is the _definition_ of this function: it shows you
(and the compiler) exactly what it does to the two ‘double’ type inputs
and that the output also has a ‘double’ type. Note that a function's
internal operations are rarely so simple and short, it can be
arbitrarily long and complicated. This unreasonably short and simple
function was chosen here for ease of reading. The declaration for this
function is:
double
sum(double a, double b);
You can think of a function's declaration as a building's address in the
city, and the definition as the building's complete blueprints. When
the compiler confronts a call to a function during its processing, it
does not need to know anything about how the inputs are processed to
generate the output. Just as the postman does not need to know the
inner structure of a building when delivering the mail. The declaration
(address) is enough. Therefore by _declaring_ the functions once at the
start of the source files, we do not have to worry about _defining_ them
after they are used.
Even for a simple real-world operation (not a simple summation like
above!), you will soon need many functions (for example, some for
reading/preparing the inputs, some for the processing, and some for
preparing the output). Although it is technically possible, managing
all the necessary functions in one file is not easy and is contrary to
the modularity principle (see *note Review of library fundamentals::),
for example, the functions for preparing the input can be usable in your
other projects with a different processing. Therefore, as we will see
later (in *note Linking::), the functions do not necessarily need to be
defined in the source file where they are used. As long as their
definitions are ultimately linked to the final executable, everything
will be fine. For now, it is just important to remember that the
functions that are called within one source file must be declared within
the source file (declarations are mandatory), but not necessarily
defined there.
In the spirit of modularity, it is common to define contextually
similar functions in one source file. For example, in Gnuastro,
functions that calculate the median, mean and other statistical
functions are defined in ‘lib/statistics.c’, while functions that deal
directly with FITS files are defined in ‘lib/fits.c’.
Keeping the definition of similar functions in a separate file
greatly helps their management and modularity, but this fact alone does
not make things much easier for the caller's source code: recall that
while definitions are optional, declarations are mandatory. So if this
was all, the caller would have to manually copy and paste (_include_)
all the declarations from the various source files into the file they
are working on now. To address this problem, programmers have adopted
the header file convention: the header file of a source code contains
all the declarations that a caller would need to be able to use any of
its functions. For example, in Gnuastro, ‘lib/statistics.c’ (file
containing function definitions) comes with ‘lib/gnuastro/statistics.h’
(only containing function declarations).
The discussion above was mainly focused on functions, however, there
are many more programming constructs such as preprocessor macros and
data structures. Like functions, they also need to be known to the
compiler when it confronts a call to them. So the header file also
contains their definitions or declarations when they are necessary for
the functions.
Preprocessor macros (or macros for short) are replaced with their
defined value by the preprocessor before compilation. Conventionally
they are written only in capital letters to be easily recognized. It is
just important to understand that the compiler does not see the macros,
it sees their fixed values. So when a header specifies macros you can
do your programming without worrying about the actual values. The
standard C types (for example, ‘int’, or ‘float’) are very low-level and
basic. We can collect multiple C types into a _structure_ for a
higher-level way to keep and pass-along data. See *note Generic data
container:: for some examples of macros and data structures.
The contents in the header need to be _include_d into the caller's
source code with a special preprocessor command: ‘#include
’. As the name suggests, the _preprocessor_ goes
through the source code prior to the processor (or compiler). One of
its jobs is to include, or merge, the contents of files that are
mentioned with this directive in the source code. Therefore the
compiler sees a single entity containing the contents of the main file
and all the included files. This allows you to include many (sometimes
thousands of) declarations into your code with only one line. Since the
headers are also installed with the library into your system, you do not
even need to keep a copy of them for each separate program, making
things even more convenient.
Try opening some of the ‘.c’ files in Gnuastro's ‘lib/’ directory
with a text editor to check out the include directives at the start of
the file (after the copyright notice). Let's take ‘lib/fits.c’ as an
example. You will notice that Gnuastro's header files (like
‘gnuastro/fits.h’) are indeed within this directory (the ‘fits.h’ file
is in the ‘gnuastro/’ directory). You will notice that files like
‘stdio.h’, or ‘string.h’ are not in this directory (or anywhere within
Gnuastro).
On most systems the basic C header files (like ‘stdio.h’ and
‘string.h’ mentioned above) are located in ‘/usr/include/’(2). Your
compiler is configured to automatically search that directory (and
possibly others), so you do not have to explicitly mention these
directories. Go ahead, look into the ‘/usr/include’ directory and find
‘stdio.h’ for example. When the necessary header files are not in those
specific libraries, the preprocessor can also search in places other
than the current directory. You can specify those directories with this
preprocessor option(3):
‘-I DIR’
"Add the directory ‘DIR’ to the list of directories to be searched
for header files. Directories named by '-I' are searched before
the standard system include directories. If the directory ‘DIR’ is
a standard system include directory, the option is ignored to
ensure that the default search order for system directories and the
special treatment of system headers are not defeated..." (quoted
from the GNU Compiler Collection manual). Note that the space
between and the directory is optional and commonly not used.
If the preprocessor cannot find the included files, it will abort
with an error. In fact a common error when building programs that
depend on a library is that the compiler does not know where a library's
header is (see *note Known issues::). So you have to manually tell the
compiler where to look for the library's headers with the ‘-I’ option.
For a small software with one or two source files, this can be done
manually (see *note Summary and example on libraries::). However, to
enhance modularity, Gnuastro (and most other bin/libraries) contain many
source files, so the compiler is invoked many times(4). This makes
manual addition or modification of this option practically impossible.
To solve this problem, in the GNU build system, there are
conventional environment variables for the various kinds of compiler
options (or flags). These environment variables are used in every call
to the compiler (they can be empty). The environment variable used for
the C preprocessor (or CPP) is ‘CPPFLAGS’. By giving ‘CPPFLAGS’ a value
once, you can be sure that each call to the compiler will be affected.
See *note Known issues:: for an example of how to set this variable at
configure time.
As described in *note Installation directory::, you can select the
top installation directory of a software using the GNU build system,
when you ‘./configure’ it. All the separate components will be put in
their separate sub-directory under that, for example, the programs,
compiled libraries and library headers will go into ‘$prefix/bin’
(replace ‘$prefix’ with a directory), ‘$prefix/lib’, and
‘$prefix/include’ respectively. For enhanced modularity, libraries that
contain diverse collections of functions (like GSL, WCSLIB, and
Gnuastro), put their header files in a sub-directory unique to
themselves. For example, all Gnuastro's header files are installed in
‘$prefix/include/gnuastro’. In your source code, you need to keep the
library's sub-directory when including the headers from such libraries,
for example, ‘#include ’(5). Not all libraries need to
follow this convention, for example, CFITSIO only has one header
(‘fitsio.h’) which is directly installed in ‘$prefix/include’.
---------- Footnotes ----------
(1) Recall that in C, functions only have one output.
(2) The ‘include/’ directory name is taken from the pre-processor's
‘#include’ directive, which is also the motivation behind the 'I' in the
‘-I’ option to the pre-processor.
(3) Try running Gnuastro's ‘make’ and find the directories given to
the compiler with the ‘-I’ option.
(4) Nearly every command you see being executed after running ‘make’
is one call to the compiler.
(5) the top ‘$prefix/include’ directory is usually known to the
compiler
12.1.2 Linking
--------------
To enhance modularity, similar functions are defined in one source file
(with a ‘.c’ suffix, see *note Headers:: for more). After running
‘make’, each human-readable, ‘.c’ file is translated (or compiled) into
a computer-readable "object" file (ending with ‘.o’). Note that object
files are also created when building programs, they are not particular
to libraries. Try opening Gnuastro's ‘lib/’ and ‘bin/progname/’
directories after running ‘make’ to see these object files(1).
Afterwards, the object files are _linked_ together to create an
executable program or a library.
The object files contain the full definition of the functions in the
respective ‘.c’ file along with a list of any other function (or
generally "symbol") that is referenced there. To get a list of those
functions you can use the ‘nm’ program which is part of GNU Binutils.
For example, from the top Gnuastro directory, run:
$ nm bin/arithmetic/arithmetic.o
This will print a list of all the functions (more generally, 'symbols')
that were called within ‘bin/arithmetic/arithmetic.c’ along with some
further information (for example, a ‘T’ in the second column shows that
this function is actually defined here, ‘U’ says that it is undefined
here). Try opening the ‘.c’ file to check some of these functions for
yourself. Run ‘info nm’ for more information.
To recap, the _compiler_ created the separate object files mentioned
above for each ‘.c’ file. The _linker_ will then combine all the
symbols of the various object files (and libraries) into one program or
library. In the case of Arithmetic (a program) the contents of the
object files in ‘bin/arithmetic/’ are copied (and re-ordered) into one
final executable file which we can run from the operating system.
There are two ways to _link_ all the necessary symbols: static and
dynamic/shared. When the symbols (computer-readable function
definitions in most cases) are copied into the output, it is called
_static_ linking. When the symbols are kept in their original file and
only a reference to them is kept in the executable, it is called
_dynamic_, or _shared_ linking.
Let's have a closer look at the executable to understand this better:
we will assume you have built Gnuastro without any customization and
installed Gnuastro into the default ‘/usr/local/’ directory (see *note
Installation directory::). If you tried the ‘nm’ command on one of
Arithmetic's object files above, then with the command below you can
confirm that all the functions that were defined in the object file
above (had a ‘T’ in the second column) are also defined in the
‘astarithmetic’ executable:
$ nm /usr/local/bin/astarithmetic
These symbols/function have been statically linked (copied) in the final
executable. But you will notice that there are still many undefined
symbols in the executable (those with a ‘U’ in the second column). One
class of such functions are Gnuastro's own library functions that start
with '‘gal_’':
$ nm /usr/local/bin/astarithmetic | grep gal_
These undefined symbols (functions) are present in another file and
will be linked to the Arithmetic program every time you run it.
Therefore they are known as dynamically _linked_ libraries (2). As we
saw above, static linking is done when the executable is being built.
However, when a program is dynamically linked to a library, at
build-time, the library's symbols are only checked with the available
libraries: they are not actually copied into the program's executable.
Every time you run the program, the (dynamic) linker will be activated
and will try to link the program to the installed library before the
program starts.
If you want all the libraries to be statically linked to the
executables, you have to tell Libtool (which Gnuastro uses for the
linking) to disable shared libraries at configure time(3):
$ configure --disable-shared
Try configuring Gnuastro with the command above, then build and install
it (as described in *note Quick start::). Afterwards, check the ‘gal_’
symbols in the installed Arithmetic executable like before. You will
see that they are actually copied this time (have a ‘T’ in the second
column). If the second column does not convince you, look at the
executable file size with the following command:
$ ls -lh /usr/local/bin/astarithmetic
It should be around 4.2 Megabytes with this static linking. If you
configure and build Gnuastro again with shared libraries enabled (which
is the default), you will notice that it is roughly 100 Kilobytes!
This huge difference would have been very significant in the old
days, but with the roughly Terabyte storage drives commonly in use
today, it is negligible. Fortunately, output file size is not the only
benefit of dynamic linking: since it links to the libraries at run-time
(rather than build-time), you do not have to rebuild a higher-level
program or library when an update comes for one of the lower-level
libraries it depends on. You just install the new low-level library and
it will automatically be used/linked next time in the programs that use
it. To be fair, this also creates a few complications(4):
• Reproducibility: Even though your high-level tool has the same
version as before, with the updated library, you might not get the
same results.
• Broken links: if some functions have been changed or removed in the
updated library, then the linker will abort with an error at
run-time. Therefore you need to rebuild your higher-level program
or library.
To see a list of all the shared libraries that are needed for a
program or a shared library to run, you can use GNU C library's ‘ldd’(5)
program, for example:
$ ldd /usr/local/bin/astarithmetic
Library file names (in their installation directory) start with a
‘lib’ and their ending (suffix) shows if they are static (‘.a’) or
dynamic (‘.so’), as described below. The name of the library is in the
middle of these two, for example, ‘libgsl.a’ or ‘libgnuastro.a’ (GSL and
Gnuastro's static libraries), and ‘libgsl.so.23.0.0’ or
‘libgnuastro.so.4.0.0’ (GSL and Gnuastro's shared library, the numbers
may be different).
• A static library is known as an archive file and has the ‘.a’
suffix. A static library is not an executable file.
• A shared library ends with the ‘.so.X.Y.Z’ suffix and is
executable. The three numbers in the suffix, describe the version
of the shared library. Shared library versions are defined to
allow multiple versions of a shared library simultaneously on a
system and to help detect possible updates in the library and
programs that depend on it by the linker.
It is very important to mention that this version number is
different from the software version number (see *note Version
numbering::), so do not confuse the two. See the "Library
interface versions" chapter of GNU Libtool for more.
For each shared library, we also have two symbolic links ending
with ‘.so.X’ and ‘.so’. They are automatically set by the
installer, but you can change them (point them to another version
of the library) when you have multiple versions of a library on
your system.
Libraries that are built with GNU Libtool (including Gnuastro and its
dependencies), build both static and dynamic libraries by default and
install them in ‘prefix/lib/’ directory (for more on ‘prefix’, see *note
Installation directory::). In this way, programs depending on the
libraries can link with them however they prefer. See the contents of
‘/usr/local/lib’ with the command below to see both the static and
shared libraries available there, along with their executable nature and
the symbolic links:
$ ls -l /usr/local/lib/
To link with a library, the linker needs to know where to find the
library. _At compilation time_, these locations can be passed to the
linker with two separate options (see *note Summary and example on
libraries:: for an example) as described below. You can see these
options and their usage in practice while building Gnuastro (after
running ‘make’):
‘-L DIR’
Will tell the linker to look into ‘DIR’ for the libraries. For
example, ‘-L/usr/local/lib’, or ‘-L/home/yourname/.local/lib’. You
can make multiple calls to this option, so the linker looks into
several directories at compilation time. Note that the space
between and the directory is optional and commonly ignored
(written as ‘-LDIR’).
‘-lLIBRARY’
Specify the unique library identifier/name (not containing
directory or shared/dynamic nature) to be linked with the
executable. As discussed above, library file names have fixed
parts which must not be given to this option. So ‘-lgsl’ will
guide the linker to either look for ‘libgsl.a’ or ‘libgsl.so’
(depending on the type of linking it is suppose to do). You can
link many libraries by repeated calls to this option.
*Very important: * The place of this option on the compiler's
command matters. This is often a source of confusion for
beginners, so let's assume you have asked the linker to link with
library A using this option. As soon as the linker confronts this
option, it looks into the list of the undefined symbols it has
found until that point and does a search in library A for any of
those symbols. If any pending undefined symbol is found in library
A, it is used. After the search in undefined symbols is complete,
the contents of library A are completely discarded from the
linker's memory. Therefore, if a later object file or library uses
an unlinked symbol in library A, the linker will abort after it has
finished its search in all the input libraries or object files.
As an example, Gnuastro's ‘gal_fits_img_read’ function depends on
the ‘fits_read_pix’ function of CFITSIO (specified with
‘-lcfitsio’, which in turn depends on the cURL library, called with
‘-lcurl’). So the proper way to link something that uses this
function is ‘-lgnuastro -lcfitsio -lcurl’. If instead, you give:
‘-lcfitsio -lgnuastro’ the linker will complain and abort. To
avoid such linking complexities when using Gnuastro's library, we
recommend using *note BuildProgram::.
If you have compiled and linked your program with a dynamic library,
then the dynamic linker also needs to know the location of the libraries
after building the program: _every time_ the program is run afterwards.
Therefore, it may happen that you do not get any errors when
compiling/linking a program, but are unable to run your program because
of a failure to find a library. This happens because the dynamic linker
has not found the dynamic library _at run time_.
To find the dynamic libraries at run-time, the linker looks into the
paths, or directories, in the ‘LD_LIBRARY_PATH’ environment variable.
For a discussion on environment variables, especially search paths like
‘LD_LIBRARY_PATH’, and how you can add new directories to them, see
*note Installation directory::.
---------- Footnotes ----------
(1) Gnuastro uses GNU Libtool for portable library creation. Libtool
will also make a ‘.lo’ file for each ‘.c’ file when building libraries
(‘.lo’ files are human-readable).
(2) Do not confuse dynamically _linked_ libraries with dynamically
_loaded_ libraries. The former (that is discussed here) are only loaded
once at the program startup. However, the latter can be loaded anytime
during the program's execution, they are also known as plugins.
(3) Libtool is very common and is commonly used. Therefore, you can
use this option to configure on most programs using the GNU build system
if you want static linking.
(4) Both of these can be avoided by joining the mailing lists of the
lower-level libraries and checking the changes in newer versions before
installing them. Updates that result in such behaviors are generally
heavily emphasized in the release notes.
(5) If your operating system is not using the GNU C library, you
might need another tool.
12.1.3 Summary and example on libraries
---------------------------------------
After the mostly abstract discussions of *note Headers:: and *note
Linking::, we will give a small tutorial here. But before that, let's
recall the general steps of how your source code is prepared, compiled
and linked to the libraries it depends on so you can run it:
1. The *preprocessor* includes the header (‘.h’) files into the
function definition (‘.c’) files, expands preprocessor macros.
Generally the preprocessor prepares the human-readable source for
compilation (reviewed in *note Headers::).
2. The *compiler* will translate (compile) the human-readable contents
of each source (merged ‘.c’ and the ‘.h’ files, or generally the
output of the preprocessor) into the computer-readable code of ‘.o’
files.
3. The *linker* will link the called function definitions from various
compiled files to create one unified object. When the unified
product has a ‘main’ function, this function is the product's only
entry point, enabling the operating system or user to directly
interact with it, so the product is a program. When the product
does not have a ‘main’ function, the linker's product is a library
and it is exported functions can be linked to other executables (it
has many entry points).
The GNU Compiler Collection (or GCC for short) will do all three
steps. So as a first example, from Gnuastro's source, go to
‘tests/lib/’. This directory contains the library tests, you can use
these as some simple tutorials. For this demonstration, we will compile
and run the ‘arraymanip.c’. This small program will call Gnuastro
library for some simple operations on an array (open it and have a
look). To compile this program, run this command inside the directory
containing it.
$ gcc arraymanip.c -lgnuastro -lm -o arraymanip
The two ‘-lgnuastro’ and ‘-lm’ options (in this order) tell GCC to first
link with the Gnuastro library and then with C's math library. The ‘-o’
option is used to specify the name of the output executable, without it
the output file name will be ‘a.out’ (on most OSs), independent of your
input file name(s).
If your top Gnuastro installation directory (let's call it ‘$prefix’,
see *note Installation directory::) is not recognized by GCC, you will
get preprocessor errors for unknown header files. Once you fix it, you
will get linker errors for undefined functions. To fix both, you should
run GCC as follows: additionally telling it which directories it can
find Gnuastro's headers and compiled library (see *note Headers:: and
*note Linking::):
$ gcc -I$prefix/include -L$prefix/lib arraymanip.c -lgnuastro -lm \
-o arraymanip
This single command has done all the preprocessor, compilation and
linker operations. Therefore no intermediate files (object files in
particular) were created, only a single output executable was created.
You are now ready to run the program with:
$ ./arraymanip
The Gnuastro functions called by this program only needed to be
linked with the C math library. But if your program needs WCS
coordinate transformations, needs to read a FITS file, needs special
math operations (which include its linear algebra operations), or you
want it to run on multiple CPU threads, you also need to add these
libraries in the call to GCC: ‘-lgnuastro -lwcs -lcfitsio -lgsl
-lgslcblas -pthread -lm’. In *note Gnuastro library::, where each
function is documented, it is mentioned which libraries (if any) must
also be linked when you call a function. If you feel all these linkings
can be confusing, please consider Gnuastro's *note BuildProgram::
program.
12.2 BuildProgram
=================
The number and order of libraries that are necessary for linking a
program with Gnuastro library might be too confusing when you need to
compile a small program for one particular job (with one source file).
BuildProgram will use the information gathered during configuring
Gnuastro and link with all the appropriate libraries on your system.
This will allow you to easily compile, link and run programs that use
Gnuastro's library with one simple command and not worry about which
libraries to link to, or the linking order.
BuildProgram uses GNU Libtool to find the necessary libraries to link
against (GNU Libtool is the same program that builds all of Gnuastro's
libraries and programs when you run ‘make’). So in the future, if
Gnuastro's prerequisite libraries change or other libraries are added,
you do not have to worry, you can just run BuildProgram and internal
linking will be done correctly.
*BuildProgram requires GNU Libtool:* BuildProgram depends on GNU
Libtool, other implementations do not have some necessary features. If
GNU Libtool is not available at Gnuastro's configure time, you will get
a notice at the end of the configuration step and BuildProgram will not
be built or installed. Please see *note Optional dependencies:: for
more information.
12.2.1 Invoking BuildProgram
----------------------------
BuildProgram will compile and link a C source program with Gnuastro's
library and all its dependencies, greatly facilitating the compilation
and running of small programs that use Gnuastro's library. The
executable name is ‘astbuildprog’ with the following general template:
$ astbuildprog [OPTION...] C_SOURCE_FILE
One line examples:
## Compile, link and run `myprogram.c':
$ astbuildprog myprogram.c
## Similar to previous, but with optimization and compiler warnings:
$ astbuildprog -Wall -O2 myprogram.c
## Compile and link `myprogram.c', then run it with `image.fits'
## as its argument:
$ astbuildprog myprogram.c image.fits
## Also look in other directories for headers and linking:
$ astbuildprog -Lother -Iother/dir myprogram.c
## Just build (compile and link) `myprogram.c', do not run it:
$ astbuildprog --onlybuild myprogram.c
If BuildProgram is to run, it needs a C programming language source
file as input. By default it will compile and link the given source
into a final executable program and run it. The built executable name
can be set with the optional ‘--output’ option. When no output name is
set, BuildProgram will use Gnuastro's *note Automatic output:: system to
remove the suffix of the input source file (usually ‘.c’) and use the
resulting name as the built program name.
For the full list of options that BuildProgram shares with other
Gnuastro programs, see *note Common options::. You may also use
Gnuastro's *note Configuration files:: to specify other
libraries/headers to use for special directories and not have to type
them in every time.
The C compiler can be chosen with the ‘--cc’ option, or environment
variables, please see the description of ‘--cc’ for more. The two
common ‘LDFLAGS’ and ‘CPPFLAGS’ environment variables are also checked
and used in the build by default. Note that they are placed after the
values to the corresponding options ‘--includedir’ and ‘--linkdir’.
Therefore BuildProgram's own options take precedence. Using environment
variables can be disabled with the ‘--noenv’ option. Just note that
BuildProgram also keeps the important flags in these environment
variables in its configuration file. Therefore, in many cases, even
though you may needed them to build Gnuastro, you will not need them in
BuildProgram.
The first argument is considered to be the C source file that must be
compiled and linked. Any other arguments (non-option tokens on the
command-line) will be passed onto the program when BuildProgram wants to
run it. Recall that by default BuildProgram will run the program after
building it. This behavior can be disabled with the ‘--onlybuild’
option.
When the ‘--quiet’ option (see *note Operating mode options::) is not
called, BuildPrograms will print the compilation and running commands.
Once your program grows and you break it up into multiple files (which
are much more easily managed with Make), you can use the linking flags
of the non-quiet output in your ‘Makefile’.
‘-c STR’
‘--cc=STR’
C compiler to use for the compilation, if not given environment
variables will be used as described in the next paragraph. If the
compiler is in your system's search path, you can simply give its
name, for example, ‘--cc=gcc’. If it is not in your system's
search path, you can give its full path, for example,
‘--cc=/path/to/your/custom/cc’.
If this option has no value after parsing the command-line and all
configuration files (see *note Configuration file precedence::),
then BuildProgram will look into the following environment
variables in the given order ‘CC’ and ‘GCC’. If they are also not
defined, BuildProgram will ultimately default to the ‘gcc’ command
which is present in many systems (sometimes as a link to other
compilers).
‘-I STR’
‘--includedir=STR’
Directory to search for files that you ‘#include’ in your C
program. Note that headers relating to Gnuastro and its
dependencies do not need this option. This is only necessary if
you want to use other headers. It may be called multiple times and
order matters. This directory will be searched before those of
Gnuastro's build and also the system search directories. See *note
Headers:: for a thorough introduction.
From the GNU C preprocessor manual: "Add the directory ‘STR’ to the
list of directories to be searched for header files. Directories
named by ‘-I’ are searched before the standard system include
directories. If the directory ‘STR’ is a standard system include
directory, the option is ignored to ensure that the default search
order for system directories and the special treatment of system
headers are not defeated".
‘-L STR’
‘--linkdir=STR’
Directory to search for compiled libraries to link the program
with. Note that all the directories that Gnuastro was built with
will already be used by BuildProgram (GNU Libtool). This option is
only necessary if your libraries are in other directories.
Multiple calls to this option are possible and order matters. This
directory will be searched before those of Gnuastro's build and
also the system search directories. See *note Linking:: for a
thorough introduction.
‘-l STR’
‘--linklib=STR’
Library to link with your program. Note that all the libraries
that Gnuastro was built with will already be linked by BuildProgram
(GNU Libtool). This option is only necessary if you want to link
with other directories. Multiple calls to this option are possible
and order matters. This library will be linked before Gnuastro's
library or its dependencies. See *note Linking:: for a thorough
introduction.
‘-O INT/STR’
‘--optimize=INT/STR’
Compiler optimization level: 0 (for no optimization, good
debugging), 1, 2, 3 (for the highest level of optimizations). From
the GNU Compiler Collection (GCC) manual: "Without any optimization
option, the compiler's goal is to reduce the cost of compilation
and to make debugging produce the expected results. Statements are
independent: if you stop the program with a break point between
statements, you can then assign a new value to any variable or
change the program counter to any other statement in the function
and get exactly the results you expect from the source code.
Turning on optimization flags makes the compiler attempt to improve
the performance and/or code size at the expense of compilation time
and possibly the ability to debug the program." Please see your
compiler's manual for the full list of acceptable values to this
option.
‘-g’
‘--debug’
Emit extra information in the compiled binary for use by a
debugger. When calling this option, it is best to explicitly
disable optimization with ‘-O0’. To combine both options you can
run ‘-gO0’ (see *note Options:: for how short options can be merged
into one).
‘-W STR’
‘--warning=STR’
Print compiler warnings on command-line during compilation.
"Warnings are diagnostic messages that report constructions that
are not inherently erroneous but that are risky or suggest there
may have been an error." (from the GCC manual). It is always
recommended to compile your programs with warnings enabled.
All compiler warning options that start with ‘W’ are usable by this
option in BuildProgram also, see your compiler's manual for the
full list. Some of the most common values to this option are:
‘pedantic’ (Warnings related to standard C) and ‘all’ (all issues
the compiler confronts).
‘-t’
‘--tag=STR’
The language configuration information. Libtool can build objects
and libraries in many languages. In many cases, it can identify
the language automatically, but when it does not you can use this
option to explicitly notify Libtool of the language. The
acceptable values are: ‘CC’ for C, ‘CXX’ for C++, ‘GCJ’ for Java,
‘F77’ for Fortran 77, ‘FC’ for Fortran, ‘GO’ for Go and ‘RC’ for
Windows Resource. Note that the Gnuastro library is not yet fully
compatible with all these languages.
‘-b’
‘--onlybuild’
Only build the program, do not run it. By default, the built
program is immediately run afterwards.
‘-d’
‘--deletecompiled’
Delete the compiled binary file after running it. This option is
only relevant when the compiled program is run after being built.
In other words, it is only relevant when ‘--onlybuild’ is not
called. It can be useful when you are busy testing a program or
just want a fast result and the actual binary/compiled file is not
of later use.
‘-a STR’
‘--la=STR’
Use the given ‘.la’ file (Libtool control file) instead of the one
that was produced from Gnuastro's configuration results. The
Libtool control file keeps all the necessary information for
building and linking a program with a library built by Libtool.
The default ‘prefix/lib/libgnuastro.la’ keeps all the information
necessary to build a program using the Gnuastro library gathered
during configure time (see *note Installation directory:: for
prefix). This option is useful when you prefer to use another
Libtool control file.
‘-e’
‘--noenv’
Do not use environment variables in the build, just use the values
given to the options. As described above, environment variables
like ‘CC’, ‘GCC’, ‘LDFLAGS’, ‘CPPFLAGS’ will be read by default and
used in the build if they have been defined.
12.3 Gnuastro library
=====================
Gnuastro library's programming constructs (function declarations,
macros, data structures, or global variables) are classified by context
into multiple header files (see *note Headers::)(1). In this section,
the functions in each header will be discussed under a separate
sub-section, which includes the name of the header. Assuming a function
declaration is in ‘headername.h’, you can include its declaration in
your source code with:
# include
The names of all constructs in ‘headername.h’ are prefixed with
‘gal_headername_’ (or ‘GAL_HEADERNAME_’ for macros). The ‘gal_’ prefix
stands for _G_NU _A_stronomy _L_ibrary.
Gnuastro library functions are compiled into a single file which can
be linked on the command-line with the ‘-lgnuastro’ option. See *note
Linking:: and *note Summary and example on libraries:: for an
introduction on linking and some fully working examples of the
libraries.
Gnuastro's library is a high-level library which depends on lower
level libraries for some operations (see *note Dependencies::).
Therefore if at least one of Gnuastro's functions in your program use
functions from the dependencies, you will also need to link those
dependencies after linking with Gnuastro. See *note BuildProgram:: for
a convenient way to deal with the dependencies. BuildProgram will take
care of the libraries to link with your program (which uses the Gnuastro
library), and can even run the built program afterwards. Therefore it
allows you to conveniently focus on your exciting science/research when
using Gnuastro's libraries.
*Libraries are still under heavy development: * Gnuastro was initially
created to be a collection of command-line programs. However, as the
programs and their the shared functions grew, internal (not installed)
libraries were added. Since the 0.2 release, the libraries are
install-able. Hence the libraries are currently under heavy development
and will significantly evolve between releases and will become more
mature and stable in due time. It will stabilize with the removal of
this notice. Check the ‘NEWS’ file for interface changes. If you use
the Info version of this manual (see *note Info::), you do not have to
worry: the documentation will correspond to your installed version.
---------- Footnotes ----------
(1) Within Gnuastro's source, all installed ‘.h’ files in
‘lib/gnuastro/’ are accompanied by a ‘.c’ file in ‘/lib/’.
12.3.1 Configuration information (‘config.h’)
---------------------------------------------
The ‘gnuastro/config.h’ header contains information about the full
Gnuastro installation on your system. Gnuastro developers should note
that this is the only header that is not available within Gnuastro, it
is only available to a Gnuastro library user _after_ installation.
Within Gnuastro, ‘config.h’ (which is included in every Gnuastro ‘.c’
file, see *note Coding conventions::) has more than enough information
about the overall Gnuastro installation.
-- Macro: GAL_CONFIG_VERSION
This macro can be used as a string literal(1) containing the
version of Gnuastro that is being used. See *note Version
numbering:: for the version formats. For example:
printf("Gnuastro version: %s\n", GAL_CONFIG_VERSION);
or
char *gnuastro_version=GAL_CONFIG_VERSION;
-- Macro: GAL_CONFIG_HAVE_GSL_INTERP_STEFFEN
GNU Scientific Library (GSL) is a mandatory dependency of Gnuastro
(see *note GNU Scientific Library::). The Steffen interpolation
function that can be used in Gnuastro was introduced in GSL version
2.0 (released in October 2015). This macro will have a value of
‘1’ if the host GSL contains this feature at configure time, and
‘0’ otherwise.
-- Macro: GAL_CONFIG_HAVE_FITS_IS_REENTRANT
This macro will have a value of 1 when the CFITSIO of the host
system has the ‘fits_is_reentrant’ function (available from CFITSIO
version 3.30). This function is used to see if CFITSIO was
configured to read a FITS file simultaneously on different threads.
-- Macro: GAL_CONFIG_HAVE_WCSLIB_VERSION
WCSLIB is the reference library for world coordinate system
transformation (see *note WCSLIB:: and *note World Coordinate
System::). However, only more recent versions of WCSLIB also
provide its version number. If the WCSLIB that is installed on the
system provides its version (through the possibly existing
‘wcslib_version’ function), this macro will have a value of one,
otherwise it will have a value of zero.
-- Macro: GAL_CONFIG_HAVE_WCSLIB_DIS_H
This macro has a value of 1 if the host's WCSLIB has the
‘wcslib/dis.h’ header for distortion-related operations.
-- Macro: GAL_CONFIG_HAVE_WCSLIB_MJDREF
This macro has a value of 1 if the host's WCSLIB reads and stores
the ‘MJDREF’ FITS header keyword as part of its core ‘wcsprm’
structure.
-- Macro: GAL_CONFIG_HAVE_WCSLIB_OBSFIX
This macro has a value of 1 if the host's WCSLIB supports the
‘OBSFIX’ feature (used by ‘wcsfix’ function to parse the input WCS
for known errors).
-- Macro: GAL_CONFIG_HAVE_PTHREAD_BARRIER
The POSIX threads standard define barriers as an optional
requirement. Therefore, some operating systems choose to not
include it. As one of the ‘./configure’ step checks, Gnuastro we
check if your system has this POSIX thread barriers. If so, this
macro will have a value of ‘1’, otherwise it will have a value of
‘0’. see *note Implementation of pthread_barrier:: for more.
-- Macro: GAL_CONFIG_SIZEOF_LONG
-- Macro: GAL_CONFIG_SIZEOF_SIZE_T
The size of (number of bytes in) the system's ‘long’ and ‘size_t’
types. Their values are commonly either 4 or 8 for 32-bit and
64-bit systems. You can also get this value with the expression
'‘sizeof size_t’' for example, without having to include this
header.
-- Macro: GAL_CONFIG_HAVE_LIBGIT2
Libgit2 is an optional dependency of Gnuastro (see *note Optional
dependencies::). When it is installed and detected at configure
time, this macro will have a value of ‘1’ (one). Otherwise, it
will have a value of ‘0’ (zero). Gnuastro also comes with some
wrappers to make it easier to use libgit2 (see *note Git
wrappers::).
-- Macro: GAL_CONFIG_HAVE_PYTHON
Gnuastro can optionally provide a set of basic functions to
facilitate wrapper libraries in Python (see *note Python
interface::). If a version of Python 3.X was found on the host
system that has the necessary Numpy headers, this macro will be
given a value of ‘1’. Otherwise, it will be given a value of ‘0’
and the the Python interface functions won't be available in the
host's Gnuastro library.
-- Macro: GAL_CONFIG_HAVE_GNUMAKE_H
Gnuastro provides a set of GNU Make extension functions (see *note
Makefile extensions::). In order to use those, the host should
have ‘gnumake.h’ in its include paths. This check is done at
Gnuastro's configuration time. If it was found, this macro is
given a value of ‘1’, otherwise, it will have a value of ‘0’.
---------- Footnotes ----------
(1)
12.3.2 Multithreaded programming (‘threads.h’)
----------------------------------------------
In recent years, newer CPUs do not have significantly higher frequencies
any more. However, CPUs are being manufactured with more cores,
enabling more than one operation (thread) at each instant. This can be
very useful to speed up many aspects of processing and in particular
image processing.
Most of the programs in Gnuastro utilize multi-threaded programming
for the CPU intensive processing steps. This can potentially lead to a
significant decrease in the running time of a program, see *note A note
on threads::. In terms of reading the code, you do not need to know
anything about multi-threaded programming. You can simply follow the
case where only one thread is to be used. In these cases, threads are
not used and can be completely ignored.
When the C language was defined (the K&R's book was written), using
threads was not common, so C's threading capabilities are not introduced
there. Gnuastro uses POSIX threads for multi-threaded programming,
defined in the ‘pthread.h’ system wide header. There are various
resources for learning to use POSIX threads. An excellent tutorial
(https://computing.llnl.gov/tutorials/pthreads/) is provided by the
Lawrence Livermore National Laboratory, with abundant figures to better
understand the concepts, it is a very good start. The book 'Advanced
programming in the Unix environment'(1), by Richard Stevens and Stephen
Rago, Addison-Wesley, 2013 (Third edition) also has two chapters
explaining the POSIX thread constructs which can be very helpful.
An alternative to POSIX threads was OpenMP, but POSIX threads are low
level, allowing much more control, while being easier to understand, see
*note Why C::. All the situations where threads are used in Gnuastro
currently are completely independent with no need of coordination
between the threads. Such problems are known as "embarrassingly
parallel" problems. They are some of the simplest problems to solve
with threads and are also the ones that benefit most from them, see the
LLNL introduction(2).
One very useful POSIX thread concept is ‘pthread_barrier’.
Unfortunately, it is only an optional feature in the POSIX standard, so
some operating systems do not include it. Therefore in *note
Implementation of pthread_barrier::, we introduce our own
implementation. This is a rather technical section only necessary for
more technical readers and you can safely ignore it. Following that, we
describe the helper functions in this header that can greatly simplify
writing a multi-threaded program, see *note Gnuastro's thread related
functions:: for more.
---------- Footnotes ----------
(1) Do not let the title scare you! The two chapters on
Multi-threaded programming are very self-sufficient and do not need any
more knowledge than K&R.
(2)
12.3.2.1 Implementation of ‘pthread_barrier’
............................................
One optional feature of the POSIX Threads standard is the
‘pthread_barrier’ concept. It is a very useful high-level construct
that allows for independent threads to "wait" behind a "barrier" for the
rest after they finish. Barriers can thus greatly simplify the code in
a multi-threaded program, so they are heavily used in Gnuastro.
However, since it is an optional feature in the POSIX standard, some
operating systems do not include it. So to make Gnuastro portable, we
have written our own implementation of those ‘pthread_barrier’
functions.
At ‘./configure’ time, Gnuastro will check if ‘pthread_barrier’
constructs are available on your system or not. If ‘pthread_barrier’ is
not available, our internal implementation will be compiled into the
Gnuastro library and the definitions and declarations below will be
usable in your code with ‘#include ’.
-- Type: pthread_barrierattr_t
Type to specify the attributes of a POSIX threads barrier.
-- Type: pthread_barrier_t
Structure defining the POSIX threads barrier.
-- Function:
int
pthread_barrier_init (pthread_barrier_t *b,
pthread_barrierattr_t *attr, unsigned int limit)
Initialize the barrier ‘b’, with the attributes ‘attr’ and total
‘limit’ (a number of) threads that must wait behind it. This
function must be called before spinning off threads.
-- Function:
int
pthread_barrier_wait (pthread_barrier_t *b)
This function is called within each thread, just before it is ready
to return. Once a thread's function hits this, it will "wait"
until all the other functions are also finished.
-- Function:
int
pthread_barrier_destroy (pthread_barrier_t *b)
Destroy all the information in the barrier structure. This should
be called by the function that spun-off the threads after all the
threads have finished.
*Destroy a barrier before re-using it:* It is very important to
destroy the barrier before (possibly) reusing it. This destroy
function not only destroys the internal structures, it also waits
(in 1 microsecond intervals, so you will not notice!) until all
the threads do not need the barrier structure any more. If you
immediately start spinning off new threads with a not-destroyed
barrier, then the internal structure of the remaining threads will
get mixed with the new ones and you will get very strange and
apparently random errors that are extremely hard to debug.
12.3.2.2 Gnuastro's thread related functions
............................................
The POSIX Threads functions offered in the C library are very low-level
and offer a great range of control over the properties of the threads.
So if you are interested in customizing your tools for complicated
thread applications, it is strongly encouraged to get a nice familiarity
with them. Some resources were introduced in *note Multithreaded
programming::.
However, in many cases used in astronomical data analysis, you do not
need communication between threads and each target operation can be done
independently. Since such operations are very common, Gnuastro provides
the tools below to facilitate the creation and management of jobs
without any particular knowledge of POSIX Threads for such operations.
The most interesting high-level functions of this section are the
‘gal_threads_number’ and ‘gal_threads_spin_off’ that identify the number
of threads on the system and spin-off threads. You can see a
demonstration of using these functions in *note Library demo -
multi-threaded operation::.
-- C struct: gal_threads_params
Structure keeping the parameters of each thread. When each thread
is created, a pointer to this structure is passed to it. The
‘params’ element can be the pointer to a structure defined by the
user which contains all the necessary parameters to pass onto the
worker function. The rest of the elements within this structure
are set internally by ‘gal_threads_spin_off’ and are relevant to
the worker function.
struct gal_threads_params
{
size_t id; /* Id of this thread. */
void *params; /* User-identified pointer. */
size_t *indexs; /* Target indices given to this thread. */
pthread_barrier_t *b; /* Barrier for all threads. */
};
-- Function:
size_t
gal_threads_number ()
Return the number of threads that the operating system has
available for your program. This number is usually fixed for a
single machine and does not change. So this function is useful
when you want to run your program on different machines (with
different CPUs).
-- Function:
void
gal_threads_spin_off (void *(*worker)(void *), void
*caller_params, size_t numactions, size_t numthreads, size_t
minmapsize, int quietmmap)
Distribute ‘numactions’ jobs between ‘numthreads’ threads and
spin-off each thread by calling the ‘worker’ function. The
‘caller_params’ pointer will also be passed to ‘worker’ as part of
the ‘gal_threads_params’ structure. For a fully working example of
this function, please see *note Library demo - multi-threaded
operation::.
If there are many jobs (millions or billions) to organize, memory
issues may become important. With ‘minmapsize’ you can specify the
minimum byte-size to allocate the necessary space in a
memory-mapped file or alternatively in RAM. If ‘quietmmap’ is
non-zero, then a warning will be printed upon creating a
memory-mapped file. For more on Gnuastro's memory management, see
*note Memory management::.
-- Function:
void
gal_threads_attr_barrier_init (pthread_attr_t *attr,
pthread_barrier_t *b, size_t limit)
This is a low-level function in case you do not want to use
‘gal_threads_spin_off’. It will initialize the general thread
attribute ‘attr’ and the barrier ‘b’ with ‘limit’ threads to wait
behind the barrier. For maximum efficiency, the threads
initialized with this function will be detached. Therefore no
communication is possible between these threads and in particular
‘pthread_join’ will not work on these threads. You have to use the
barrier constructs to wait for all threads to finish.
-- Function:
char *
gal_threads_dist_in_threads (size_t numactions, size_t
numthreads, size_t minmapsize, int quietmmap, size_t **indexs,
size_t *icols)
This is a low-level function in case you do not want to use
‘gal_threads_spin_off’. The job of this function is to distribute
‘numactions’ jobs/actions in ‘numthreads’ threads. To do this, it
will assign each job an ID, ranging from 0 to ‘numactions’-1. The
output is the allocated ‘*indexs’ array and the ‘*icols’ number.
In memory, it is just a simple 1D array that has ‘numthreads’
$\times$ ‘*icols’ elements. But you can visualize it as a 2D array
with ‘numthreads’ rows and ‘*icols’ columns. For more on the logic
of the distribution, see below.
When you have millions/billions of jobs to distribute, ‘indexs’
will become very large. For memory management (when to use a
memory-mapped file, and when to use RAM), you need to specify the
‘minmapsize’ and ‘quietmmap’ arguments. For more on memory
management, see *note Memory management::. In general, if your
distributed jobs will not be on the scale of billions (and you want
everything to always be written in RAM), just set ‘minmapsize=-1’
and ‘quietmmap=1’.
When ‘indexs’ is actually in a memory-mapped file, this function
will return a string containing the name of the file (that you can
later give to ‘gal_pointer_mmap_free’ to free/delete). When
‘indexs’ is in RAM, this function will return a ‘NULL’ pointer. So
after you are finished with ‘indexs’, you can free it like this:
char *mmapname;
int quietmmap=1;
size_t *indexs, thrdcols;
size_t numactions=5000, minmapsize=-1;
size_t numthreads=gal_threads_number();
/* Distribute the jobs. */
mmapname=gal_threads_dist_in_threads(numactions, numthreads,
minmapsize, quietmmap,
&indexs, &thrdcols);
/* Do any processing you want... */
/* Free the 'indexs' array. */
if(mmapname) gal_pointer_mmap_free(&mmapname, quietmmap);
else free(indexs);
Here is a brief description of the reasoning behind the ‘indexs’
array and how the jobs are distributed. Let's assume you have $A$
actions (where there is only one function and the input values
differ for each action) and $T$ threads available to the system
with $A>T$ (common values for these two would be $A>1000$ and
$T<10$). Spinning off a thread is not a cheap job and requires a
significant number of CPU cycles. Therefore, creating $A$ threads
is not the best way to address such a problem. The most efficient
way to manage the actions is such that only $T$ threads are
created, and each thread works on a list of actions identified for
it in series (one after the other). This way your CPU will get all
the actions done with minimal overhead.
The purpose of this function is to do what we explained above: each
row in the ‘indexs’ array contains the indices of actions which
must be done by one thread (so it has ‘numthreads’ rows with
‘*icols’ columns). However, when using ‘indexs’, you do not have
to know the number of columns. It is guaranteed that all the rows
finish with ‘GAL_BLANK_SIZE_T’ (see *note Library blank values::).
The ‘GAL_BLANK_SIZE_T’ macro plays a role very similar to a
string's ‘\0’: every row finishes with this macro, so can easily
stop parsing the indexes in the row as soon as you confront
‘GAL_BLANK_SIZE_T’. For some real examples, please see the example
program in ‘tests/lib/multithread.c’ for a demonstration.
12.3.3 Library data types (‘type.h’)
------------------------------------
Data in astronomy can have many types, numeric (numbers) and strings
(names, identifiers). The former can also be divided into integers and
floats, see *note Numeric data types:: for a thorough discussion of the
different numeric data types and which one is useful for different
contexts.
To deal with the very large diversity of types that are available
(and used in different contexts), in Gnuastro each type is identified
with global integer variable with a fixed name, this variable is then
passed onto functions that can work on any type or is stored in
Gnuastro's *note Generic data container:: as one piece of meta-data.
The actual values within these integer constants is irrelevant and
you should never rely on them. When you need to check, explicitly use
the named variable in the table below. If you want to check with more
than one type, you can use C's ‘switch’ statement.
Since Gnuastro heavily deals with file input-output, the types it
defines are fixed width types, these types are portable to all systems
and are defined in the standard C header ‘stdint.h’. You do not need to
include this header, it is included by any Gnuastro header that deals
with the different types. However, the most commonly used types in a C
(or C++) program (for example, ‘int’ or ‘long’) are not defined by their
exact width (storage size), but by their minimum storage. So for
example, on some systems, ‘int’ may be 2 bytes (16-bits, the minimum
required by the standard) and on others it may be 4 bytes (32-bits,
common in modern systems).
With every type, a unique "blank" value (or place-holder showing the
absence of data) can be defined. Please see *note Library blank
values:: for constants that Gnuastro recognizes as a blank value for
each type. See *note Numeric data types:: for more explanation on the
limits and particular aspects of each type.
-- Global integer: GAL_TYPE_INVALID
This is just a place-holder to specifically mark that no type has
been set.
-- Global integer: GAL_TYPE_BIT
Identifier for a bit-stream. Currently no program in Gnuastro
works directly on bits, but features will be added in the future.
-- Global integer: GAL_TYPE_UINT8
Identifier for an unsigned, 8-bit integer type: ‘uint8_t’ (from
‘stdint.h’), or an ‘unsigned char’ in most modern systems.
-- Global integer: GAL_TYPE_INT8
Identifier for a signed, 8-bit integer type: ‘int8_t’ (from
‘stdint.h’), or an ‘signed char’ in most modern systems.
-- Global integer: GAL_TYPE_UINT16
Identifier for an unsigned, 16-bit integer type: ‘uint16_t’ (from
‘stdint.h’), or an ‘unsigned short’ in most modern systems.
-- Global integer: GAL_TYPE_INT16
Identifier for a signed, 16-bit integer type: ‘int16_t’ (from
‘stdint.h’), or a ‘short’ in most modern systems.
-- Global integer: GAL_TYPE_UINT32
Identifier for an unsigned, 32-bit integer type: ‘uint32_t’ (from
‘stdint.h’), or an ‘unsigned int’ in most modern systems.
-- Global integer: GAL_TYPE_INT32
Identifier for a signed, 32-bit integer type: ‘int32_t’ (from
‘stdint.h’), or an ‘int’ in most modern systems.
-- Global integer: GAL_TYPE_UINT64
Identifier for an unsigned, 64-bit integer type: ‘uint64_t’ (from
‘stdint.h’), or an ‘unsigned long’ in most modern 64-bit systems.
-- Global integer: GAL_TYPE_INT64
Identifier for a signed, 64-bit integer type: ‘int64_t’ (from
‘stdint.h’), or an ‘long’ in most modern 64-bit systems.
-- Global integer: GAL_TYPE_INT
Identifier for a ‘int’ type. This is just an alias to ‘int16’, or
‘int32’ types, depending on the system.
-- Global integer: GAL_TYPE_UINT
Identifier for a ‘unsigned int’ type. This is just an alias to
‘uint16’, or ‘uint32’ types, depending on the system.
-- Global integer: GAL_TYPE_ULONG
Identifier for a ‘unsigned long’ type. This is just an alias to
‘uint32’, or ‘uint64’ types for 32-bit, or 64-bit systems
respectively.
-- Global integer: GAL_TYPE_LONG
Identifier for a ‘long’ type. This is just an alias to ‘int32’, or
‘int64’ types for 32-bit, or 64-bit systems respectively.
-- Global integer: GAL_TYPE_SIZE_T
Identifier for a ‘size_t’ type. This is just an alias to ‘uint32’,
or ‘uint64’ types for 32-bit, or 64-bit systems respectively.
-- Global integer: GAL_TYPE_FLOAT32
Identifier for a 32-bit single precision floating point type or
‘float’ in C.
-- Global integer: GAL_TYPE_FLOAT64
Identifier for a 64-bit double precision floating point type or
‘double’ in C.
-- Global integer: GAL_TYPE_COMPLEX32
Identifier for a complex number composed of two ‘float’ types.
Note that the complex type is not yet fully implemented in all
Gnuastro's programs.
-- Global integer: GAL_TYPE_COMPLEX64
Identifier for a complex number composed of two ‘double’ types.
Note that the complex type is not yet fully implemented in all
Gnuastro's programs.
-- Global integer: GAL_TYPE_STRING
Identifier for a string of characters (‘char *’).
-- Global integer: GAL_TYPE_STRLL
Identifier for a linked list of string of characters
(‘gal_list_str_t’, see *note List of strings::).
The functions below are defined to make working with the integer
constants above easier. In the functions below, the constants above can
be used for the ‘type’ input argument.
-- Function:
size_t
gal_type_sizeof (uint8_t type)
Return the number of bytes occupied by ‘type’. Internally, this
function uses C's ‘sizeof’ operator to measure the size of each
type. For strings, this function will return the size of ‘char *’.
-- Function:
char *
gal_type_name (uint8_t type, int long_name)
Return a string literal that contains the name of ‘type’. It can
return both short and long formats of the type names (for example,
‘f32’ and ‘float32’). If ‘long_name’ is non-zero, the long format
will be returned, otherwise the short name will be returned. The
output string is statically allocated, so it should not be freed.
This function is the inverse of the ‘gal_type_from_name’ function.
For the full list of names/strings that this function will return,
see *note Numeric data types::.
-- Function:
uint8_t
gal_type_from_name (char *str)
Return the Gnuastro integer constant that corresponds to the string
‘str’. This function is the inverse of the ‘gal_type_name’
function and accepts both the short and long formats of each type.
For the full list of names/strings that this function will return,
see *note Numeric data types::.
-- Function:
void
gal_type_min (uint8_t type, void *in)
Put the minimum possible value of ‘type’ in the space pointed to by
‘in’. Since the value can have any type, this function does not
return anything, it assumes the space for the given type is
available to ‘in’ and writes the value there. Here is one example
int32_t min;
gal_type_min(GAL_TYPE_INT32, &min);
Note: Do not use the minimum value for a blank value of a general
(initially unknown) type, please use the constants/functions
provided in *note Library blank values:: for the definition and
usage of blank values.
-- Function:
void
gal_type_max (uint8_t type, void *in)
Put the maximum possible value of ‘type’ in the space pointed to by
‘in’. Since the value can have any type, this function does not
return anything, it assumes the space for the given type is
available to ‘in’ and writes the value there. Here is one example
uint16_t max;
gal_type_max(GAL_TYPE_INT16, &max);
Note: Do not use the maximum value for a blank value of a general
(initially unknown) type, please use the constants/functions
provided in *note Library blank values:: for the definition and
usage of blank values.
-- Function:
int
gal_type_is_int (uint8_t type)
Return 1 if the type is an integer (any width and any sign).
-- Function:
int
gal_type_is_list (uint8_t type)
Return 1 if the type is a linked list and zero otherwise.
-- Function:
int
gal_type_out (int first_type, int second_type)
Return the larger of the two given types which can be used for the
type of the output of an operation involving the two input types.
-- Function:
char *
gal_type_bit_string (void *in, size_t size)
Return the bit-string in the ‘size’ bytes that ‘in’ points to. The
string is dynamically allocated and must be freed afterwards. You
can use it to inspect the bits within one region of memory. Here
is one short example:
int32_t a=2017;
char *bitstr=gal_type_bit_string(&a, 4);
printf("%d: %s (%X)\n", a, bitstr, a);
free(bitstr);
which will produce:
2017: 11100001000001110000000000000000 (7E1)
As the example above shows, the bit-string is not the most
efficient way to inspect bits. If you are familiar with
hexadecimal notation, it is much more compact, see
. You can use
‘printf’'s ‘%x’ or ‘%X’ to print integers in hexadecimal format.
-- Function:
char *
gal_type_to_string (void *ptr, uint8_t type, int
quote_if_str_has_space);
Read the contents of the memory that ‘ptr’ points to (assuming it
has type ‘type’ and print it into an allocated string which is
returned.
If the memory is a string of characters and
‘quote_if_str_has_space’ is non-zero, the output string will have
double-quotes around it if it contains space characters. Also,
note that in this case, ‘ptr’ must be a pointer to an array of
characters (or ‘char **’), as in the example below (which will put
‘"sample string"’ into ‘out’):
char *out, *string="sample string"
out = gal_type_to_string(&string, GAL_TYPE_STRING, 1);
-- Function:
int
gal_type_from_string (void **out, char *string, uint8_t type)
Read a string as a given data type and put a pointer to it in
‘*out’. When ‘*out!=NULL’, then it is assumed to be already
allocated and the value will be simply put there. If ‘*out==NULL’,
then space will be allocated for the given type and the string will
be read into that type.
Note that when we are dealing with a string type, ‘*out’ should be
interpreted as ‘char **’ (one element in an array of pointers to
different strings). In other words, ‘out’ should be ‘char ***’.
This function can be used to fill in arrays of numbers from strings
(in an already allocated data structure), or add nodes to a linked
list (if the type is a list type). For an array, you have to pass
the pointer to the ‘i’th element where you want the value to be
stored, for example, ‘&(array[i])’.
If the string was successfully parsed to the requested type, this
function will return a ‘0’ (zero), otherwise it will return ‘1’
(one). This output format will help you check the status of the
conversion in a code like the example below where we will try
reading a string as a single precision floating point number.
float out;
void *outptr=&out;
if( gal_type_from_string(&outptr, string, GAL_TYPE_FLOAT32) )
{
fprintf(stderr, "%s could not be read as float32\n", string);
exit(EXIT_FAILURE);
}
When you need to read many numbers into an array, ‘out’ would be an
array, and you can simply increment ‘outptr=out+i’ (where you
increment ‘i’).
-- Function:
void *
gal_type_string_to_number (char *string, uint8_t *type)
Read ‘string’ into smallest type that can host the number, the
allocated space for the number will be returned and the type of the
number will be put into the memory that ‘type’ points to. If
‘string’ could not be read as a number, this function will return
‘NULL’.
This function first calls the C library's ‘strtod’ function to read
‘string’ as a double-precision floating point number. When
successful, it will check the value to put it in the smallest
numerical data type that can handle it; for example, ‘120’ and
‘50000’ will be read as a signed 8-bit integer and unsigned 16-bit
integer types. When reading as an integer, the C library's
‘strtol’ function is used (in base-10) to parse the string again.
This re-parsing as an integer is necessary because integers with
many digits (for example, the Unix epoch seconds) will not be
accurately stored as a floating point and we cannot use the result
of ‘strtod’.
When ‘string’ is successfully parsed as a number _and_ there is ‘.’
in ‘string’, it will force the number into floating point types.
For example, ‘"5"’ is read as an integer, while ‘"5."’ or ‘"5.0"’,
or ‘"5.00"’ will be read as a floating point (single-precision).
For floating point types, this function will count the number of
significant digits and determine if the given string is single or
double precision as described in *note Numeric data types::.
For integers, negative numbers will always be placed in signed
types (as expected). If a positive integer falls below the maximum
of a signed type of a certain width, it will be signed (for
example, ‘10’ and ‘150’ will be defined as a signed and unsigned
8-bit integer respectively). In other words, even though ‘10’ can
be unsigned, it will be read as a signed 8-bit integer. This is
done to respect the C implicit type conversion in binary operators,
where signed integers will be interpreted as unsigned, when the
other operand is an unsigned integer of the same width.
For example, see the short program below. It will print ‘-50 is
larger than 100000’ (which is wrong!). This happens because when a
negative number is parsed as an unsigned, the value is effectively
subtracted from the maximum and $4294967295-50$ is indeed larger
than 100000 (recall that $4294967295$ is the largest unsigned
32-bit integer, see *note Numeric data types::).
#include
#include
#include
int
main(void)
{
int32_t a=-50;
uint32_t b=100000;
printf("%d is %s than %d\n", a,
a>b ? "larger" : "less or equal", b);
return 0;
}
However, if we read 100000 as a signed 32-bit integer, there will
not be any problem and the printed sentence will be logically
correct (for someone who does not know anything about numeric data
types: users of your programs). For the advantages of integers,
see *note Integer benefits and pitfalls::.
12.3.4 Pointers (‘pointer.h’)
-----------------------------
Pointers play an important role in the C programming language. As the
name suggests, they _point_ to a byte in memory (like an address in a
city). The C programming language gives you complete freedom in how to
use the byte (and the bytes that follow it). Pointers are thus a very
powerful feature of C. However, as the saying goes: "With great power
comes great responsibility", so they must be approached with care. The
functions in this header are not very complex, they are just wrappers
over some basic pointer functionality regarding pointer arithmetic and
allocation (in memory or HDD/SSD).
-- Function:
void *
gal_pointer_increment (void *pointer, size_t increment,
uint8_t type)
Return a pointer to an element that is ‘increment’ elements ahead
of ‘pointer’, assuming each element has type of ‘type’. For the
type codes, see *note Library data types::.
When working with the ‘array’ elements of ‘gal_data_t’, we are
actually dealing with ‘void *’ pointers. However, pointer
arithmetic does not apply to ‘void *’, because the system does not
know how many bytes there are in each element to increment the
pointer respectively. This function will use the given ‘type’ to
calculate where the incremented element is located in memory.
-- Function:
size_t
gal_pointer_num_between (void *earlier, void *later, uint8_t
type)
Return the number of elements (in the given ‘type’) between
‘earlier’ and ‘later’. For the type codes, see *note Library data
types::).
-- Function:
void *
gal_pointer_allocate (uint8_t type, size_t size, int clear,
const char *funcname, const char *varname)
Allocate an array of type ‘type’ with ‘size’ elements in RAM (for
the type codes, see *note Library data types::). If ‘clear!=0’,
then the allocated space is set to zero (cleared).
This is effectively just a wrapper around C's ‘malloc’ or ‘calloc’
functions but takes Gnuastro's integer type codes and will also
abort with a clear error if there the allocation was not
successful. The number of allocated bytes is the value given to
‘size’ that is multiplied by the returned value of
‘gal_type_sizeof’ for the given type. So if you want to allocate
space for an array of strings you should pass the type
‘GAL_TYPE_STRING’. Otherwise, if you just want space for one
string (for example, 6 bytes for ‘hello’, including the
string-termination character), you should set the type
‘GAL_TYPE_UINT8’.
When space cannot be allocated, this function will abort the
program with a message containing the reason for the failure.
‘funcname’ (name of the function calling this function) and
‘varname’ (name of variable that needs this space) will be used in
this error message if they are not ‘NULL’. In most modern
compilers, you can use the generic ‘__func__’ variable for
‘funcname’. In this way, you do not have to manually copy and
paste the function name or worry about it changing later
(‘__func__’ was standardized in C99). To use this function
effectively and avoid memory leaks, make sure to free the allocated
array after you are done with it. Also, be mindful of any
functions that make use of this function as they should also free
any allocated arrays to maintain memory management and prevent
issues with the system.
-- Function:
void *
gal_pointer_allocate_ram_or_mmap (uint8_t type, size_t size,
int clear, size_t minmapsize, char **mmapname, int quietmmap,
const char *funcname, const char *varname)
Allocate the given space either in RAM or in a memory-mapped file.
This function is just a high-level wrapper to
‘gal_pointer_allocate’ (to allocate in RAM) or
‘gal_pointer_mmap_allocate’ (to use a memory-mapped file). For
more on memory management in Gnuastro, please see *note Memory
management::. The various arguments are more fully explained in
the two functions above.
-- Function:
void *
gal_pointer_mmap_allocate (size_t size, uint8_t type, int
clear, char **mmapname, int allocfailed)
Allocate the necessary space to keep ‘size’ elements of type ‘type’
in HDD/SSD (a file, not in RAM). For the type codes, see *note
Library data types::. If ‘clear!=0’, then the allocated space will
also be cleared. The allocation is done using C's ‘mmap’ function.
The name of the file containing the allocated space is an allocated
string that will be put in ‘*mmapname’.
Note that the kernel does not allow an infinite number of memory
mappings to files. So it is not recommended to use this function
with every allocation. The best-case scenario to use this function
is for arrays that are very large and can fill up the RAM. Keep the
smaller arrays in RAM, which is faster and can have a
(theoretically) unlimited number of allocations.
When you are done with the dataset and do not need it anymore, do
not use ‘free’ (the dataset is not in RAM). Just delete the file
(and the allocated space for the filename) with the commands below,
or simply use ‘gal_pointer_mmap_free’.
remove(mmapname);
free(mmapname);
If ‘allocfailed!=0’ and the memory mapping attempt fails, the
warning message will say something like this (assuming you have
tried something like ‘malloc’ before calling this function): even
though there was enough space in RAM, the previous attempts at
allocation in RAM failed, so we tried memory mapping, but that also
failed.
-- Function:
void
gal_pointer_mmap_free (char **mmapname, int quietmmap)
"Free" (actually delete) the memory-mapped file that is named
‘*mmapname’, then free the string. If ‘quietmmap’ is non-zero,
then a warning will be printed for the user to know that the given
file has been deleted.
12.3.5 Library blank values (‘blank.h’)
---------------------------------------
When the position of an element in a dataset is important (for example,
a pixel in an image), a place-holder is necessary for the element if we
do not have a value to fill it with (for example, the CCD cannot read
those pixels). We cannot simply shift all the other pixels to fill in
the one we have no value for. In other cases, it often occurs that the
field of sky that you are studying is not a clean rectangle to nicely
fit into the boundaries of an image. You need a way to separate the
pixels outside your scientific field from those inside it. Blank values
act as these place holders in a dataset. They have no usable value but
they have a position.
Every type needs a corresponding blank value (see *note Numeric data
types:: and *note Library data types::). Floating point types have a
unique value identified by IEEE known as Not-a-Number (or NaN) which is
a unique value that is recognized by the compiler. However, integer and
string types do not have any standard value. For integers, in Gnuastro
we take an extremum of the given type: for signed types (that allow
negatives), the minimum possible value is used as blank and for unsigned
types (that only accept positives), the maximum possible value is used.
To be generic and easy to read/write we define a macro for these blank
values and strongly encourage you only use these, and never make any
assumption on the value of a type's blank value.
The IEEE NaN blank value type is defined to fail on any comparison,
so if you are dealing with floating point types, you cannot use equality
(a NaN will _not_ be equal to a NaN). If you know your dataset is
floating point, you can use the ‘isnan’ function in C's ‘math.h’ header.
For a description of numeric data types see *note Numeric data types::.
For the constants identifying integers, please see *note Library data
types::.
-- Global integer: GAL_BLANK_UINT8
Blank value for an unsigned, 8-bit integer.
-- Global integer: GAL_BLANK_INT8
Blank value for a signed, 8-bit integer.
-- Global integer: GAL_BLANK_UINT16
Blank value for an unsigned, 16-bit integer.
-- Global integer: GAL_BLANK_INT16
Blank value for a signed, 16-bit integer.
-- Global integer: GAL_BLANK_UINT32
Blank value for an unsigned, 32-bit integer.
-- Global integer: GAL_BLANK_INT32
Blank value for a signed, 32-bit integer.
-- Global integer: GAL_BLANK_UINT64
Blank value for an unsigned, 64-bit integer.
-- Global integer: GAL_BLANK_INT64
Blank value for a signed, 64-bit integer.
-- Global integer: GAL_BLANK_INT
Blank value for ‘int’ type (‘int16_t’ or ‘int32_t’ depending on the
system.
-- Global integer: GAL_BLANK_UINT
Blank value for ‘int’ type (‘int16_t’ or ‘int32_t’ depending on the
system.
-- Global integer: GAL_BLANK_LONG
Blank value for ‘long’ type (‘int32_t’ or ‘int64_t’ in 32-bit or
64-bit systems).
-- Global integer: GAL_BLANK_ULONG
Blank value for ‘unsigned long’ type (‘uint32_t’ or ‘uint64_t’ in
32-bit or 64-bit systems).
-- Global integer: GAL_BLANK_SIZE_T
Blank value for ‘size_t’ type (‘uint32_t’ or ‘uint64_t’ in 32-bit
or 64-bit systems).
-- Global integer: GAL_BLANK_FLOAT32
Blank value for a single precision, 32-bit floating point type
(IEEE NaN value).
-- Global integer: GAL_BLANK_FLOAT64
Blank value for a double precision, 64-bit floating point type
(IEEE NaN value).
-- Global integer: GAL_BLANK_STRING
Blank value for string types (this is itself a string, it is not
the ‘NULL’ pointer).
The functions below can be used to work with blank pixels.
-- Function:
void
gal_blank_write (void *pointer, uint8_t type)
Write the blank value for the given ‘type’ into the space that
‘pointer’ points to. This can be used when the space is already
allocated (for example, one element in an array or a statically
allocated variable).
-- Function:
void *
gal_blank_alloc_write (uint8_t type)
Allocate the space required to keep the blank for the given data
type ‘type’, write the blank value into it and return the pointer
to it.
-- Function:
void
gal_blank_initialize (gal_data_t *input)
Initialize all the elements in the ‘input’ dataset to the blank
value that corresponds to its type. If ‘input’ is not a string,
and is a tile over a larger dataset, only the region that the tile
covers will be set to blank. For strings, the full dataset will be
initialized.
-- Function:
void
gal_blank_initialize_array (void *array, size_t size, uint8_t
type)
Initialize all the elements in the ‘array’ to the blank value that
corresponds to its type (identified with ‘type’), assuming the
array has ‘size’ elements.
-- Function:
char *
gal_blank_as_string (uint8_t type, int width)
Write the blank value for the given data type ‘type’ into a string
and return it. The space for the string is dynamically allocated
so it must be freed after you are done with it. If ‘width!=0’,
then the final string will be padded with white space characters to
have the requested width if it is smaller.
-- Function:
int
gal_blank_is (void *pointer, uint8_t type)
Return 1 if the contents of ‘pointer’ (assuming a type of ‘type’)
is blank. Otherwise, return 0. Note that this function only works
on one element of the given type. So if ‘pointer’ is an array,
only its first element will be checked. Therefore for strings, the
type of ‘pointer’ is assumed to be ‘char *’. To check if an
array/dataset has blank elements or to find which elements in an
array are blank, you can use ‘gal_blank_present’ or
‘gal_blank_flag’ respectively (described below).
-- Function:
int
gal_blank_present (gal_data_t *input, int updateflag)
Return 1 if the dataset has a blank value and zero if it does not.
Before checking the dataset, this function will look at ‘input’'s
flags. If the ‘GAL_DATA_FLAG_BLANK_CH’ bit of ‘input->flag’ is on,
this function will not do any check and will just use the
information in the flags. This can greatly speed up processing
when a dataset needs to be checked multiple times.
When the dataset's flags were not used and ‘updateflags’ is
non-zero, this function will set the flags appropriately to avoid
having to re-check the dataset in future calls. When
‘updateflags==0’, this function has no side-effects on the dataset:
it will not toggle the flags.
If you want to re-check a dataset with the blank-value-check flag
already set (for example, if you have made changes to it), then
explicitly set the ‘GAL_DATA_FLAG_BLANK_CH’ bit to zero before
calling this function. When there are no other flags, you can just
set the flags to zero (‘input->flag=0’), otherwise you can use this
expression:
input->flag &= ~GAL_DATA_FLAG_BLANK_CH;
-- Function:
size_t
gal_blank_number (gal_data_t *input, int updateflag)
Return the number of blank elements in ‘input’. If
‘updateflag!=0’, then the dataset blank keyword flags will be
updated. See the description of ‘gal_blank_present’ (above) for
more on these flags. If ‘input==NULL’, then this function will
return ‘GAL_BLANK_SIZE_T’.
-- Function:
gal_data_t *
gal_blank_flag (gal_data_t *input)
Return a "flag" dataset with the same size as the input, but with
an ‘uint8_t’ type that has a value of 1 for data elements that are
blank and 0 for those that are not.
-- Function:
gal_data_t *
gal_blank_flag_not (gal_data_t *input)
Return a "flag" dataset with the same size as the input, but with
an ‘uint8_t’ type that has a value of 1 for data elements that are
_not_ blank and 0 for those that are blank.
-- Function:
size_t *
gal_blank_not_minmax_coords (gal_data_t *input)
Find the minimum and maximum coordinates of the non-blank regions
within the input dataset. The coordinates are in C order: starting
from 0, and with the slowest dimension being first. The output is
an allocated array (that should be freed later) with $2\times N$
elements; where $N$ is the number of dimensions. The first two
elements contain the minimum and maximum of regions containing
non-blank elements along the 0-th dimension (the slowest), the
second two elements contain the next dimension's extrema; and so
on.
-- Function:
gal_data_t *
gal_blank_trim (gal_data_t *input, int inplace)
Trim all the outer layers of blank values from the input dataset.
For example in the 2D image, "layers" would correspond to columns
or rows that are fully blank and touching the edge of the image.
For a more complete description, see the description of the ‘trim’
operator in *note Dimensionality changing operators::.
-- Function:
void
gal_blank_flag_apply (gal_data_t *input, gal_data_t *flag)
Set all non-zero and non-blank elements of ‘flag’ to blank in
‘input’. ‘flag’ has to have an unsigned 8-bit type and be the same
size as ‘input’.
-- Function:
void
gal_blank_flag_remove (gal_data_t *input, gal_data_t *flag)
Remove all elements within ‘input’ that are flagged, convert it to
a 1D dataset and adjust the size properly (the number of
non-flagged elements). In practice this function does not‘realloc’
the input array (see ‘gal_blank_remove_realloc’ for
shrinking/re-allocating also), it just shifts the blank elements to
the end and adjusts the size elements of the ‘gal_data_t’, see
*note Generic data container::.
Note that elements that are blank, but not flagged will not be
removed. This function will only remove flagged elements.
If all the elements were flagged, then ‘input->size’ will be zero.
This is thus a good parameter to check after calling this function
to see if there actually were any non-flagged elements in the input
or not and take the appropriate measure. This check is highly
recommended because it will avoid strange bugs in later steps.
-- Function:
void
gal_blank_remove (gal_data_t *input)
Remove blank elements from a dataset, convert it to a 1D dataset,
adjust the size properly (the number of non-blank elements), and
toggle the blank-value-related bit-flags. In practice this
function does not‘realloc’ the input array (see
‘gal_blank_remove_realloc’ for shrinking/re-allocating also), it
just shifts the blank elements to the end and adjusts the size
elements of the ‘gal_data_t’, see *note Generic data container::.
If all the elements were blank, then ‘input->size’ will be zero.
This is thus a good parameter to check after calling this function
to see if there actually were any non-blank elements in the input
or not and take the appropriate measure. This check is highly
recommended because it will avoid strange bugs in later steps.
-- Function:
void
gal_blank_remove_realloc (gal_data_t *input)
Similar to ‘gal_blank_remove’, but also shrinks/re-allocates the
dataset's allocated memory.
-- Function:
gal_data_t *
gal_blank_remove_rows (gal_data_t *columns, gal_list_sizet_t
*column_indexs, int onlydim0)
Remove (in place) any row that has at least one blank value in any
of the input columns and return a "flag" dataset (that should be
freed later). The input ‘columns’ is a list of ‘gal_data_t’s (see
*note List of gal_data_t::). When ‘onlydim0!=0’ the vector columns
(with 2 dimensions) will not be checked for the presence of blank
values.
After this function, all the elements in ‘columns’ will still have
the same size as each other, but if any of the searched columns has
blank elements, all their sizes will decrease together.
The returned flag dataset has the same size as the original input
dataset, with a type of ‘uint8_t’. Every row that has been removed
from the original dataset has a value of 1, and the rest have a
value of 0.
When ‘column_indexs!=NULL’, only the columns whose index (counting
from zero) is in ‘column_indexs’ will be used to check for blank
values (see *note List of size_t::. Therefore, if you want to
check all columns, just set this to ‘NULL’. In any case (no matter
which columns are checked for blanks), the selected rows from all
columns will be removed.
12.3.6 Data container (‘data.h’)
--------------------------------
Astronomical datasets have various dimensions, for example, 1D spectra
or table columns, 2D images, or 3D Integral field data cubes. Datasets
can also have various numeric data types, depending on the
operation/purpose, for example, processed images are commonly stored in
floating point format, but their mask images are integers (allowing
bitwise flags to identify certain classes of pixels to keep or mask, see
*note Numeric data types::). Certain other information about a dataset
are also commonly necessary, for example, the units of the dataset, the
name of the dataset and some comments. To deal with any generic
dataset, Gnuastro defines the ‘gal_data_t’ as input or output.
12.3.6.1 Generic data container (‘gal_data_t’)
..............................................
To be able to deal with any dataset (various dimensions, numeric data
types, units and higher-level structures), Gnuastro defines the
‘gal_data_t’ type which is the input/output container of choice for many
of Gnuastro library's functions. It is defined in ‘gnuastro/data.h’.
If you will be using ('‘# include’'ing) those libraries, you do not need
to include this header explicitly, it is already included by any library
header that uses ‘gal_data_t’.
-- Type (C struct): gal_data_t
The main container for datasets in Gnuastro. It can host data of
any dimensions, with any numeric data type. It is actually a
structure, but ‘typedef’'d as a new type to avoid having to write
the ‘struct’ before any declaration. The actual structure is shown
below which is followed by a description of each element.
typedef struct gal_data_t
{
void *restrict array; /* Basic array information. */
uint8_t type;
size_t ndim;
size_t *dsize;
size_t size;
int quietmmap;
char *mmapname;
size_t minmapsize;
int nwcs; /* WCS information. */
struct wcsprm *wcs;
uint8_t flag; /* Content description. */
int status;
char *name;
char *unit;
char *comment;
int disp_fmt; /* For text printing. */
int disp_width;
int disp_precision;
struct gal_data_t *next; /* For higher-level datasets. */
struct gal_data_t *block;
} gal_data_t;
The list below contains a description for each ‘gal_data_t’ element.
‘void *restrict array’
This is the pointer to the main array of the dataset containing the
raw data (values). All the other elements in this data-structure
are actually meta-data enabling us to use/understand the series of
values in this array. It must allow data of any type (see *note
Numeric data types::), so it is defined as a ‘void *’ pointer. A
‘void *’ array is not directly usable in C, so you have to cast it
to proper type before using it, please see *note Library demo -
reading a image:: for a demonstration.
The ‘restrict’ keyword was formally introduced in C99 and is used
to tell the compiler that at any moment only this pointer will
modify what it points to (a pixel in an image for example)(1).
This extra piece of information can greatly help in compiler
optimizations and thus the running time of the program. But older
compilers might not have this capability, so at ‘./configure’ time,
Gnuastro checks this feature and if the user's compiler does not
support ‘restrict’, it will be removed from this definition.
‘uint8_t type’
A fixed code (integer) used to identify the type of data in ‘array’
(see *note Numeric data types::). For the list of acceptable
values to this variable, please see *note Library data types::.
‘size_t ndim’
The dataset's number of dimensions.
‘size_t *dsize’
The size of the dataset along each dimension. This is an array
(with ‘ndim’ elements), of positive integers in row-major order(2)
(based on C). When a data file is read into memory with Gnuastro's
libraries, this array is dynamically allocated based on the number
of dimensions that the dataset has.
It is important to remember that C's row-major ordering is the
opposite of the FITS standard which is in column-major order: in
the FITS standard the fastest dimension's size is specified by
‘NAXIS1’, and slower dimensions follow. The FITS standard was
defined mainly based on the FORTRAN language which is the opposite
of C's approach to multi-dimensional arrays (and also starts
counting from 1 not 0). Hence if a FITS image has ‘NAXIS1==20’ and
‘NAXIS2==50’, the ‘dsize’ array must be filled with ‘dsize[0]==50’
and ‘dsize[1]==20’.
The fastest dimension is the one that is contiguous in memory: to
increment by one along that dimension, just go to the next element
in the array. As we go to slower dimensions, the number of memory
cells we have to skip for an increment along that dimension becomes
larger.
‘size_t size’
The total number of elements in the dataset. This is actually a
multiplication of all the values in the ‘dsize’ array, so it is not
an independent parameter. However, low-level operations with the
dataset (irrespective of its dimensions) commonly need this number,
so this element is designed to avoid calculating it every time.
‘int quietmmap’
When this value is zero, and the dataset must not be allocated in
RAM (see ‘mmapname’ and ‘minmapsize’ below), a warning will be
printed to inform the user when the file is created and when it is
deleted. The warning includes the filename, the size in bytes, and
the fact that they can toggle this behavior through ‘--minmapsize’
option in Gnuastro's programs.
‘char *mmapname’
Name of file hosting the ‘mmap’'d contents of ‘array’. If the
value of this variable is ‘NULL’, then the contents of ‘array’ are
actually stored in RAM, not in a file on the HDD/SSD. See the
description of ‘minmapsize’ below for more.
If a file is used, it will be kept in the ‘gnuastro_mmap’ directory
of the running directory. Its name is randomly selected to allow
multiple arrays at the same time, see description of ‘--minmapsize’
in *note Processing options::. When ‘gal_data_free’ is called the
randomly named file will be deleted.
‘size_t minmapsize’
The minimum size of an array (in bytes) to store the contents of
‘array’ as a file (on the non-volatile HDD/SSD), not in RAM. This
can be very useful for large datasets which can be very memory
intensive and the user's RAM might not be sufficient to
keep/process it. A random filename is assigned to the array which
is available in the ‘mmapname’ element of ‘gal_data_t’ (above), see
there for more. ‘minmapsize’ is stored in each ‘gal_data_t’, so it
can be passed on to subsequent/derived datasets.
See the description of the ‘--minmapsize’ option in *note
Processing options:: for more on using this value.
‘nwcs’
The number of WCS coordinate representations (for WCSLIB).
‘struct wcsprm *wcs’
The main WCSLIB structure keeping all the relevant information
necessary for WCSLIB to do its processing and convert data-set
positions into real-world positions. When it is given a ‘NULL’
value, all possible WCS calculations/measurements will be ignored.
‘uint8_t flag’
Bitwise flags to describe general properties of the dataset. The
number of bytes available in this flag is stored in the
‘GAL_DATA_FLAG_SIZE’ macro. Note that you should use bitwise
operators(3) to check these flags. The currently recognized bits
are stored in these macros:
‘GAL_DATA_FLAG_BLANK_CH’
Marking that the dataset has been checked for blank values or
not. When a dataset does not have any blank values, the
‘GAL_DATA_FLAG_HASBLANK’ bit will be zero. But upon
initialization, all bits also get a value of zero. Therefore,
a checker needs this flag to see if the value in
‘GAL_DATA_FLAG_HASBLANK’ is reliable (dataset has actually
been parsed for a blank value) or not.
Also, if it is necessary to re-check the presence of flags,
you just have to set this flag to zero and call
‘gal_blank_present’ for example, to parse the dataset and
check for blank values. Note that for improved efficiency,
when this flag is set, ‘gal_blank_present’ will not actually
parse the dataset, it will just use ‘GAL_DATA_FLAG_HASBLANK’.
‘GAL_DATA_FLAG_HASBLANK’
This bit has a value of ‘1’ when the given dataset has blank
values. If this bit is ‘0’ and ‘GAL_DATA_FLAG_BLANK_CH’ is
‘1’, then the dataset has been checked and it did not have any
blank values, so there is no more need for further checks.
‘GAL_DATA_FLAG_SORT_CH’
Marking that the dataset is already checked for being sorted
or not and thus that the possible ‘0’ values in
‘GAL_DATA_FLAG_SORTED_I’ and ‘GAL_DATA_FLAG_SORTED_D’ are
meaningful. The logic behind this is similar to that in
‘GAL_DATA_FLAG_BLANK_CH’.
‘GAL_DATA_FLAG_SORTED_I’
This bit has a value of ‘1’ when the given dataset is sorted
in an increasing manner. If this bit is ‘0’ and
‘GAL_DATA_FLAG_SORT_CH’ is ‘1’, then the dataset has been
checked and was not sorted (increasing), so there is no more
need for further checks.
‘GAL_DATA_FLAG_SORTED_D’
This bit has a value of ‘1’ when the given dataset is sorted
in a decreasing manner. If this bit is ‘0’ and
‘GAL_DATA_FLAG_SORT_CH’ is ‘1’, then the dataset has been
checked and was not sorted (decreasing), so there is no more
need for further checks.
The macro ‘GAL_DATA_FLAG_MAXFLAG’ contains the largest internally
used bit-position. Higher-level flags can be defined with the
bitwise shift operators using this macro to define internal flags
for libraries/programs that depend on Gnuastro without causing any
possible conflict with the internal flags discussed above or having
to check the values manually on every release.
‘int status’
A context-specific status values for this data-structure. This
integer will not be set by Gnuastro's libraries. You can use it
keep some additional information about the dataset (with integer
constants) depending on your applications.
‘char *name’
The name of the dataset. If the dataset is a multi-dimensional
array and read/written as a FITS image, this will be the value in
the ‘EXTNAME’ FITS keyword. If the dataset is a one-dimensional
table column, this will be the column name. If it is set to ‘NULL’
(by default), it will be ignored.
‘char *unit’
The units of the dataset (for example, ‘BUNIT’ in the standard FITS
keywords) that will be read from or written to files/tables along
with the dataset. If it is set to ‘NULL’ (by default), it will be
ignored.
‘char *comment’
Any further explanation about the dataset which will be written to
any output file if present.
‘disp_fmt’
Format to use for printing each element of the dataset to a plain
text file, the acceptable values to this element are defined in
*note Table input output::. Based on C's ‘printf’ standards.
‘disp_width’
Width of printing each element of the dataset to a plain text file,
the acceptable values to this element are defined in *note Table
input output::. Based on C's ‘printf’ standards.
‘disp_precision’
Precision of printing each element of the dataset to a plain text
file, the acceptable values to this element are defined in *note
Table input output::. Based on C's ‘printf’ standards.
‘gal_data_t *next’
Through this pointer, you can link a ‘gal_data_t’ with other
datasets related datasets, for example, the different columns in a
dataset each have one ‘gal_data_t’ associate with them and they are
linked to each other using this element. There are several
functions described below to facilitate using ‘gal_data_t’ as a
linked list. See *note Linked lists:: for more on these wonderful
high-level constructs.
‘gal_data_t *block’
Pointer to the start of the complete allocated block of memory.
When this pointer is not ‘NULL’, the dataset is not treated as a
contiguous patch of memory. Rather, it is seen as covering only a
portion of the larger patch of memory that ‘block’ points to. See
*note Tessellation library:: for a more thorough explanation and
functions to help work with tiles that are created from this
pointer.
---------- Footnotes ----------
(1) Also see .
(2) Also see
.
(3) See .
12.3.6.2 Dataset allocation
...........................
Gnuastro's main data container was defined in *note Generic data
container::. The functions listed in this section describe the most
basic operations on ‘gal_data_t’: those related to allocation and
freeing. These functions are declared in ‘gnuastro/data.h’ which is
also visible from the function names (see *note Gnuastro library::).
-- Function:
gal_data_t *
gal_data_alloc (void *array, uint8_t type, size_t ndim, size_t
*dsize, struct wcsprm *wcs, int clear, size_t minmapsize, int
quietmmap, char *name, char *unit, char *comment)
Dynamically allocate a ‘gal_data_t’ and initialize it will all the
given values. See the description of ‘gal_data_initialize’ and
*note Generic data container:: for more information. This function
will often be the most frequently used because it allocates the
‘gal_data_t’ hosting all the values _and_ initializes it. Once you
are done with the dataset, be sure to clean up all the allocated
spaces with ‘gal_data_free’.
-- Function:
void
gal_data_initialize (gal_data_t *data, void *array, uint8_t
type, size_t ndim, size_t *dsize, struct wcsprm *wcs, int
clear, size_t minmapsize, int quietmmap, char *name, char
*unit, char *comment)
Initialize the given data structure (‘data’) with all the given
values. Note that the raw input ‘gal_data_t’ must already have
been allocated before calling this function. For a description of
each variable see *note Generic data container::. It will set the
values and do the necessary allocations. If they are not ‘NULL’,
all input arrays (‘dsize’, ‘wcs’, ‘name’, ‘unit’, ‘comment’) are
separately copied (allocated) by this function for usage in ‘data’,
so you can safely use one value to initialize many datasets or use
statically allocated variables in this function call. Once you are
done with the dataset, you can free all the allocated spaces with
‘gal_data_free_contents’.
If ‘array’ is not ‘NULL’, it will be directly copied into
‘data->array’ (based on the total number of elements calculated
from ‘dsize’) and no new space will be allocated for the array of
this dataset, this has many low-level advantages and can be used to
work on regions of a dataset instead of the whole allocated array
(see the description under ‘block’ in *note Generic data
container:: for one example). If the given pointer is not the
start of an allocated block of memory or it is used in multiple
datasets, be sure to set it to ‘NULL’ (with ‘data->array=NULL’)
before cleaning up with ‘gal_data_free_contents’.
‘ndim’ may be zero. In this case no allocation will occur,
‘data->array’ and ‘data->dsize’ will be set to ‘NULL’ and
‘data->size’ will be zero. However (when necessary) ‘dsize’ must
not have any zero values (a dimension of length zero is not
defined).
-- Function:
gal_data_t *
gal_data_alloc_empty (size_t ndim, size_t minmapsize, int
quietmmap)
Allocate an empty dataset with a certain number of dimensions, but
no 'array' component. The ‘size’ element will be set to zero and
the ‘dsize’ array will be properly allocated (based on the number
of dimensions), but all elements will be zero. This is useful in
scenarios where you just need a ‘gal_data_t’ for metadata.
-- Function:
void
gal_data_free_contents (gal_data_t *data)
Free all the non-‘NULL’ pointers in ‘gal_data_t’ except for ‘next’
and ‘block’. All freed arrays are set to ‘NULL’. If ‘data’ is
actually a tile (‘data->block!=NULL’, see *note Tessellation
library::), then ‘data->array’ is not freed. For a complete
description of ‘gal_data_t’ and its contents, see *note Generic
data container::.
-- Function:
void
gal_data_free (gal_data_t *data)
Free all the non-‘NULL’ pointers in ‘gal_data_t’, then free the
actual data structure.
12.3.6.3 Arrays of datasets
...........................
Gnuastro's generic data container (‘gal_data_t’) is a very versatile
structure that can be used in many higher-level contexts. One such
higher-level construct is an array of ‘gal_data_t’ structures to
simplify the allocation (and later cleaning) of several ‘gal_data_t’s
that are related.
For example, each column in a table is usually represented by one
‘gal_data_t’ (so it has its own name, data type, units, etc.). A table
(with many columns) can be seen as an array of ‘gal_data_t’s (when the
number of columns is known a-priori). The functions below are defined
to create a cleared array of data structures and to free them when none
are necessary any more. These functions are declared in
‘gnuastro/data.h’ which is also visible from the function names (see
*note Gnuastro library::).
-- Function:
gal_data_t *
gal_data_array_calloc (size_t size)
Allocate an array of ‘gal_data_t’ with ‘size’ elements. This
function will also initialize all the values (‘NULL’ for pointers
and 0 for other types). You can use ‘gal_data_initialize’ to fill
each element of the array afterwards. The following code snippet
is one example of doing this.
size_t i;
gal_data_t *dataarr;
dataarr=gal_data_array_calloc(10);
for(i=0;i<10;++i) gal_data_initialize(&dataarr[i], ...);
...
gal_data_array_free(dataarr, 10, 1);
-- Function:
void
gal_data_array_free (gal_data_t *dataarr, size_t num, int
free_array)
Free all the ‘num’ elements within ‘dataarr’ and the actual
allocated array. If ‘free_array’ is not zero, then the ‘array’
element of all the datasets will also be freed, see *note Generic
data container::.
-- Function:
gal_data_t **
gal_data_array_ptr_calloc (size_t size)
Allocate an array of pointers to Gnuastro's generic data structure
and initialize all pointers to ‘NULL’. This is useful when you
want to allocate individual datasets later (for example, with
‘gal_data_alloc’).
-- Function:
void
gal_data_array_ptr_free (gal_data_t **dataptr, size_t size,
int free_array);
Free all the individual datasets within the elements of ‘dataptr’,
then free ‘dataptr’ itself (the array of pointers that was probably
allocated with ‘gal_data_array_ptr_calloc’.
12.3.6.4 Copying datasets
.........................
The functions in this section describes Gnuastro's facilities to copy a
given dataset into another. The new dataset can have a different type
(including a string), it can be already allocated (in which case only
the values will be written into it). In all these cases, if the input
dataset is a tile or a list, only the data within the given tile, or the
given node in a list, are copied. If the input is a list, the ‘next’
pointer will also be copied to the output, see *note List of
gal_data_t::.
In many of the functions here, it is possible to copy the dataset to
a new numeric data type (see *note Numeric data types::. In such cases,
Gnuastro's library is going to use the native conversion by C. So if you
are converting to a smaller type, it is up to you to make sure that the
values fit into the output type.
-- Function:
gal_data_t *
gal_data_copy (gal_data_t *in)
Return a new dataset that is a copy of ‘in’, all of ‘in’'s
meta-data will also copied into the output, except for ‘block’. If
the dataset is a tile/list, only the given tile/node will be
copied, the ‘next’ pointer will also be copied however.
-- Function:
gal_data_t *
gal_data_copy_to_new_type (gal_data_t *in, uint8_t newtype)
Return a copy of the dataset ‘in’, converted to ‘newtype’, see
*note Library data types:: for Gnuastro library's type identifiers.
The returned dataset will have all meta-data except their type and
‘block’ equal to the input's metadata. If the dataset is a
tile/list, only the given tile/node will be copied, the ‘next’
pointer will also be copied however.
-- Function:
gal_data_t *
gal_data_copy_to_new_type_free (gal_data_t *in, uint8_t
newtype)
Return a copy of the dataset ‘in’ that is converted to ‘newtype’
and free the input dataset. See *note Library data types:: for
Gnuastro library's type identifiers. The returned dataset will
have all meta-data, except their type, equal to the input's
metadata (including ‘next’). Note that if the input is a tile
within a larger block, it will not be freed. This function is
similar to ‘gal_data_copy_to_new_type’, except that it will free
the input dataset.
-- Function:
void
gal_data_copy_to_allocated (gal_data_t *in, gal_data_t *out)
Copy the contents of the array in ‘in’ into the already allocated
array in ‘out’. The types of the input and output may be
different, type conversion will be done internally. When ‘in->size
!= out->size’ this function will behave as follows:
‘out->size < in->size’
This function will not re-allocate the necessary space, it
will abort with an error, so please check before calling this
function.
‘out->size > in->size’
This function will write the values in ‘out->size’ and
‘out->dsize’ from the same values of ‘in’. So if you want to
use a pre-allocated space/dataset multiple times with varying
input sizes, be sure to reset ‘out->size’ before every call to
this function.
-- Function:
gal_data_t *
gal_data_copy_string_to_number (char *string)
Read ‘string’ into the smallest type that can store the value (see
*note Numeric data types::). This function is just a wrapper for
the ‘gal_type_string_to_number’, but will put the value into a
single-element dataset.
12.3.7 Dimensions (‘dimension.h’)
---------------------------------
An array is a contiguous region of memory. Hence, at the lowest level,
every element of an array just has one single-valued position: the
number of elements that lie between it and the first element in the
array. This is also known as the _index_ of the element within the
array. A dataset's number of dimensions is high-level abstraction
(meta-data) that we project onto that contiguous patch of memory. When
the array is interpreted as a one-dimensional dataset, this index is
also the _coordinate_ of the element. But once we associate the patch
of memory with a higher dimension, there must also be one coordinate for
each dimension.
The functions and macros in this section provide you with the tools
to convert an index into a coordinate and vice-versa along with several
other issues for example, issues with the neighbors of an element in a
multi-dimensional context.
-- Function:
size_t
gal_dimension_total_size (size_t ndim, size_t *dsize)
Return the total number of elements for a dataset with ‘ndim’
dimensions that has ‘dsize’ elements along each dimension.
-- Function:
int
gal_dimension_is_different (gal_data_t *first, gal_data_t
*second)
Return ‘1’ (one) if the two datasets do not have the same size
along all dimensions. This function will also return ‘1’ when the
number of dimensions of the two datasets are different.
-- Function:
size_t *
gal_dimension_increment (size_t ndim, size_t *dsize)
Return an allocated array that has the number of elements necessary
to increment an index along every dimension. For example, along
the fastest dimension (last element in the ‘dsize’ and returned
arrays), the value is ‘1’ (one).
-- Function:
size_t
gal_dimension_num_neighbors (size_t ndim)
The maximum number of neighbors (any connectivity) that a data
element can have in ‘ndim’ dimensions. Effectively, this function
just returns $3^n-1$ (where $n$ is the number of dimensions).
-- Function-like macro: GAL_DIMENSION_FLT_TO_INT (FLT)
Calculate the integer pixel position that the floating point ‘FLT’
number belongs to. In the FITS format (and thus in Gnuastro), the
center of each pixel is allocated on an integer (not it edge), so
the pixel which hosts a floating point number cannot simply be
found with internal type conversion.
-- Function:
void
gal_dimension_add_coords (size_t *c1, size_t *c2, size_t *out,
size_t ndim)
For every dimension, add the coordinates in ‘c1’ with ‘c2’ and put
the result into ‘out’. In other words, for dimension ‘i’ run
‘out[i]=c1[i]+c2[i];’. Hence ‘out’ may be equal to any one of ‘c1’
or ‘c2’.
-- Function:
size_t
gal_dimension_coord_to_index (size_t ndim, size_t *dsize,
size_t *coord)
Return the index (counting from zero) from the coordinates in
‘coord’ (counting from zero) assuming the dataset has ‘ndim’
elements and the size of the dataset along each dimension is in the
‘dsize’ array.
-- Function:
void
gal_dimension_index_to_coord (size_t index, size_t ndim,
size_t *dsize, size_t *coord)
Fill in the ‘coord’ array with the coordinates that correspond to
‘index’ assuming the dataset has ‘ndim’ elements and the size of
the dataset along each dimension is in the ‘dsize’ array. Note
that both ‘index’ and each value in ‘coord’ are assumed to start
from ‘0’ (zero). Also that the space which ‘coord’ points to must
already be allocated before calling this function.
-- Function:
size_t
gal_dimension_dist_manhattan (size_t *a, size_t *b, size_t
ndim)
Return the manhattan distance (see Wikipedia
(https://en.wikipedia.org/wiki/Taxicab_geometry)) between the two
coordinates ‘a’ and ‘b’ (each an array of ‘ndim’ elements).
-- Function:
float
gal_dimension_dist_radial (size_t *a, size_t *b, size_t ndim)
Return the radial distance between the two coordinates ‘a’ and ‘b’
(each an array of ‘ndim’ elements).
-- Function:
float
gal_dimension_dist_elliptical (double *center, double *pa_deg,
double *q, size_t ndim, double *point)
Return the elliptical/ellipsoidal distance of the single point
‘point’ (containing ‘ndim’ values: coordinates of the point in each
dimension) from an ellipse that is defined by ‘center’, ‘pa_deg’
and ‘q’. ‘center’ is the coordinates of the ellipse center (also
with ‘ndim’ elements). ‘pa’ is the position-angle in degrees (the
angle of the semi-major axis from the first dimension in a 2D
ellipse) and ‘q’ is the axis ratio.
In a 2D ellipse, ‘pa’ and ‘q’ are a single-element array. However,
in a 3D ellipsoid, ‘pa’ must have three elements, and ‘q’ must have
2 elements. For more see *note Defining an ellipse and
ellipsoid::.
-- Function:
gal_data_t *
gal_dimension_collapse_sum (gal_data_t *in, size_t c_dim,
gal_data_t *weight)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by summing all elements in that direction. If
‘weight!=NULL’, it must be a single-dimensional array, with the
same size as the dimension to be collapsed. The respective weight
will be multiplied to each element during the collapse.
For generality, the returned dataset will have a ‘GAL_TYPE_FLOAT64’
type. See *note Copying datasets:: for converting the returned
dataset to a desired type. Also, for more on the application of
this function, see the Arithmetic program's ‘collapse-sum’ operator
(which uses this function) in *note Arithmetic operators::.
-- Function:
gal_data_t *
gal_dimension_collapse_mean (gal_data_t *in, size_t c_dim,
gal_data_t *weight)
Similar to ‘gal_dimension_collapse_sum’ (above), but the collapse
will be done by calculating the mean along the requested dimension,
not summing over it.
-- Function:
gal_data_t *
gal_dimension_collapse_number (gal_data_t *in, size_t c_dim)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by counting how many non-blank elements there are along
that dimension.
For generality, the returned dataset will have a ‘GAL_TYPE_INT32’
type. See *note Copying datasets:: for converting the returned
dataset to a desired type. Also, for more on the application of
this function, see the Arithmetic program's ‘collapse-number’
operator (which uses this function) in *note Arithmetic
operators::.
-- Function:
gal_data_t *
gal_dimension_collapse_minmax (gal_data_t *in, size_t c_dim,
int max1_min0)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by using the largest/smallest non-blank value along
that dimension. If ‘max1_min0’ is non-zero, then the collapsed
dataset will have the maximum value along the given dimension and
if it is zero, the minimum.
-- Function:
gal_data_t *
gal_dimension_collapse_median (gal_data_t *in, size_t c_dim,
size_t numthreads, size_t minmapsize, int quietmmap)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by finding the median non-blank value along that
dimension. Since the median involves sorting, this operator
benefits from many threads (which needs to be set with
‘numthreads’). For more on ‘minmapsize’ and ‘quietmmap’ see *note
Memory management::.
-- Function:
gal_data_t *
gal_dimension_collapse_sclip_std (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by finding the standard deviation of pixels along that
dimension after sigma-clipping. Since sigma-clipping involves
sorting, this operator benefits from many threads (which needs to
be set with ‘numthreads’). For more on ‘minmapsize’ and
‘quietmmap’ see *note Memory management::. For more on sigma
clipping, see *note Sigma clipping::.
-- Function:
gal_data_t *
gal_dimension_collapse_sclip_fill_std (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Similar to ‘gal_dimension_collapse_sclip_std’, but with filled
re-clipping (see *note Contiguous outliers::).
-- Function:
gal_data_t *
gal_dimension_collapse_sclip_mad (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by finding the median absolute deviation (MAD) of
pixels along that dimension after sigma-clipping. Since
sigma-clipping involves sorting, this operator benefits from many
threads (which needs to be set with ‘numthreads’). For more on
‘minmapsize’ and ‘quietmmap’ see *note Memory management::. For
more on sigma clipping, see *note Sigma clipping::.
-- Function:
gal_data_t *
gal_dimension_collapse_sclip_fill_mad (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Similar to ‘gal_dimension_collapse_sclip_mad’, but with filled
re-clipping (see *note Contiguous outliers::).
-- Function:
gal_data_t *
gal_dimension_collapse_sclip_mean (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by finding the mean of pixels along that dimension
after sigma-clipping. Since sigma-clipping involves sorting, this
operator benefits from many threads (which needs to be set with
‘numthreads’). For more on ‘minmapsize’ and ‘quietmmap’ see *note
Memory management::. For more on sigma clipping, see *note Sigma
clipping::.
-- Function:
gal_data_t *
gal_dimension_collapse_sclip_fill_mean (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Similar to ‘gal_dimension_collapse_sclip_mean’, but with filled
re-clipping (see *note Contiguous outliers::).
-- Function:
gal_data_t *
gal_dimension_collapse_sclip_median (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by finding the median of pixels along that dimension
after sigma-clipping. Since sigma-clipping involves sorting, this
operator benefits from many threads (which needs to be set with
‘numthreads’). For more on ‘minmapsize’ and ‘quietmmap’ see *note
Memory management::. For more on sigma clipping, see *note Sigma
clipping::.
-- Function:
gal_data_t *
gal_dimension_collapse_sclip_fill_median (gal_data_t *in,
size_t c_dim, float multip, float param, size_t numthreads,
size_t minmapsize, int quietmmap)
Similar to ‘gal_dimension_collapse_sclip_median’, but with filled
re-clipping (see *note Contiguous outliers::).
-- Function:
gal_data_t *
gal_dimension_collapse_sclip_number (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by finding the number of pixels along that dimension
after sigma-clipping. Since sigma-clipping involves sorting, this
operator benefits from many threads (which needs to be set with
‘numthreads’). For more on ‘minmapsize’ and ‘quietmmap’ see *note
Memory management::. For more on sigma clipping, see *note Sigma
clipping::.
-- Function:
gal_data_t *
gal_dimension_collapse_sclip_fill_number (gal_data_t *in,
size_t c_dim, float multip, float param, size_t numthreads,
size_t minmapsize, int quietmmap)
Similar to ‘gal_dimension_collapse_sclip_number’, but with filled
re-clipping (see *note Contiguous outliers::).
-- Function:
gal_data_t *
gal_dimension_collapse_mclip_std (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by finding the standard deviation of pixels along that
dimension after median absolute deviation (MAD) clipping. Since
MAD-clipping involves sorting, this operator benefits from many
threads (which needs to be set with ‘numthreads’). For more on
‘minmapsize’ and ‘quietmmap’ see *note Memory management::. For
more on MAD-clipping, see *note MAD clipping::.
-- Function:
gal_data_t *
gal_dimension_collapse_mclip_fill_std (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Similar to ‘gal_dimension_collapse_mclip_std’, but with filled
re-clipping (see *note Contiguous outliers::).
-- Function:
gal_data_t *
gal_dimension_collapse_mclip_mad (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by finding the median absolute deviation (MAD) of
pixels along that dimension after median absolute deviation (MAD)
clipping. Since MAD-clipping involves sorting, this operator
benefits from many threads (which needs to be set with
‘numthreads’). For more on ‘minmapsize’ and ‘quietmmap’ see *note
Memory management::. For more on MAD-clipping, see *note MAD
clipping::.
-- Function:
gal_data_t *
gal_dimension_collapse_mclip_fill_mad (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Similar to ‘gal_dimension_collapse_mclip_mad’, but with filled
re-clipping (see *note Contiguous outliers::).
-- Function:
gal_data_t *
gal_dimension_collapse_mclip_mean (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by finding the mean of pixels along that dimension
after median absolute deviation (MAD) clipping. Since MAD-clipping
involves sorting, this operator benefits from many threads (which
needs to be set with ‘numthreads’). For more on ‘minmapsize’ and
‘quietmmap’ see *note Memory management::. For more on
MAD-clipping, see *note MAD clipping::.
-- Function:
gal_data_t *
gal_dimension_collapse_mclip_fill_mean (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Similar to ‘gal_dimension_collapse_mclip_mean’, but with filled
re-clipping (see *note Contiguous outliers::).
-- Function:
gal_data_t *
gal_dimension_collapse_mclip_median (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by finding the median of pixels along that dimension
after median absolute deviation (MAD) clipping. Since MAD-clipping
involves sorting, this operator benefits from many threads (which
needs to be set with ‘numthreads’). For more on ‘minmapsize’ and
‘quietmmap’ see *note Memory management::. For more on
MAD-clipping, see *note MAD clipping::.
-- Function:
gal_data_t *
gal_dimension_collapse_mclip_fill_median (gal_data_t *in,
size_t c_dim, float multip, float param, size_t numthreads,
size_t minmapsize, int quietmmap)
Similar to ‘gal_dimension_collapse_mclip_median’, but with filled
re-clipping (see *note Contiguous outliers::).
-- Function:
gal_data_t *
gal_dimension_collapse_mclip_number (gal_data_t *in, size_t
c_dim, float multip, float param, size_t numthreads, size_t
minmapsize, int quietmmap)
Collapse the input dataset (‘in’) along the given dimension
(‘c_dim’, in C definition: starting from zero, from the slowest
dimension), by finding the number of pixels along that dimension
after median absolute deviation (MAD) clipping. Since MAD-clipping
involves sorting, this operator benefits from many threads (which
needs to be set with ‘numthreads’). For more on ‘minmapsize’ and
‘quietmmap’ see *note Memory management::. For more on
MAD-clipping, see *note MAD clipping::.
-- Function:
gal_data_t *
gal_dimension_collapse_mclip_fill_number (gal_data_t *in,
size_t c_dim, float multip, float param, size_t numthreads,
size_t minmapsize, int quietmmap)
Similar to ‘gal_dimension_collapse_mclip_number’, but with filled
re-clipping (see *note Contiguous outliers::).
-- Function:
size_t
gal_dimension_remove_extra (size_t ndim, size_t *dsize, struct
wcsprm *wcs)
Remove extra dimensions (those that only have a length of 1) from
the basic size information of a dataset. If the total number of
elements (in all dimensions) is 1, this function will not remove
anything (because an "extra" is only meaningful when some
dimensions have more than one element length).
‘ndim’ is the number of dimensions and ‘dsize’ is an array with
‘ndim’ elements containing the size along each dimension in the C
dimension order. When ‘wcs!=NULL’, the respective dimension will
also be removed from the WCS.
This function will return the new number of dimensions and the
‘dsize’ elements will contain the length along each new dimension.
-- Function-like macro: GAL_DIMENSION_NEIGHBOR_OP (index, ndim, dsize,
connectivity, dinc, operation)
Parse the neighbors of the element located at ‘index’ and do the
requested operation on them. This is defined as a macro to allow
easy definition of any operation on the neighbors of a given
element without having to use loops within your source code (the
loops are implemented by this macro). For an example of using this
function, please see *note Library demo - inspecting neighbors::.
The input arguments to this function-like macro are described
below:
‘index’
Distance of this element from the first element in the array
on a contiguous patch of memory (starting from 0), see the
discussion above.
‘ndim’
The number of dimensions associated with the contiguous patch
of memory.
‘dsize’
The full array size along each dimension. This must be an
array and is assumed to have the same number elements as
‘ndim’. See the discussion under the same element in *note
Generic data container::.
‘connectivity’
Most distant neighbors to consider. Depending on the number
of dimensions, different neighbors may be defined for each
element. This function-like macro distinguish between these
different neighbors with this argument. It has a value
between ‘1’ (one) and ‘ndim’. For example, in a 2D dataset,
4-connected neighbors have a connectivity of ‘1’ and
8-connected neighbors have a connectivity of ‘2’. Note that
this is inclusive, so in this example, a connectivity of ‘2’
will also include connectivity ‘1’ neighbors.
‘dinc’
An array keeping the length necessary to increment along each
dimension. You can make this array with the following
function. Just do not forget to free the array after you are
done with it:
size_t *dinc=gal_dimension_increment(ndim, dsize);
free(dinc);
‘dinc’ depends on ‘ndim’ and ‘dsize’, but it must be defined
outside this function-like macro since it involves allocation
to help in performance.
‘operation’
Any C operation that you would like to do on the neighbor.
This macro will provide you a ‘nind’ variable that can be used
as the index of the neighbor that is currently being studied.
It is defined as '‘size_t ndim;’'. Note that ‘operation’ will
be repeated the number of times there is a neighbor for this
element.
This macro works fully within its own ‘{}’ block and except for the
‘nind’ variable that shows the neighbor's index, all the variables
within this macro's block start with ‘gdn_’.
12.3.8 Linked lists (‘list.h’)
------------------------------
An array is a contiguous region of memory that is very efficient and
easy to use for recording and later accessing any random element as fast
as any other. This makes array the primary data container when you have
many elements (for example, an image which has millions of pixels). One
major problem with an array is that the number of elements that go into
it must be known in advance and adding or removing an element will
require a re-set of all the other elements. For example, if you want to
remove the 3rd element in a 1000 element array, all 997 subsequent
elements have to pulled back by one position, the reverse will happen if
you need to add an element.
In many contexts such situations never come up, for example, you do
not want to shift all the pixels in an image by one or two pixels from
some random position in the image: their positions have scientific
value. But in other contexts you will find yourself frequently
adding/removing an a-priori unknown number of elements. Linked lists
(or _lists_ for short) are the data-container of choice in such
situations. As in a chain, each _node_ in a list is an independent C
structure, keeping its own data along with pointer(s) to its immediate
neighbor(s). Below, you can see one simple linked list node structure
along with an ASCII art schematic of how we can use the ‘next’ pointer
to add any number of elements to the list that we want. By convention,
a list is terminated when ‘next’ is the ‘NULL’ pointer.
struct list_float /* --------- --------- */
{ /* | Value | | Value | */
float value; /* | --- | | --- | */
struct list_float *next; /* | next-|--> | next-|--> NULL */
} /* --------- --------- */
The schematic shows another great advantage of linked lists: it is
very easy to add or remove/pop a node anywhere in the list. If you want
to modify the first node, you just have to change one pointer. If it is
in the middle, you just have to change two. You initially define a
variable of this type with a ‘NULL’ pointer as shown below:
struct list_float *list=NULL;
To add or remove/pop a node from the list you can use functions provided
for the respective type in the sections below.
When you add an element to the list, it is conventionally added to the
"top" of the list: the general list pointer will point to the newly
created node, which will point to the previously created node and so on.
So when you "pop" from the top of the list, you are actually retrieving
the last value you put in and changing the list pointer to the next
youngest node. This is thus known as a "last-in-first-out" list. This
is the most efficient type of linked list (easier to implement and
faster to process). Alternatively, you can add each newly created node
at the end of the list. If you do that, you will get a
"first-in-first-out" list. But that will force you to go through the
whole list for each new element that is created (this will slow down the
processing)(1).
The node example above creates the simplest kind of a list. We can
define each node with two pointers to both the next and previous
neighbors, this is called a "Doubly linked list". In general, lists are
very powerful and simple constructs that can be very useful. But going
into more detail would be out of the scope of this short introduction in
this book. Wikipedia (https://en.wikipedia.org/wiki/Linked_list) has a
nice and more thorough discussion of the various types of lists. To
appreciate/use the beauty and elegance of these powerful constructs even
further, see Chapter 2 (Information Structures, in volume 1) of Donald
Knuth's "The art of computer programming".
In this section we will review the functions and structures that are
available in Gnuastro for working on lists. They differ by the type of
data that each node can keep. For each linked-list node structure, we
will first introduce the structure, then the functions for working on
the structure. All these structures and functions are defined and
declared in ‘gnuastro/list.h’.
---------- Footnotes ----------
(1) A better way to get a first-in-first-out is to first keep the
data as last-in-first-out until they are all read. Afterwards, reverse
the list by popping each node and immediately add it to the new list.
This practically reverses the last-in-first-out list to a
first-in-first-out one. All the list types discussed in this chapter
have a function with a ‘_reverse’ suffix for this job.
12.3.8.1 List of strings
........................
Probably one of the most common lists you will be using are lists of
strings. They are the best tools when you are reading the user's
inputs, or when adding comments to the output files. Below you can see
Gnuastro's string list type and several functions to help in adding,
removing/popping, reversing and freeing the list.
-- Type (C struct): gal_list_str_t
A single node in a list containing a string of characters.
typedef struct gal_list_str_t
{
char *v;
struct gal_list_str_t *next;
} gal_list_str_t;
-- Function:
void
gal_list_str_add (gal_list_str_t **list, char *value, int
allocate)
Add a new node to the list of strings (‘list’) and update it. The
new node will contain the string ‘value’. If ‘allocate’ is not
zero, space will be allocated specifically for the string of the
new node and the contents of ‘value’ will be copied into it. This
can be useful when your string may be changed later in the program,
but you want your list to remain. Here is one short/simple example
of initializing and adding elements to a string list:
gal_list_str_t *list=NULL;
gal_list_str_add(&list, "bottom of list.", 1);
gal_list_str_add(&list, "second last element of list.", 1);
-- Function:
char *
gal_list_str_pop (gal_list_str_t **list)
Pop the top element of ‘list’, change ‘list’ to point to the next
node in the list, and return the string that was in the popped
node. If ‘*list==NULL’, then this function will also return a
‘NULL’ pointer.
-- Function:
size_t
gal_list_str_number (gal_list_str_t *list)
Return the number of nodes in ‘list’.
-- Function:
gal_list_str_t *
gal_list_str_last (gal_list_str_t *list)
Return a pointer to the last node in ‘list’.
-- Function:
void
gal_list_str_print (gal_list_str_t *list)
Print the strings within each node of ‘*list’ on the standard
output in the same order that they are stored. Each string is
printed on one line. This function is mainly good for
checking/debugging your program. For program outputs, it is best
to make your own implementation with a better, more user-friendly,
format. For example, the following code snippet.
size_t i=0;
gal_list_str_t *tmp;
for(tmp=list; tmp!=NULL; tmp=tmp->next)
printf("String %zu: %s\n", ++i, tmp->v);
-- Function:
void
gal_list_str_reverse (gal_list_str_t **list)
Reverse the order of the list such that the top node in the list
before calling this function becomes the bottom node after it.
-- Function:
void
gal_list_str_free (gal_list_str_t *list, int freevalue)
Free every node in ‘list’. If ‘freevalue’ is not zero, also free
the string within the nodes.
-- Function:
gal_list_str_t *
gal_list_str_extract (char *string)
Extract space-separated components of the input string. If any
space element should be kept (and not considered as a delimiter
between two tokens), precede it with a backslash (‘\’). Be aware
that in C programming, when including a backslash character within
a string literal, the correct format is indeed to use two
backslashes ("\\") to represent a single backslash:
gal_list_str_extract("bottom of\\ list");
-- Function:
char *
gal_list_str_cat (gal_list_str_t *list, char delimiter)
Concatenate (append) the input list of strings into a single string
where each node is separated from the next with the given
‘delimiter’. The space for the output string is allocated by this
function and should be freed when you have finished with it.
If there is any delimiter characters are present in any of the
elements, a backslash (‘\’) will be printed before the SPACE
character. This is necessary, otherwise, a function like
‘gal_list_str_extract’ will not be able to extract the elements
back into separate elements in a list.
12.3.8.2 List of ‘int32_t’
..........................
Signed integers are the best types when you are dealing with a positive
or negative integers. The are generally useful in many contexts, for
example when you want to keep the order of a series of states (each
state stored as a given number in an ‘enum’ for example). On many
modern systems, ‘int32_t’ is just an alias for ‘int’, so you can use
them interchangeably. To make sure, check the size of ‘int’ on your
system:
-- Type (C struct): gal_list_i32_t
A single node in a list containing a 32-bit signed integer (see
*note Numeric data types::).
typedef struct gal_list_i32_t
{
int32_t v;
struct gal_list_i32_t *next;
} gal_list_i32_t;
-- Function:
void
gal_list_i32_add (gal_list_i32_t **list, int32_t value)
Add a new node (containing ‘value’) to the top of the ‘list’ of
‘int32_t’s (‘uint32_t’ is equal to ‘int’ on many modern systems),
and update ‘list’. Here is one short example of initializing and
adding elements to a string list:
gal_list_i32_t *list=NULL;
gal_list_i32_add(&list, 52);
gal_list_i32_add(&list, -4);
-- Function:
int32_t
gal_list_i32_pop (gal_list_i32_t **list)
Pop the top element of ‘list’ and return the value. This function
will also change ‘list’ to point to the next node in the list. If
‘*list==NULL’, then this function will also return
‘GAL_BLANK_INT32’ (see *note Library blank values::).
-- Function:
size_t
gal_list_i32_number (gal_list_i32_t *list)
Return the number of nodes in ‘list’.
-- Function:
size_t
gal_list_i32_last (gal_list_i32_t *list)
Return a pointer to the last node in ‘list’.
-- Function:
void
gal_list_i32_print (gal_list_i32_t *list)
Print the integers within each node of ‘*list’ on the standard
output in the same order that they are stored. Each integer is
printed on one line. This function is mainly good for
checking/debugging your program. For program outputs, it is best
to make your own implementation with a better, more user-friendly
format. For example, the following code snippet. You can also
modify it to print all values in one line, etc., depending on the
context of your program.
size_t i=0;
gal_list_i32_t *tmp;
for(tmp=list; tmp!=NULL; tmp=tmp->next)
printf("Number %zu: %s\n", ++i, tmp->v);
-- Function:
void
gal_list_i32_reverse (gal_list_i32_t **list)
Reverse the order of the list such that the top node in the list
before calling this function becomes the bottom node after it.
-- Function:
int32_t *
gal_list_i32_to_array (gal_list_i32_t *list, int reverse,
size_t *num)
Dynamically allocate an array and fill it with the values in
‘list’. The function will return a pointer to the allocated array
and put the number of elements in the array into the ‘num’ pointer.
If ‘reverse’ has a non-zero value, the array will be filled in the
opposite order of elements in ‘list’. This function can be useful
after you have finished reading an initially unknown number of
values and want to put them in an array for easy random access.
-- Function:
void
gal_list_i32_free (gal_list_i32_t *list)
Free every node in ‘list’.
12.3.8.3 List of ‘size_t’
.........................
The ‘size_t’ type is a unique type in C: as the name suggests it is
defined to store sizes, or more accurately, the distances between memory
locations. Hence it is always positive (an ‘unsigned’ type) and it is
directly related to the address-able spaces on the host system: on
32-bit and 64-bit systems it is an alias for ‘uint32_t’ and ‘uint64_t’,
respectively (see *note Numeric data types::).
‘size_t’ is the default compiler type to index an array (recall that
an array index in C is just a pointer increment of a given _size_).
Since it is unsigned, it is a great type for counting (where negative is
not defined), you are always sure it will never exceed the system's
(virtual) memory and since its name has the word "size" inside it, it
provides a good level of documentation(1). In Gnuastro, we do all
counting and array indexing with this type, so this list is very handy.
As discussed above, ‘size_t’ maps to different types on different
machines, so a portable way to print them with ‘printf’ is to use C99's
‘%zu’ format.
-- Type (C struct): gal_list_sizet_t
A single node in a list containing a ‘size_t’ value (which maps to
‘uint32_t’ or ‘uint64_t’ on 32-bit and 64-bit systems), see *note
Numeric data types::.
typedef struct gal_list_sizet_t
{
size_t v;
struct gal_list_sizet_t *next;
} gal_list_sizet_t;
-- Function:
void
gal_list_sizet_add (gal_list_sizet_t **list, size_t value)
Add a new node (containing ‘value’) to the top of the ‘list’ of
‘size_t’s and update ‘list’. Here is one short example of
initializing and adding elements to a string list:
gal_list_sizet_t *list=NULL;
gal_list_sizet_add(&list, 45493);
gal_list_sizet_add(&list, 930484);
-- Function:
sizet_t
gal_list_sizet_pop (gal_list_sizet_t **list)
Pop the top element of ‘list’ and return the value. This function
will also change ‘list’ to point to the next node in the list. If
‘*list==NULL’, then this function will also return
‘GAL_BLANK_SIZE_T’ (see *note Library blank values::).
-- Function:
size_t
gal_list_sizet_number (gal_list_sizet_t *list)
Return the number of nodes in ‘list’.
-- Function:
size_t
gal_list_sizet_last (gal_list_sizet_t *list)
Return a pointer to the last node in ‘list’.
-- Function:
void
gal_list_sizet_print (gal_list_sizet_t *list)
Print the values within each node of ‘*list’ on the standard output
in the same order that they are stored. Each integer is printed on
one line. This function is mainly good for checking/debugging your
program. For program outputs, it is best to make your own
implementation with a better, more user-friendly format. For
example, the following code snippet. You can also modify it to
print all values in one line, etc., depending on the context of
your program.
size_t i=0;
gal_list_sizet_t *tmp;
for(tmp=list; tmp!=NULL; tmp=tmp->next)
printf("Number %zu: %zu\n", ++i, tmp->v);
-- Function:
void
gal_list_sizet_reverse (gal_list_sizet_t **list)
Reverse the order of the list such that the top node in the list
before calling this function becomes the bottom node after it.
-- Function:
size_t *
gal_list_sizet_to_array (gal_list_sizet_t *list, int reverse,
size_t *num)
Dynamically allocate an array and fill it with the values in
‘list’. The function will return a pointer to the allocated array
and put the number of elements in the array into the ‘num’ pointer.
If ‘reverse’ has a non-zero value, the array will be filled in the
inverse of the order of elements in ‘list’. This function can be
useful after you have finished reading an initially unknown number
of values and want to put them in an array for easy random access.
-- Function:
void
gal_list_sizet_free (gal_list_sizet_t *list)
Free every node in ‘list’.
---------- Footnotes ----------
(1) So you know that a variable of this type is not used to store
some generic state for example.
12.3.8.4 List of ‘float’
........................
Single precision floating point numbers can accurately store real number
until 7.2 decimals and only consume 4 bytes (32-bits) of memory, see
*note Numeric data types::. Since astronomical data rarely reach that
level of precision, single precision floating points are the type of
choice to keep and read data. However, when processing the data, it is
best to use double precision floating points (since errors propagate).
-- Type (C struct): gal_list_f32_t
A single node in a list containing a 32-bit single precision
‘float’ value: see *note Numeric data types::.
typedef struct gal_list_f32_t
{
float v;
struct gal_list_f32_t *next;
} gal_list_f32_t;
-- Function:
void
gal_list_f32_add (gal_list_f32_t **list, float value)
Add a new node (containing ‘value’) to the top of the ‘list’ of
‘float’s and update ‘list’. Here is one short example of
initializing and adding elements to a string list:
gal_list_f32_t *list=NULL;
gal_list_f32_add(&list, 3.89);
gal_list_f32_add(&list, 1.23e-20);
-- Function:
float
gal_list_f32_pop (gal_list_f32_t **list)
Pop the top element of ‘list’ and return the value. This function
will also change ‘list’ to point to the next node in the list. If
‘*list==NULL’, then this function will return ‘GAL_BLANK_FLOAT32’
(NaN, see *note Library blank values::).
-- Function:
size_t
gal_list_f32_number (gal_list_f32_t *list)
Return the number of nodes in ‘list’.
-- Function:
size_t
gal_list_f32_last (gal_list_f32_t *list)
Return a pointer to the last node in ‘list’.
-- Function:
void
gal_list_f32_print (gal_list_f32_t *list)
Print the values within each node of ‘*list’ on the standard output
in the same order that they are stored. Each floating point number
is printed on one line. This function is mainly good for
checking/debugging your program. For program outputs, it is best
to make your own implementation with a better, more user-friendly
format. For example, in the following code snippet. You can also
modify it to print all values in one line, etc., depending on the
context of your program.
size_t i=0;
gal_list_f32_t *tmp;
for(tmp=list; tmp!=NULL; tmp=tmp->next)
printf("Number %zu: %f\n", ++i, tmp->v);
-- Function:
void
gal_list_f32_reverse (gal_list_f32_t **list)
Reverse the order of the list such that the top node in the list
before calling this function becomes the bottom node after it.
-- Function:
float *
gal_list_f32_to_array (gal_list_f32_t *list, int reverse,
size_t *num)
Dynamically allocate an array and fill it with the values in
‘list’. The function will return a pointer to the allocated array
and put the number of elements in the array into the ‘num’ pointer.
If ‘reverse’ has a non-zero value, the array will be filled in the
inverse of the order of elements in ‘list’. This function can be
useful after you have finished reading an initially unknown number
of values and want to put them in an array for easy random access.
-- Function:
void
gal_list_f32_free (gal_list_f32_t *list)
Free every node in ‘list’.
12.3.8.5 List of ‘double’
.........................
Double precision floating point numbers can accurately store real number
until 15.9 decimals and consume 8 bytes (64-bits) of memory, see *note
Numeric data types::. This level of precision makes them very good for
serious processing in the middle of a program's execution: in many
cases, the propagation of errors will still be insignificant compared to
actual observational errors in a data set. But since they consume 8
bytes and more CPU processing power, they are often not the best choice
for storing and transferring of data.
-- Type (C struct): gal_list_f64_t
A single node in a list containing a 64-bit double precision
‘double’ value: see *note Numeric data types::.
typedef struct gal_list_f64_t
{
double v;
struct gal_list_f64_t *next;
} gal_list_f64_t;
-- Function:
void
gal_list_f64_add (gal_list_f64_t **list, double value)
Add a new node (containing ‘value’) to the top of the ‘list’ of
‘double’s and update ‘list’. Here is one short example of
initializing and adding elements to a string list:
gal_list_f64_t *list=NULL;
gal_list_f64_add(&list, 3.8129395763193);
gal_list_f64_add(&list, 1.239378923931e-20);
-- Function:
double
gal_list_f64_pop (gal_list_f64_t **list)
Pop the top element of ‘list’ and return the value. This function
will also change ‘list’ to point to the next node in the list. If
‘*list==NULL’, then this function will return ‘GAL_BLANK_FLOAT64’
(NaN, see *note Library blank values::).
-- Function:
size_t
gal_list_f64_number (gal_list_f64_t *list)
Return the number of nodes in ‘list’.
-- Function:
size_t
gal_list_f64_last (gal_list_f64_t *list)
Return a pointer to the last node in ‘list’.
-- Function:
void
gal_list_f64_print (gal_list_f64_t *list)
Print the values within each node of ‘*list’ on the standard output
in the same order that they are stored. Each floating point number
is printed on one line. This function is mainly good for
checking/debugging your program. For program outputs, it is best
to make your own implementation with a better, more user-friendly
format. For example, in the following code snippet. You can also
modify it to print all values in one line, etc., depending on the
context of your program.
size_t i=0;
gal_list_f64_t *tmp;
for(tmp=list; tmp!=NULL; tmp=tmp->next)
printf("Number %zu: %f\n", ++i, tmp->v);
-- Function:
void
gal_list_f64_reverse (gal_list_f64_t **list)
Reverse the order of the list such that the top node in the list
before calling this function becomes the bottom node after it.
-- Function:
double *
gal_list_f64_to_array (gal_list_f64_t *list, int reverse,
size_t *num)
Dynamically allocate an array and fill it with the values in
‘list’. The function will return a pointer to the allocated array
and put the number of elements in the array into the ‘num’ pointer.
If ‘reverse’ has a non-zero value, the array will be filled in the
inverse of the order of elements in ‘list’. This function can be
useful after you have finished reading an initially unknown number
of values and want to put them in an array for easy random access.
-- Function:
gal_data_t *
gal_list_f64_to_data (gal_list_f64_t *list, uint8_t type,
size_t minmapsize, int quietmmap)
Write the values in the given ‘list’ into a ‘gal_data_t’ dataset of
the requested ‘type’. The order of the values in the dataset will
be the same as the order from the top of the list.
-- Function:
void
gal_list_f64_free (gal_list_f64_t *list)
Free every node in ‘list’.
12.3.8.6 List of ‘void *’
.........................
In C, ‘void *’ is the most generic pointer. Usually pointers are
associated with the type of content they point to. For example, ‘int *’
means a pointer to an integer. This ancillary information about the
contents of the memory location is very useful for the compiler,
catching bad errors and also documentation (it helps the reader see what
the address in memory actually contains). However, ‘void *’ is just a
raw address (pointer), it contains no information on the contents it
points to.
These properties make the ‘void *’ very useful when you want to treat
the contents of an address in different ways. You can use the ‘void *’
list defined in this section and its function on any kind of data: for
example, you can use it to keep a list of custom data structures that
you have built for your own separate program. Each node in the list can
keep anything and this gives you great versatility. But in using ‘void
*’, please beware that "with great power comes great responsibility".
-- Type (C struct): gal_list_void_t
A single node in a list containing a ‘void *’ pointer.
typedef struct gal_list_void_t
{
void *v;
struct gal_list_void_t *next;
} gal_list_void_t;
-- Function:
void
gal_list_void_add (gal_list_void_t **list, void *value)
Add a new node (containing ‘value’) to the top of the ‘list’ of
‘void *’s and update ‘list’. Here is one short example of
initializing and adding elements to a string list:
gal_list_void_t *list=NULL;
gal_list_f64_add(&list, some_pointer);
gal_list_f64_add(&list, another_pointer);
-- Function:
void *
gal_list_void_pop (gal_list_void_t **list)
Pop the top element of ‘list’ and return the value. This function
will also change ‘list’ to point to the next node in the list. If
‘*list==NULL’, then this function will return ‘NULL’.
-- Function:
size_t
gal_list_void_number (gal_list_void_t *list)
Return the number of nodes in ‘list’.
-- Function:
size_t
gal_list_void_last (gal_list_void_t *list)
Return a pointer to the last node in ‘list’.
-- Function:
void
gal_list_void_reverse (gal_list_void_t **list)
Reverse the order of the list such that the top node in the list
before calling this function becomes the bottom node after it.
-- Function:
void
gal_list_void_free (gal_list_void_t *list)
Free every node in ‘list’.
12.3.8.7 Ordered list of ‘size_t’
.................................
Positions/sizes in a dataset are conventionally in the ‘size_t’ type
(see *note List of size_t::) and it sometimes occurs that you want to
parse and read the values in a specific order. For example, you want to
start from one pixel and add pixels to the list based on their distance
to that pixel. So that ever time you pop an element from the list, you
know it is the nearest that has not yet been studied. The
‘gal_list_osizet_t’ type and its functions in this section are designed
to facilitate such operations.
-- Type (C struct): gal_list_osizet_t
Each node in this singly-linked list contains a ‘size_t’ value and
a floating point value. The floating point value is used as a
reference to add new nodes in a sorted manner. At any moment, the
first popped node in this list will have the smallest ‘tosort’
value, and subsequent nodes will have larger to values.
typedef struct gal_list_osizet_t
{
size_t v; /* The actual value. */
float s; /* The parameter to sort by. */
struct gal_list_osizet_t *next;
} gal_list_osizet_t;
-- Function:
void
gal_list_osizet_add (gal_list_osizet_t **list, size_t value,
float tosort)
Allocate space for a new node in ‘list’, and store ‘value’ and
‘tosort’ into it. The new node will not necessarily be at the
"top" of the list. If ‘*list!=NULL’, then the ‘tosort’ values of
existing nodes is inspected and the given node is placed in the
list such that the top element (which is popped with
‘gal_list_osizet_pop’) has the smallest ‘tosort’ value.
-- Function:
size_t
gal_list_osizet_pop (gal_list_osizet_t **list, float
*sortvalue)
Pop a node from the top of ‘list’, return the node's ‘value’ and
put its sort value in the space that ‘sortvalue’ points to. This
function will also free the allocated space for the popped node and
after this function, ‘list’ will point to the next node (which has
a larger ‘tosort’ element).
-- Function:
void
gal_list_osizet_to_sizet_free (gal_list_osizet_t *in,
gal_list_sizet_t **out)
Convert the ordered list of ‘size_t’s into an ordinary ‘size_t’
linked list. This can be useful when all the elements have been
added and you just need to pop-out elements and do not care about
the sorting values any more. After the conversion is done, this
function will free the input list. Note that the ‘out’ list does
not have to be empty. If it already contains some nodes, the new
nodes will be added on top of them.
12.3.8.8 Doubly linked ordered list of ‘size_t’
...............................................
An ordered list of indices is required in many contexts, one example was
discussed at the beginning of *note Ordered list of size_t::. But the
list that was introduced there only has one point of entry: you can
always only parse the list from smallest to largest. In this section,
the doubly-linked ‘gal_list_dosizet_t’ node is defined which will allow
us to parse the values in ascending or descending order.
-- Type (C struct): gal_list_dosizet_t
Doubly-linked, ordered ‘size_t’ list node structure. Each node in
this Doubly-linked list contains a ‘size_t’ value and a floating
point value. The floating point value is used as a reference to
add new nodes in a sorted manner. In the functions here, this
linked list can be pointed to by two pointers (largest and
smallest) with the following format:
largest pointer
|
NULL <-- (v0,s0) <--> (v1,s1) <--> ... (vn,sn) --> NULL
|
smallest pointer
At any moment, the two pointers will point to the nodes containing
the "largest" and "smallest" values and the rest of the nodes will
be sorted. This is useful when an unknown number of nodes are
being added continuously and during the operations it is important
to have the nodes in a sorted format.
typedef struct gal_list_dosizet_t
{
size_t v; /* The actual value. */
float s; /* The parameter to sort by. */
struct gal_list_dosizet_t *prev;
struct gal_list_dosizet_t *next;
} gal_list_dosizet_t;
-- Function:
void
gal_list_dosizet_add (gal_list_dosizet_t **largest,
gal_list_dosizet_t **smallest, size_t value, float tosort)
Allocate space for a new node in ‘list’, and store ‘value’ and
‘tosort’ into it. If the list is empty, both ‘largest’ and
‘smallest’ must be ‘NULL’.
-- Function:
size_t
gal_list_dosizet_pop_smallest (gal_list_dosizet_t **largest,
gal_list_dosizet_t **smallest, float tosort)
Pop the value with the smallest reference from the doubly linked
list and store the reference into the space pointed to by ‘tosort’.
Note that even though only the smallest pointer will be popped,
when there was only one node in the list, the ‘largest’ pointer
also has to change, so we need both.
-- Function:
void
gal_list_dosizet_print (gal_list_dosizet_t *largest,
gal_list_dosizet_t *smallest)
Print the largest and smallest values sequentially until the list
is parsed.
-- Function:
void
gal_list_dosizet_to_sizet (gal_list_dosizet_t *in,
gal_list_sizet_t **out)
Convert the doubly linked, ordered ‘size_t’ list into a
singly-linked list of ‘size_t’.
-- Function:
void
gal_list_dosizet_free (gal_list_dosizet_t *largest)
Free the doubly linked, ordered ‘sizet_t’ list.
12.3.8.9 List of ‘gal_data_t’
.............................
Gnuastro's generic data container has a ‘next’ element which enables it
to be used as a singly-linked list (see *note Generic data container::).
The ability to connect the different data containers offers great
advantages. For example, each column in a table in an independent
dataset: with its own name, units, numeric data type (see *note Numeric
data types::). Another application is in Tessellating an input dataset
into separate tiles or only studying particular regions, or tiles, of a
larger dataset (see *note Tessellation:: and *note Tessellation
library::). Each independent tile over the dataset can be connected to
the others as a linked list and thus any number of tiles can be
represented with one variable.
-- Function:
void
gal_list_data_add (gal_data_t **list, gal_data_t *newnode)
Add an already allocated dataset (‘newnode’) to top of ‘list’.
Note that if ‘newnode->next!=NULL’ (‘newnode’ is itself a list),
then ‘list’ will be added to its end.
In this example multiple images are linked together as a list:
int quietmmap=1;
size_t minmapsize=-1;
gal_data_t *tmp, *list=NULL;
tmp = gal_fits_img_read("file1.fits", "1", minmapsize, quietmmap,
NULL);
gal_list_data_add( &list, tmp );
tmp = gal_fits_img_read("file2.fits", "1", minmapsize, quietmmap,
NULL);
gal_list_data_add( &list, tmp );
-- Function:
void
gal_list_data_add_alloc (gal_data_t **list, void *array,
uint8_t type, size_t ndim, size_t *dsize, struct wcsprm *wcs,
int clear, size_t minmapsize, int quietmmap, char *name, char
*unit, char *comment)
Allocate a new dataset (with ‘gal_data_alloc’ in *note Dataset
allocation::) and put it as the first element of ‘list’. Note that
if this is the first node to be added to the list, ‘list’ must be
‘NULL’.
-- Function:
gal_data_t *
gal_list_data_pop (gal_data_t **list)
Pop the top node from ‘list’ and return it.
-- Function:
void
gal_list_data_remove (gal_data_t **list, gal_data_t *node)
Remove ‘node’ from the given ‘list’. After finding the given node,
this function will just set ‘node->next=NULL’ and correct the
‘next’ node of its previous element to its next element (thus
"removing" it from the list). If ‘node’ doesn't exist in the list,
this function won't make any change to list.
-- Function:
gal_data_t *
gal_list_data_select_by_name (gal_data_t *list, char *name)
Select the dataset within the list, that has a ‘name’ element that
is identical (case-sensitive) to the given ‘name’. If not found, a
‘NULL’ pointer will be returned.
Note that this dataset will not be popped from the list, only a
pointer to it will be returned and if you free it or change its
‘next’ element, it may harm your original list.
-- Function:
gal_data_t *
gal_list_data_select_by_id (gal_data_t *table, char *idstr,
size_t *index)
Select the dataset within the list that can be identified with the
string given to ‘idstr’ (which can be a counter, starting from 1,
or a name). If not found, a ‘NULL’ pointer will be returned.
Note that this dataset will not be popped from the list, only a
pointer to it will be returned and if you free it or change its
‘next’ element, it may harm your original list.
-- Function:
void
gal_list_data_reverse (gal_data_t **list)
Reverse the order of the list such that the top node in the list
before calling this function becomes the bottom node after it.
-- Function:
gal_data_t **
gal_list_data_to_array_ptr (gal_data_t *list, size_t *num)
Allocate and return an array of ‘gal_data_t *’ pointers with the
same number of elements as the nodes in ‘list’. The pointers will
be put in the same order that the list is parsed. Hence the N-th
element in the array will point to the same dataset that the N-th
node in the list points to.
-- Function:
size_t
gal_list_data_number (gal_data_t *list)
Return the number of nodes in ‘list’.
-- Function:
gal_data_t *
gal_list_data_last (gal_data_t *list)
Return a pointer to the last node in ‘list’.
-- Function:
void
gal_list_data_free (gal_data_t *list)
Free all the datasets in ‘list’ along with all the allocated spaces
in each.
12.3.9 Array input output
-------------------------
Getting arrays (commonly images or cubes) from a file into your program
or writing them after the processing into an output file are some of the
most common operations. The functions in this section are designed for
such operations with the known file types. The functions here are thus
just wrappers around functions of lower-level file type functions of
this library, for example, *note FITS files:: or *note TIFF files::. If
the file type of the input/output file is already known, you can use the
functions in those sections respectively.
-- Function:
int
gal_array_name_recognized (char *filename)
Return 1 if the given file name corresponds to one of the
recognized file types for reading arrays.
-- Function:
int
gal_array_name_recognized_multiext (char *filename)
Return 1 if the given file name corresponds to one of the
recognized file types for reading arrays which may contain multiple
extensions (for example FITS or TIFF) formats.
-- Function:
int
gal_array_file_recognized (char *filename)
Similar to ‘gal_array_name_recognized’, but for FITS files, it will
also check the contents of the file if the recognized file name
suffix is not found. See the description of
‘gal_fits_file_recognized’ for more (*note FITS macros errors
filenames::).
-- Function:
gal_data_t
gal_array_read (char *filename, char *extension,
gal_list_str_t *lines, size_t minmapsize, int quietmmap, char
*hdu_option_name)
Read the array within the given extension (‘extension’) of
‘filename’, or the ‘lines’ list (see below). If the array is
larger than ‘minmapsize’ bytes, then it will not be read into RAM,
but a file on the HDD/SSD (no difference for the programmer).
Messages about the memory-mapped file can be disabled with
‘quietmmap’.
‘extension’ will be ignored for files that do not support them (for
example JPEG or text). For FITS files, ‘extension’ can be a number
or a string (name of the extension), but for TIFF files, it has to
be number. In both cases, counting starts from zero.
For multi-channel formats (like RGB images in JPEG or TIFF), this
function will return a *note List of gal_data_t::: one data
structure per channel. Thus if you just want a single array (and
want to check if the user has not given a multi-channel input), you
can check the ‘next’ pointer of the returned ‘gal_data_t’.
‘lines’ is a list of strings with each node representing one line
(including the new-line character), see *note List of strings::.
It will mostly be the output of ‘gal_txt_stdin_read’, which is used
to read the program's input as separate lines from the standard
input (see *note Text files::). Note that ‘filename’ and ‘lines’
are mutually exclusive and one of them must be ‘NULL’.
‘hdu_option_name’ is used in error messages related to extensions
(e.g., HDUs in FITS) and is expected to be the command-line option
name that users of your program can specify to select an input HDU
for this particular input; for example, if the kernel is used as
the input, users should determine ‘--khdu’ for this option. If the
given ‘extension’ doesn't exist in ‘filename’, a descriptive error
message is printed instructing the users how to find and fix the
problem. This error message also suggests the option to use in
order to help users of your program to inform them what option they
should give a HDU to. If you don't have an option that is
configured by the users of your program, you can set this to ‘NONE’
or ‘NULL’.
-- Function:
void
gal_array_read_to_type (char *filename, char *extension,
gal_list_str_t *lines, uint8_t type, size_t minmapsize, int
quietmmap, char *hdu_option_name)
Similar to ‘gal_array_read’, but the output data structure(s) will
have a numeric data type of ‘type’, see *note Numeric data types::.
-- Function:
void
gal_array_read_one_ch (char *filename, char *extension,
gal_list_str_t *lines, size_t minmapsize, int quietmmap, char
*hdu_option_name)
Read the dataset within ‘filename’ (extension/hdu/dir ‘extension’)
and make sure it is only a single channel. This is just a simple
wrapper around ‘gal_array_read’ that checks if there was more than
one dataset and aborts with an informative error if there is more
than one channel in the dataset.
Formats like JPEG or TIFF support multiple channels per input, but
it may happen that your program only works on a single dataset.
This function can be a convenient way to make sure that the data
that comes into your program is only one channel.
Regarding ‘*hdu_option_name’, see the description for the same
argument in the description of ‘gal_array_read’.
-- Function:
void
gal_array_read_one_ch_to_type (char *filename, char
*extension, gal_list_str_t *lines, uint8_t type, size_t
minmapsize, int quietmmap, char *hdu_option_name)
Similar to ‘gal_array_read_one_ch’, but the output data structure
will has a numeric data type of ‘type’, see *note Numeric data
types::.
12.3.10 Table input output (‘table.h’)
--------------------------------------
Tables are a collection of one dimensional datasets that are packed
together into one file. They are the single most common format to store
high-level (processed) information, hence they play a very important
role in Gnuastro. For a more thorough introduction, please see *note
Table::. Gnuastro's Table program, and all the other programs that can
read from and write into tables, use the functions of this section for
reading and writing their input/output tables. For a simple
demonstration of using the constructs introduced here, see *note Library
demo - reading and writing table columns::.
Currently only plain text (see *note Gnuastro text table format::)
and FITS (ASCII and binary) tables are supported by Gnuastro. However,
the low-level table infra-structure is written such that accommodating
other formats is also possible and in future releases more formats will
hopefully be supported. Please do not hesitate to suggest your favorite
format so it can be implemented when possible.
-- Macro: GAL_TABLE_DEF_WIDTH_STR
-- Macro: GAL_TABLE_DEF_WIDTH_INT
-- Macro: GAL_TABLE_DEF_WIDTH_LINT
-- Macro: GAL_TABLE_DEF_WIDTH_FLT
-- Macro: GAL_TABLE_DEF_WIDTH_DBL
-- Macro: GAL_TABLE_DEF_PRECISION_INT
-- Macro: GAL_TABLE_DEF_PRECISION_FLT
-- Macro: GAL_TABLE_DEF_PRECISION_DBL
The default width and precision for generic types to use in writing
numeric types into a text file (plain text and FITS ASCII tables).
When the dataset does not have any pre-set width and precision (see
‘disp_width’ and ‘disp_precision’ in *note Generic data
container::) these will be directly used in C's ‘printf’ command to
write the number as a string.
-- Macro: GAL_TABLE_DISPLAY_FMT_STRING
-- Macro: GAL_TABLE_DISPLAY_FMT_DECIMAL
-- Macro: GAL_TABLE_DISPLAY_FMT_UDECIMAL
-- Macro: GAL_TABLE_DISPLAY_FMT_OCTAL
-- Macro: GAL_TABLE_DISPLAY_FMT_HEX
-- Macro: GAL_TABLE_DISPLAY_FMT_FIXED
-- Macro: GAL_TABLE_DISPLAY_FMT_EXP
-- Macro: GAL_TABLE_DISPLAY_FMT_GENERAL
The display format used in C's ‘printf’ to display data of
different types. The ‘_STRING’ and ‘_DECIMAL’ are unique for
printing strings and signed integers, they are mainly here for
completeness. However, unsigned integers and floating points can
be displayed in multiple formats:
Unsigned integer
For unsigned integers, it is possible to choose from
‘_UDECIMAL’ (unsigned decimal), ‘_OCTAL’ (octal notation, for
example, ‘125’ in decimal will be displayed as ‘175’), and
‘_HEX’ (hexadecimal notation, for example, ‘125’ in decimal
will be displayed as ‘7D’).
Floating point
For floating point, it is possible to display the number in
‘_FLOAT’ (floating point, for example, ‘1500.345’), ‘_EXP’
(exponential, for example, ‘1.500345e+03’), or ‘_GENERAL’
which is the best of the two for the given number.
-- Macro: GAL_TABLE_FORMAT_INVALID
-- Macro: GAL_TABLE_FORMAT_TXT
-- Macro: GAL_TABLE_FORMAT_AFITS
-- Macro: GAL_TABLE_FORMAT_BFITS
All the current acceptable table formats to Gnuastro. The ‘AFITS’
and ‘BFITS’ represent FITS ASCII tables and FITS Binary tables.
You can use these anywhere you see the ‘tableformat’ variable.
-- Macro: GAL_TABLE_SEARCH_INVALID
-- Macro: GAL_TABLE_SEARCH_NAME
-- Macro: GAL_TABLE_SEARCH_UNIT
-- Macro: GAL_TABLE_SEARCH_COMMENT
When the desired column is not a number, these values determine if
the string to match, or regular expression to search, be in the
_name_, _units_ or _comments_ of the column metadata. These values
should be used for the ‘searchin’ variables of the functions.
-- Function:
uint8_t
gal_table_displayflt_from_str (char *string)
Convert the input ‘string’ into one of the
‘GAL_TABLE_DISPLAY_FMT_FIXED’ (for fixed-point notation) or
‘GAL_TABLE_DISPLAY_FMT_EXP’ (for exponential notation).
-- Function:
char *
gal_table_displayflt_to_str (uint8_t id)
Convert the input identifier (one of the
‘GAL_TABLE_DISPLAY_FMT_FIXED’; for fixed-point notation, or
‘GAL_TABLE_DISPLAY_FMT_EXP’; for exponential notation) into a
standard string that is used to identify them.
-- Function:
gal_data_t *
gal_table_info (char *filename, char *hdu, gal_list_str_t
*lines, size_t *numcols, size_t *numrows, int *tableformat)
Store the information of each column of a table into an array of
meta-data ‘gal_data_t’s. In a metadata ‘gal_data_t’, the size
elements are zero (‘ndim=size=0’ and ‘dsize=NULL’) but other
relevant elements are filled). See the end of this description for
the exact components of each ‘gal_data_t’ that are filled.
The returned array of ‘gal_data_t’s has ‘numcols’ datasets (one
data structure for each column). The number of rows in each
dataset is stored in ‘numrows’ (in a table, all the columns have
the same number of rows). The format of the table (e.g., ASCII
text file, or FITS binary or ASCII table) will be put in
‘tableformat’ (macros defined above). If the ‘filename’ is not a
FITS file, then ‘hdu’ will not be used (can be ‘NULL’).
The input must be either a file (specified by ‘filename’) or a list
of strings (‘lines’). ‘lines’ is a list of strings with each node
representing one line (including the new-line character), see *note
List of strings::. It will mostly be the output of
‘gal_txt_stdin_read’, which is used to read the program's input as
separate lines from the standard input (see *note Text files::).
Note that ‘filename’ and ‘lines’ are mutually exclusive and one of
them must be ‘NULL’.
In the output datasets, only the meta-data strings (column name,
units and comments), will be allocated and set as shown below.
This function is just for column information (meta-data), not
column contents.
*restrict array -> Blank value (if present, in col's own type).
type -> Type of column data.
ndim -> 0
*dsize -> NULL
size -> 0
quietmmap -> ------------
*mmapname -> ------------
minmapsize -> Repeat (length of vector; 1 if not vector).
nwcs -> ------------
*wcs -> ------------
flag -> 'GAL_TABLEINTERN_FLAG_*' macros.
status -> ------------
*name -> Column name.
*unit -> Column unit.
*comment -> Column comments.
disp_fmt -> 'GAL_TABLE_DISPLAY_FMT' macros.
disp_width -> Width of string columns.
disp_precision -> ------------
*next -> Pointer to next column's metadata
*block -> ------------
-- Function:
void
gal_table_print_info (gal_data_t *allcols, size_t numcols,
size_t numrows, char *hdu_option_name)
Print the column information for all the columns (output of
‘gal_table_info’) to standard output. The output is in the same
format as this command with Gnuastro Table program (see *note
Invoking asttable::):
$ asttable --info table.fits
-- Function:
gal_data_t *
gal_table_read (char *filename, char *hdu, gal_list_str_t
*lines, gal_list_str_t *cols, int searchin, int ignorecase,
size_t numthreads, size_t minmapsize, int quietmmap, size_t
*colmatch, char *hdu_option_name)
Read the specified columns in a file (named ‘filename’), or list of
strings (‘lines’) into a linked list of data structures. If the
file is FITS, then ‘hdu’ will also be used, otherwise, ‘hdu’ is
ignored. For more on ‘hdu_option_name’ see the description of
‘gal_array_read’ in *note Array input output::.
‘lines’ is a list of strings with each node representing one line
(including the new-line character), see *note List of strings::.
It will mostly be the output of ‘gal_txt_stdin_read’, which is used
to read the program's input as separate lines from the standard
input (see *note Text files::). Note that ‘filename’ and ‘lines’
are mutually exclusive and one of them must be ‘NULL’.
The information to search for columns should be specified by the
‘cols’ list of strings (see *note List of strings::). The string
in each node of the list may be a number, an exact match to a
column name, or a regular expression (in GNU AWK format) enclosed
in ‘/ /’. The ‘searchin’ value must be one of the macros defined
above. If ‘cols’ is NULL, then this function will read the full
table. Also, the ‘ignorecase’ value should be 1 if you want to
ignore the case of alphabetic characters while matching/searching
column meta-data (see *note Input output options::).
For FITS tables, each column will be read independently. Therefore
they will be read in ‘numthreads’ CPU threads to greatly speed up
the reading when there are many columns and rows. However, this
only happens if CFITSIO was configured with ‘--enable-reentrant’.
This test has been done at Gnuastro's configuration time; if so,
‘GAL_CONFIG_HAVE_FITS_IS_REENTRANT’ will have a value of 1,
otherwise, it will have a value of 0. For more on this macro, see
*note Configuration information::). Multi-threaded table reading
is not currently applicable to other table formats (only for FITS
tables).
The output is an individually allocated list of datasets (see *note
List of gal_data_t::) with the same order of the ‘cols’ list. Note
that one column node in the ‘cols’ list might give multiple columns
(for example, from regular expressions), in this case, the order of
output columns that correspond to that one input, are in order of
the table (which column was read first). So the first requested
column is the first popped data structure and so on.
if ‘colmatch!=NULL’, it is assumed to be an array that has at least
the same number of elements as nodes in the ‘cols’ list. The
number of columns that matched each input column will be stored in
each element.
-- Function:
gal_list_sizet_t *
gal_table_list_of_indexs (gal_list_str_t *cols, gal_data_t
*allcols, size_t numcols, int searchin, int ignorecase, char
*filename, char *hdu, size_t *colmatch)
Returns a list of indices (starting from 0) of the input columns
that match the names/numbers given to ‘cols’. This is a low-level
operation which is called by ‘gal_table_read’ (described above),
see there for more on each argument's description. ‘allcols’ is
the returned array of ‘gal_table_info’.
-- Function:
void
gal_table_comments_add_intro (gal_list_str_t **comments, char
*program_string, time_t *rawtime)
Add some basic information to the list of ‘comments’. This basic
information includes the following information
• If the program is run in a Git version controlled directory,
Git's description is printed (see description under ‘COMMIT’
in *note Output FITS files::).
• The calendar time that is stored in ‘rawtime’ (‘time_t’ is C's
calendar time format defined in ‘time.h’). You can calculate
the time in this format with the following expressions:
time_t rawtime;
time(&rawtime);
• The name of your program in ‘program_string’. If it is
‘NULL’, this line is ignored.
-- Function:
void
gal_table_write (gal_data_t *cols, struct gal_fits_list_key_t
**keylist, gal_list_str_t *comments, int tableformat, char
*filename, char *extname, uint8_t colinfoinstdout, int
freekeys)
Write ‘cols’ (a list of datasets, see *note List of gal_data_t::)
into a table stored in ‘filename’. The format of the table can be
determined with ‘tableformat’ that accepts the macros defined
above. When ‘filename==NULL’, the column information will be
printed on the standard output (command-line).
If ‘comments!=NULL’, the list of comments (see *note List of
strings::) will also be printed into the output table. When the
output table is a plain text file, every node of ‘comments’ will be
printed after a ‘#’ (so it can be considered as a comment) and in
FITS table they will follow a ‘COMMENT’ keyword.
If a file named ‘filename’ already exists, the operation depends on
the type of output. When ‘filename’ is a FITS file, the table will
be added as a new extension after all existing extensions. If
‘filename’ is a plain text file, this function will abort with an
error.
If ‘filename’ is a FITS file, the table extension will have the
name ‘extname’.
When ‘colinfoinstdout!=0’ and ‘filename==NULL’ (columns are printed
in the standard output), the dataset metadata will also printed in
the standard output. When printing to the standard output, the
column information can be piped into another program for further
processing and thus the meta-data (lines starting with a ‘#’) must
be ignored. In such cases, you only print the column values by
passing ‘0’ to ‘colinfoinstdout’.
-- Function:
void
gal_table_write_log (gal_data_t *logll, char *program_string,
time_t *rawtime, gal_list_str_t *comments, char *filename, int
quiet)
Write the ‘logll’ list of datasets into a table in ‘filename’ (see
*note List of gal_data_t::). This function is just a wrapper
around ‘gal_table_comments_add_intro’ and ‘gal_table_write’ (see
above). If ‘quiet’ is non-zero, this function will print a message
saying that the ‘filename’ has been created.
-- Function:
gal_data_t *
gal_table_col_vector_extract (gal_data_t *vector,
gal_list_sizet_t *indexs)
Given the "vector" column ‘vector’ (which is assumed to be a 2D
dataset), extract the tokens that are identified in the ‘indexs’
list into a list of one dimensional datasets. For more on vector
columns in tables, see *note Vector columns::.
-- Function:
gal_data_t *
gal_table_cols_to_vector (gal_data_t *list)
Merge the one-dimensional datasets in the given list into one
2-dimensional dataset that can be treated as a vector column. All
the input datasets have to have the same size and type. For more
on vector columns in tables, see *note Vector columns::.
12.3.11 FITS files (‘fits.h’)
-----------------------------
The FITS format is the most common format to store data (images and
tables) in astronomy. The CFITSIO library already provides a very good
low-level collection of functions for manipulating FITS data. The
low-level nature of CFITSIO is defined for versatility and portability.
As a result, even a simple and basic operation, like reading an image or
table column into memory, will require a special sequence of CFITSIO
function calls which can be inconvenient and buggy to manage in separate
locations. To ease this process, Gnuastro's library provides wrappers
for CFITSIO functions. With these, it much easier to read, write, or
modify FITS file data, header keywords and extensions. Hence, if you
feel these functions do not exactly do what you want, we strongly
recommend reading the CFITSIO manual to use its great features directly
(afterwards, send us your wrappers so we can include it here for others
to benefit also).
All the functions and macros introduced in this section are declared
in ‘gnuastro/fits.h’. When you include this header, you are also
including CFITSIO's ‘fitsio.h’ header. So you do not need to explicitly
include ‘fitsio.h’ anymore and can freely use any of its macros or
functions in your code along with those discussed here.
12.3.11.1 FITS Macros, errors and filenames
...........................................
Some general constructs provided by Gnuastro's FITS handling functions
are discussed here. In particular there are several useful functions
about FITS file names.
-- Macro: GAL_FITS_MAX_NDIM
The maximum number of dimensions a dataset can have in FITS format,
according to the FITS standard this is 999.
-- Function:
void
gal_fits_io_error (int status, char *message)
If ‘status’ is non-zero, this function will print the CFITSIO error
message corresponding to status, print ‘message’ (optional) in the
next line and abort the program. If ‘message==NULL’, it will print
a default string after the CFITSIO error.
-- Function:
int
gal_fits_name_is_fits (char *name)
If the ‘name’ is an acceptable CFITSIO FITS filename return ‘1’
(one), otherwise return ‘0’ (zero). The currently acceptable FITS
suffixes are ‘.fits’, ‘.fit’, ‘.fits.gz’, ‘.fits.Z’, ‘.imh’,
‘.fits.fz’. IMH is the IRAF format which is acceptable to CFITSIO.
-- Function:
int
gal_fits_suffix_is_fits (char *suffix)
Similar to ‘gal_fits_name_is_fits’, but only for the suffix. The
suffix does not have to start with '<.>': this function will return
‘1’ (one) for both ‘fits’ and ‘.fits’.
-- Function:
int
gal_fits_file_recognized (char *name)
Return ‘1’ if the given file name (possibly including its contents)
is a FITS file. This is necessary when the contents of a FITS file
do follow the FITS standard, but the file does not have a
Gnuastro-recognized FITS suffix. Therefore, it will first call
‘gal_fits_name_is_fits’, if the result is negative, then this
function will attempt to open the file with CFITSIO and if it
works, it will close it again and return 1. In the process of
opening the file, CFITSIO will just open the file, and no reading
will take place, so it should have a minimal CPU footprint.
-- Function:
char *
gal_fits_name_save_as_string (char *filename, char *hdu)
If the name is a FITS name, then put a ‘(hdu: ...)’ after it and
return the string. If it is not a FITS file, just print the name,
if ‘filename==NULL’, then return the string ‘stdin’. Note that the
output string's space is allocated.
This function is useful when you want to report a random file to
the user which may be FITS or not (for a FITS file, simply the
filename is not enough, the HDU is also necessary).
12.3.11.2 CFITSIO and Gnuastro types
....................................
Both Gnuastro and CFITSIO have special and different identifiers for
each type that they accept. Gnuastro's type identifiers are fully
described in *note Library data types:: and are usable for all kinds of
datasets (images, table columns, etc) as part of Gnuastro's *note
Generic data container::. However, following the FITS standard, CFITSIO
has different identifiers for images and tables. Following CFITSIO's
own convention, we will use ‘bitpix’ for image type identifiers and
‘datatype’ for its internal identifiers (and mainly used in tables).
The functions introduced in this section can be used to convert between
CFITSIO and Gnuastro's type identifiers.
One important issue to consider is that CFITSIO's types are not fixed
width (for example, ‘long’ may be 32-bits or 64-bits on different
systems). However, Gnuastro's types are defined by their width. These
functions will use information on the host system to do the proper
conversion. To have a portable (usable on different systems) code, is
thus recommended to use these functions and not to assume a fixed
correspondence between CFITSIO and Gnuastro's types.
-- Function:
uint8_t
gal_fits_bitpix_to_type (int bitpix)
Return the Gnuastro type identifier that corresponds to CFITSIO's
‘bitpix’ on this system.
-- Function:
int
gal_fits_type_to_bitpix (uint8_t type)
Return the CFITSIO ‘bitpix’ value that corresponds to Gnuastro's
‘type’.
-- Function:
char
gal_fits_type_to_bin_tform (uint8_t type)
Return the FITS standard binary table ‘TFORM’ character that
corresponds to Gnuastro's ‘type’.
-- Function:
int
gal_fits_type_to_datatype (uint8_t type)
Return the CFITSIO ‘datatype’ that corresponds to Gnuastro's ‘type’
on this machine.
-- Function:
uint8_t
gal_fits_datatype_to_type (int datatype, int is_table_column)
Return Gnuastro's type identifier that corresponds to the CFITSIO
‘datatype’. Note that when dealing with CFITSIO's ‘TLONG’, the
fixed width type differs between tables and images. So if the
corresponding dataset is a table column, put a non-zero value into
‘is_table_column’.
12.3.11.3 FITS HDUs
...................
A FITS file can contain multiple HDUs/extensions. The functions in this
section can be used to get basic information about the extensions or
open them. Note that ‘fitsfile’ is defined in CFITSIO's ‘fitsio.h’
which is automatically included by Gnuastro's ‘gnuastro/fits.h’.
-- Function:
fitsfile *
gal_fits_open_to_write (char *filename)
If ‘filename’ exists, open it and return the ‘fitsfile’ pointer
that corresponds to it. If ‘filename’ does not exist, the file
will be created which contains a blank first extension and the
pointer to its next extension will be returned.
-- Function:
size_t
gal_fits_hdu_num (char *filename)
Return the number of HDUs/extensions in ‘filename’.
-- Function:
unsigned long
gal_fits_hdu_datasum (char *filename, char *hdu, char
*hdu_option_name)
Return the ‘DATASUM’ of the given HDU in the given FITS file. For
more on ‘DATASUM’ in the FITS standard, see *note Keyword
inspection and manipulation:: (under the ‘checksum’ component of
‘--write’). For more on ‘hdu_option_name’ see the description of
‘gal_array_read’ in *note Array input output::.
-- Function:
unsigned long
gal_fits_hdu_datasum_encoded (char *filename, char *hdu, char
*hdu_option_name)
Similar to ‘gal_fits_hdu_datasum’, but the returned value is always
a 16-character string following the encoding that is described in
the FITS standard (primarily for the ‘CHECKSUM’ keyword, but can
also be used for ‘DATASUM’.
-- Function:
unsigned long
gal_fits_hdu_datasum_ptr (fitsfile *fptr)
Return the ‘DATASUM’ of the already opened HDU in ‘fptr’. For more
on ‘DATASUM’ in the FITS standard, see *note Keyword inspection and
manipulation:: (under the ‘checksum’ component of ‘--write’).
-- Function:
int
gal_fits_hdu_format (char *filename, char *hdu, char
*hdu_option_name)
Return the format of the HDU as one of CFITSIO's recognized macros:
‘IMAGE_HDU’, ‘ASCII_TBL’, or ‘BINARY_TBL’. For more on
‘hdu_option_name’ see the description of ‘gal_array_read’ in *note
Array input output::.
-- Function:
int
gal_fits_hdu_is_healpix (fitsfile *fptr)
Return ‘1’ if the dataset may be a HEALpix grid and ‘0’ otherwise.
Technically, it is considered to be a HEALPix if the HDU is not an
ASCII table, and has the ‘NSIDE’, ‘FIRSTPIX’ and ‘LASTPIX’.
-- Function:
fitsfile *
gal_fits_hdu_open (char *filename, char *hdu, int iomode, int
exitonerror, char *hdu_option_name)
Open the HDU/extension ‘hdu’ from ‘filename’ and return a pointer
to CFITSIO's ‘fitsfile’. ‘iomode’ determines how the FITS file
will be opened using CFITSIO's macros: ‘READONLY’ or ‘READWRITE’.
The string in ‘hdu’ will be appended to ‘filename’ in square
brackets so CFITSIO only opens this extension. You can use any
formatting for the ‘hdu’ that is acceptable to CFITSIO. See the
description under ‘--hdu’ in *note Input output options:: for more.
If ‘exitonerror!=0’ and the given HDU cannot be opened for any
reason, the function will exit the program, and print an
informative message. Otherwise, when the HDU cannot be opened, it
will just return a NULL pointer. For more on ‘hdu_option_name’ see
the description of ‘gal_array_read’ in *note Array input output::.
-- Function:
fitsfile *
gal_fits_hdu_open_format (char *filename, char *hdu, int
img0_tab1, char *hdu_option_name)
Open (in read-only format) the ‘hdu’ HDU/extension of ‘filename’ as
an image or table. When ‘img0_tab1’ is ‘0’(zero) but the HDU is a
table, this function will abort with an error. It will also abort
with an error when ‘img0_tab1’ is ‘1’ (one), but the HDU is an
image. For more on ‘hdu_option_name’ see the description of
‘gal_array_read’ in *note Array input output::.
A FITS HDU may contain both tables or images. When your program
needs one of these formats, you can call this function so if the
user provided the wrong HDU/file, it will abort and inform the user
that the file/HDU is has the wrong format.
12.3.11.4 FITS header keywords
..............................
Each FITS extension/HDU contains a raw dataset which can either be a
table or an image along with some header keywords. The keywords can be
used to store meta-data about the actual dataset. The functions in this
section describe Gnuastro's high-level functions for reading and writing
FITS keywords. Similar to all Gnuastro's FITS-related functions, these
functions are all wrappers for CFITSIO's low-level functions.
The necessary meta-data (header keywords) for a particular dataset
are commonly numerous, it is much more efficient to list them in one
variable and call the reading/writing functions once. Hence the
functions in this section use linked lists, a thorough introduction to
them is given in *note Linked lists::. To reading FITS keywords, these
functions use a list of Gnuastro's generic dataset format that is
discussed in *note List of gal_data_t::. To write FITS keywords we
define the ‘gal_fits_list_key_t’ node that is defined below.
-- Type (C struct): gal_fits_list_key_t
Structure for writing FITS keywords. This structure is used for
one keyword and you do not need to set all elements. With the
‘next’ element, you can link it to another keyword thus creating a
linked list to add any number of keywords easily and at any step
during your program (see *note Linked lists:: for an introduction
on lists). See the functions below for adding elements to the
list.
typedef struct gal_fits_list_key_t
{
int tfree; /* ==1, free title string. */
int kfree; /* ==1, free keyword name. */
int vfree; /* ==1, free keyword value. */
int cfree; /* ==1, free comment. */
int ufree; /* ==1, free unit. */
uint8_t type; /* Keyword value type. */
char *title; /* !=NULL, only print title.*/
char *keyname; /* Keyword Name. */
void *value; /* Keyword value. */
char *comment; /* Keyword comment. */
char *unit; /* Keyword unit. */
struct gal_fits_list_key_t *next; /* Pointer next keyword. */
} gal_fits_list_key_t;
-- Function:
int
gal_fits_key_exists_fptr (fitsfile *fptr, char *keyname)
Return 1 (true) if the opened FITS file pointer contains the
requested keyword and 0 (false) otherwise.
-- Function:
void *
gal_fits_key_img_blank (uint8_t type)
Returns a pointer to an allocated space containing the value to the
FITS ‘BLANK’ header keyword, when the input array has a type of
‘type’. This is useful when you want to write the ‘BLANK’ keyword
using CFITSIO's ‘fits_write_key’ function.
According to the FITS standard: "If the ‘BSCALE’ and ‘BZERO’
keywords do not have the default values of 1.0 and 0.0,
respectively, then the value of the ‘BLANK’ keyword must equal the
actual value in the FITS data array that is used to represent an
undefined pixel and not the corresponding physical value".
Therefore a special ‘BLANK’ value is needed for datasets containing
signed 8-bit, unsigned 16-bit, unsigned 32-bit, and unsigned 64-bit
integers (types that are defined with ‘BSCALE’ and ‘BZERO’ in the
FITS standard).
*Not usable when reading a dataset:* As quoted from the FITS
standard above, the value returned by this function can only be
generically used for the writing of the ‘BLANK’ keyword header. It
_must not_ be used as the blank pointer when when reading a FITS
array using CFITSIO. When reading an array with CFITSIO, you can
use ‘gal_blank_alloc_write’ to generate the necessary pointer.
-- Function:
void
gal_fits_key_clean_str_value (char *string)
Remove the single quotes and possible extra spaces around the
keyword values that CFITSIO returns when reading a string keyword.
CFITSIO does not remove the two single quotes around the string
value of a keyword. Hence the strings it reads are like: ‘'value
'’, or ‘'some_very_long_value'’. To use the value during your
processing, it is commonly necessary to remove the single quotes
(and possible extra spaces). This function will do this within the
allocated space of the string.
-- Function:
char *
gal_fits_key_date_to_struct_tm (char *fitsdate, struct tm *tp)
Parse ‘fitsdate’ as a FITS date format string (most generally:
‘YYYY-MM-DDThh:mm:ss.ddd...’) into the C library's broken-down time
structure, or ‘struct tm’ (declared in ‘time.h’) and return a
pointer to a newly allocated array for the sub-second part of the
format (‘.ddd...’). Therefore it needs to be freed afterwards (if
it is not ‘NULL’) When there is no sub-second portion, this pointer
will be ‘NULL’.
This is a relatively low-level function, an easier function to use
is ‘gal_fits_key_date_to_seconds’ which will return the sub-seconds
as double precision floating point.
Note that the FITS date format mentioned above is the most complete
representation. The following two formats are also acceptable:
‘YYYY-MM-DDThh:mm:ss’ and ‘YYYY-MM-DD’. This option can also
interpret the older FITS date format where only two characters are
given to the year and the date format is reversed
(‘DD/MM/YYThh:mm:ss.ddd...’). In this case (following the GNU C
Library), this option will make the following assumption: values 68
to 99 correspond to the years 1969 to 1999, and values 0 to 68 as
the years 2000 to 2068.
-- Function:
size_t
gal_fits_key_date_to_seconds (char *fitsdate, char
**subsecstr, double *subsec)
Return the Unix epoch time (number of seconds that have passed
since 00:00:00 Thursday, January 1st, 1970) corresponding to the
FITS date format string ‘fitsdate’ (see description of
‘gal_fits_key_date_to_struct_tm’ above). This function will return
‘GAL_BLANK_SIZE_T’ if the broken-down time could not be converted
to seconds.
The Unix epoch time is in units of seconds, but the FITS date
format allows sub-second accuracy. The last two arguments are for
the optional sub-second portion. If you do not want sub-second
information, just set the second argument to ‘NULL’.
If ‘fitsdate’ contains sub-second accuracy and ‘subsecstr!=NULL’,
then the starting of the sub-second part's string is stored in
‘subsecstr’ (malloc'ed), and ‘subsec’ will be the corresponding
numerical value (between 0 and 1, in double precision floating
point). So to avoid leaking memory, if a sub-second string is
requested, it must be freed after calling this function. When a
sub-second string does not exist (and it is requested), then a
value of ‘NULL’ and NaN will be written in ‘*subsecstr’ and
‘*subsec’ respectively.
This is a very useful function for operations on the FITS date
values, for example, sorting FITS files by their dates, or finding
the time difference between two FITS files. The advantage of
working with the Unix epoch time is that you do not have to worry
about calendar details (such as the number of days in different
months or leap years).
-- Function:
void
gal_fits_key_read_from_ptr (fitsfile *fptr, gal_data_t
*keysll, int readcomment, int readunit)
Read the list of keyword values from a FITS pointer. The input
should be a linked list of Gnuastro's generic data container
(‘gal_data_t’). Before calling this function, you just have to set
the ‘name’, and optionally, the desired ‘type’ of the value of each
keyword. The given ‘name’ value will be directly passed to CFITSIO
to read the desired keyword name. This function will allocate
space to keep the value. If no pre-defined type is requested for a
certain keyword's value, the smallest possible type to host the
value will be found and used. If ‘readcomment’ and ‘readunit’ are
non-zero, this function will also try to read the possible comments
and units of the keyword.
Here is one example of using this function:
/* Allocate an array of datasets. */
gal_data_t *keysll=gal_data_array_calloc(N);
/* Make the array usable as a list too (by setting `next'). */
for(i=0;i0) and used as
the keyword name. If ‘top1end0!=0’, then the keywords containing
the filename will be added to the top of the list.
The FITS standard sets a maximum length of 69 characters for the
string values of a keyword(1). This creates problems with file
names (which include directories) because file names/addresses can
become longer than the maximum number of characters in a FITS
keyword (around 70 characters). Therefore, when ‘filename’ is
longer than the maximum length of a FITS keyword value, this
function will break it into several keywords (breaking up the
string on directory separators). So the full file/directory
address (including directories) can be longer than 69 characters.
However, if a single file or directory name (within a larger
address) is longer than 69 characters, this function will truncate
the name and print a warning. If ‘quiet!=0’, then the warning will
not be printed.
-- Function:
void
gal_fits_key_write_wcsstr (fitsfile *fptr, struct wcsprm wcs,
char *wcsstr, int nkeyrec)
Write the WCS header string (produced with WCSLIB's ‘wcshdo’
function) into the CFITSIO ‘fitsfile’ pointer. ‘nkeyrec’ is the
number of FITS header keywords in ‘wcsstr’. This function will put
a few blank keyword lines along with a comment ‘WCS information’
before writing each keyword record.
-- Function:
void
gal_fits_key_write (gal_fits_list_key_t *keylist, char
*filename, char *hdu, char *hdu_option_name, int freekeys, int
create_fits_not_exists)
Write the list of keywords in ‘keylist’ into the ‘hdu’ extension of
the file called ‘filename’. If the file may not exist before this
function is activated, set ‘create_fits_not_exists’ to non-zero and
set the HDU to ‘"0"’. If the keywords should be freed after they
are written, set the ‘freekeys’ value to non-zero. For more on
‘hdu_option_name’ see the description of ‘gal_array_read’ in *note
Array input output::.
The list nodes are meant to be dynamically allocated (because they
will be freed after being written). We thus recommend using the
‘gal_fits_key_list_add’ or ‘gal_fits_key_list_add_end’ to create
and fill the list. Below is one fully working example of using
this function to write a keyword into an existing FITS file.
#include
#include
#include
int main()
{
char *filename="test.fits";
gal_fits_list_key_t *keylist=NULL;
char *unit="unit";
float value=123.456;
char *keyname="MYKEY";
char *comment="A good description of the key";
gal_fits_key_list_add_end(&keylist, GAL_TYPE_FLOAT32, keyname, 0,
&value, 0, comment, 0, unit, 0);
gal_fits_key_list_title_add(&keylist, "Matching metadata", 0);
gal_fits_key_write(keylist, filename, "1", "NONE", 1, 0);
return EXIT_SUCCESS;
}
-- Function:
void
gal_fits_key_write_in_ptr (gal_fits_list_key_t *keylist,
fitsfile *fptr, int freekeys)
Write the list of keywords in ‘keylist’ into the given CFITSIO
‘fitsfile’ pointer and free keylist. For more on the input
‘keylist’, see the description and example for
‘gal_fits_key_write’, above.
-- Function:
gal_list_str_t *
gal_fits_with_keyvalue (gal_list_str_t *files, char *hdu, char
*name, gal_list_str_t *values, char *hdu_option_name)
Given a list of FITS file names (‘files’), a certain HDU (‘hdu’), a
certain keyword name (‘name’), and a list of acceptable values
(‘values’), return the subset of file names where the requested
keyword name has one of the acceptable values. For more on
‘hdu_option_name’ see the description of ‘gal_array_read’ in *note
Array input output::.
-- Function:
gal_list_str_t *
gal_fits_unique_keyvalues (gal_list_str_t *files, char *hdu,
char *name, char *hdu_option_name)
Given a list of FITS file names (‘files’), a certain HDU (‘hdu’), a
certain keyword name (‘name’), return the list of unique values to
that keyword name in all the files. For more on ‘hdu_option_name’
see the description of ‘gal_array_read’ in *note Array input
output::.
---------- Footnotes ----------
(1) The limit is actually 71 characters (which is the full 80
character length, subtracted by 8 for the keyword name and one character
for the <=>). However, for strings, FITS also requires two single
quotes.
12.3.11.5 FITS arrays (images)
..............................
Images (or multi-dimensional arrays in general) are one of the common
data formats that is stored in FITS files. Only one image may be stored
in each FITS HDU/extension. The functions described here can be used to
get the information of, read, or write images in FITS files.
-- Function:
void
gal_fits_img_info (fitsfile *fptr, int *type, size_t *ndim,
size_t **dsize, char **name, char **unit)
Read the type (see *note Library data types::), number of
dimensions, and size along each dimension of the CFITSIO ‘fitsfile’
into the ‘type’, ‘ndim’, and ‘dsize’ pointers respectively. If
‘name’ and ‘unit’ are not ‘NULL’ (point to a ‘char *’), then if the
image has a name and units, the respective string will be put in
these pointers.
-- Function:
size_t *
gal_fits_img_info_dim (char *filename, char *hdu, size_t
*ndim, char *hdu_option_name)
Put the number of dimensions in the ‘hdu’ extension of ‘filename’
in the space that ‘ndim’ points to and return the size of the
dataset along each dimension as an allocated array with ‘*ndim’
elements. For more on ‘hdu_option_name’ see the description of
‘gal_array_read’ in *note Array input output::.
-- Function:
gal_data_t *
gal_fits_img_read (char *filename, char *hdu, size_t
minmapsize, int quietmmap, char *hdu_option_name)
Read the contents of the ‘hdu’ extension/HDU of ‘filename’ into a
Gnuastro generic data container (see *note Generic data
container::) and return it. If the necessary space is larger than
‘minmapsize’, then do not keep the data in RAM, but in a file on
the HDD/SSD. For more on ‘minmapsize’ and ‘quietmmap’ see the
description under the same name in *note Generic data container::.
For more on ‘hdu_option_name’ see the description of
‘gal_array_read’ in *note Array input output::.
Note that this function only reads the main data within the
requested FITS extension, the WCS will not be read into the
returned dataset. To read the WCS, you can use ‘gal_wcs_read’
function as shown below. Afterwards, the ‘gal_data_free’ function
will free both the dataset and any WCS structure (if there are
any).
data=gal_fits_img_read(filename, hdu, -1, 1, NULL);
data->wcs=gal_wcs_read(filename, hdu, 0, 0, 0, &data->wcs->nwcs,
NULL);
-- Function:
gal_data_t *
gal_fits_img_read_to_type (char *inputname, char *inhdu,
uint8_t type, size_t minmapsize, int quietmmap, char
*hdu_option_name)
Read the contents of the ‘hdu’ extension/HDU of ‘filename’ into a
Gnuastro generic data container (see *note Generic data
container::) of type ‘type’ and return it.
This is just a wrapper around ‘gal_fits_img_read’ (to read the
image/array of any type) and ‘gal_data_copy_to_new_type_free’ (to
convert it to ‘type’ and free the initially read dataset). See the
description there for more.
-- Function:
gal_data_t *
gal_fits_img_read_kernel (char *filename, char *hdu, size_t
minmapsize, int quietmmap, char *hdu_option_name)
Read the ‘hdu’ of ‘filename’ as a convolution kernel. A
convolution kernel must have an odd size along all dimensions, it
must not have blank (NaN in floating point types) values and must
be flipped around the center to make the proper convolution (see
*note Convolution process::). If there are blank values, this
function will change the blank values to ‘0.0’. If the input image
does not have the other two requirements, this function will abort
with an error describing the condition to the user. The finally
returned dataset will have a ‘float32’ type.
For more on ‘hdu_option_name’ see the description of
‘gal_array_read’ in *note Array input output::.
-- Function:
fitsfile *
gal_fits_img_write_to_ptr (gal_data_t *input, char *filename,
gal_fits_list_key_t *keylist, int freekeys)
Write the ‘input’ dataset into a FITS file named ‘filename’ and
return the corresponding CFITSIO ‘fitsfile’ pointer. This function
will not close ‘fitsfile’, so you can still add other extensions to
it after this function or make other modifications.
In case you want to add keywords into the HDU that contain the
data, you can use the second two arguments (see the description of
‘gal_fits_key_write’). These keywords will be written into the HDU
before writing the data: when there are more than roughly 5
keywords (assuming your dataset has WCS) and your dataset is large,
this can result in significant optimization of the running time
(because adding a keyword beyond the 36 key slots will cause the
whole data to shift for another block of 36 keywords).
-- Function:
void
gal_fits_img_write (gal_data_t *input, char *filename,
gal_fits_list_key_t *keylist, int freekeys)
Write the ‘input’ dataset into the FITS file named ‘filename’.
Also add the list of header keywords (‘keylist’) to the newly
created HDU/extension The list of keywords will be freed after
writing into the HDU, if you need them later, keep a separate copy
of the list before calling this function.
For the importance of why it is better to add your keywords in this
function (before writing the data) or after it, see the description
of ‘gal_fits_img_write_to_ptr’.
-- Function:
void
gal_fits_img_write_to_type (gal_data_t *input, char *filename,
gal_fits_list_key_t *keylist, int type, int freekeys)
Convert the ‘input’ dataset into ‘type’, then write it into the
FITS file named ‘filename’. Also add the ‘keylist’ keywords to the
newly created HDU/extension along with your program's name
(‘program_string’). After the FITS file is written, this function
will free the copied dataset (with type ‘type’) from memory.
For the importance of why it is better to add your keywords in this
function (before writing the data) or after it, see the description
of ‘gal_fits_img_write_to_ptr’. This is just a wrapper for the
‘gal_data_copy_to_new_type’ and ‘gal_fits_img_write’ functions.
-- Function:
void
gal_fits_img_write_corr_wcs_str (gal_data_t *input, char
*filename, char *wcsstr, int nkeyrec, double *crpix,
gal_fits_list_key_t *keylist, int freekeys)
Write the ‘input’ dataset into ‘filename’ using the ‘wcsstr’ while
correcting the ‘CRPIX’ values. For the importance of why it is
better to add your keywords in this function (before writing the
data) or after it, see the description of
‘gal_fits_img_write_to_ptr’.
This function is mainly useful when you want to make FITS files in
parallel (from one main WCS structure, with just a differing
CRPIX), for more on the arguments, see the description of
‘gal_fits_img_write’. This can happen in the following cases for
example:
• When a large number of FITS images (with WCS) need to be
created in parallel, it can be much more efficient to write
the header's WCS keywords once at first, write them in the
FITS file, then just correct the CRPIX values.
• WCSLIB's header writing function is not thread safe. So when
writing FITS images in parallel, we cannot write the header
keywords in each thread.
12.3.11.6 FITS tables
.....................
Tables are one of the common formats of data that is stored in FITS
files. Only one table may be stored in each FITS HDU/extension, but
each table column must be viewed as a different dataset (with its own
name, units and numeric data type for example). The only constraint of
the column datasets in a table is that they must be one-dimensional and
have the same number of elements as the other columns. The functions
described here can be used to get the information of, read, or write
columns into FITS tables.
-- Function:
void
gal_fits_tab_size (fitsfile *fitsptr, size_t *nrows, size_t
*ncols)
Read the number of rows and columns in the table within CFITSIO's
‘fitsptr’.
-- Function:
int
gal_fits_tab_format (fitsfile *fitsptr)
Return the format of the FITS table contained in CFITSIO's
‘fitsptr’. Recall that FITS tables can be in binary or ASCII
formats. This function will return ‘GAL_TABLE_FORMAT_AFITS’ or
‘GAL_TABLE_FORMAT_BFITS’ (defined in *note Table input output::).
If the ‘fitsptr’ is not a table, this function will abort the
program with an error message informing the user of the problem.
-- Function:
gal_data_t *
gal_fits_tab_info (char *filename, char *hdu, size_t *numcols,
size_t *numrows, int *tableformat, char *hdu_option_name)
Store the information of each column in ‘hdu’ of ‘filename’ into an
array of data structures with ‘numcols’ elements (one data
structure for each column) see *note Arrays of datasets::. The
total number of rows in the table is also put into the memory that
‘numrows’ points to. The format of the table (e.g., FITS binary or
ASCII table) will be put in ‘tableformat’ (macros defined in *note
Table input output::). For more on ‘hdu_option_name’ see the
description of ‘gal_array_read’ in *note Array input output::.
This function is just for column information. Therefore it only
stores meta-data like column name, units and comments. No actual
data (contents of the columns for example, the ‘array’ or ‘dsize’
elements) will be allocated by this function. This is a low-level
function particular to reading tables in FITS format. To be
generic, it is recommended to use ‘gal_table_info’ which will allow
getting information from a variety of table formats based on the
filename (see *note Table input output::).
-- Function:
gal_data_t *
gal_fits_tab_read (char *filename, char *hdu, size_t numrows,
gal_data_t *colinfo, gal_list_sizet_t *indexll, size_t
numthreads, size_t minmapsize, int quietmmap, char
*hdu_option_name)
Read the columns given in the list ‘indexll’ from a FITS table (in
‘filename’ and HDU/extension ‘hdu’) into the returned linked list
of data structures, see *note List of size_t:: and *note List of
gal_data_t::. For more on ‘hdu_option_name’ see the description of
‘gal_array_read’ in *note Array input output::.
Each column will be read independently, therefore they will be read
in ‘numthreads’ CPU threads to greatly speed up the reading when
there are many columns and rows. However, this only happens if
CFITSIO was configured with ‘--enable-reentrant’. This test has
been done at Gnuastro's configuration time; if so,
‘GAL_CONFIG_HAVE_FITS_IS_REENTRANT’ will have a value of 1,
otherwise, it will have a value of 0. For more on this macro, see
*note Configuration information::).
If the necessary space for each column is larger than ‘minmapsize’,
do not keep it in the RAM, but in a file in the HDD/SSD. For more
on ‘minmapsize’ and ‘quietmmap’, see the description under the same
name in *note Generic data container::.
Each column will have ‘numrows’ rows and ‘colinfo’ contains any
further information about the columns (returned by
‘gal_fits_tab_info’, described above). Note that this is a
low-level function, so the output data linked list is the inverse
of the input indexes linked list. It is recommended to use
‘gal_table_read’ for generic reading of tables, see *note Table
input output::.
-- Function:
void
gal_fits_tab_write (gal_data_t *cols, gal_list_str_t
*comments, int tableformat, char *filename, char *extname,
gal_fits_list_key_t *keywords, int freekeys)
Write the list of datasets in ‘cols’ (see *note List of
gal_data_t::) as separate columns in a FITS table in ‘filename’.
If ‘filename’ already exists then this function will write the
table as a new extension called ‘extname’, after all existing ones.
The format of the table (ASCII or binary) may be specified with the
‘tableformat’ (see *note Table input output::). If
‘comments!=NULL’, each node of the list of strings will be written
as a ‘COMMENT’ keywords in the output FITS file (see *note List of
strings::.
In case your table needs metadata keywords, you can use the
‘listkeys’ and ‘freekeys’. For more on these, see the description
of ‘gal_fits_key_write_in_ptr’.
This is a low-level function for tables. It is recommended to use
‘gal_table_write’ for generic writing of tables in a variety of
formats, see *note Table input output::.
12.3.12 File input output
-------------------------
The most commonly used file format in astronomical data analysis is the
FITS format (see *note Fits:: for an introduction), therefore Gnuastro's
library provides a large and separate collection of functions to
read/write data from/to them (see *note FITS files::). However, FITS is
not well recognized outside the astronomical community and cannot be
imported into documents or slides. Therefore, in this section, we
discuss the other different file formats that Gnuastro's library
recognizes.
12.3.12.1 Text files (‘txt.h’)
..............................
The most universal and portable format for data storage are plain text
files. They can be viewed and edited on any text editor or even on the
command-line. This section are describes some functions that help in
reading from and writing to plain text files.
Lines are one of the most basic building blocks (delimiters) of a
text file. Some operating systems like Microsoft Windows, terminate
their ASCII text lines with a carriage return character and a new-line
character (two characters, also known as CRLF line terminators). While
Unix-like operating systems just use a single new-line character. The
functions below that read an ASCII text file are able to identify lines
with both kinds of line terminators.
Gnuastro defines a simple format for metadata of table columns in a
plain text file that is discussed in *note Gnuastro text table format::.
The functions to get information from, read from and write to plain text
files also follow those conventions.
-- Macro: GAL_TXT_LINESTAT_INVALID
-- Macro: GAL_TXT_LINESTAT_BLANK
-- Macro: GAL_TXT_LINESTAT_COMMENT
-- Macro: GAL_TXT_LINESTAT_DATAROW
Status codes for lines in a plain text file that are returned by
‘gal_txt_line_stat’. Lines which have a <#> character as their
first non-white character are considered to be comments. Lines
with nothing but white space characters are considered blank. The
remaining lines are considered as containing data.
-- Function:
int
gal_txt_line_stat (char *line)
Check the contents of ‘line’ and see if it is a blank, comment, or
data line. The returned values are the macros that start with
‘GAL_TXT_LINESTAT’.
-- Function:
char *
gal_txt_trim_space (char *str)
Trim the white space characters before and after the given string.
The operation is done within the allocated space of the string, so
if you need the string untouched, please pass an allocated copy of
the string to this function. The returned pointer is within the
input string. If the input pointer is ‘NULL’, or the string only
has white-space characters, the returned pointer will be ‘NULL’.
-- Function:
int
gal_txt_contains_string (char *full, char *match)
Return 1 if the string that ‘match’ points to, can be exactly found
within the string that ‘full’ points to (character by character).
The to-match string can be in any part of the full string. If any
of the two strings have zero length or are a ‘NULL’ pointer, this
function will return 0.
-- Function:
gal_data_t *
gal_txt_table_info (char *filename, gal_list_str_t *lines,
size_t *numcols, size_t *numrows)
Store the information of each column in a text file ‘filename’, or
list of strings (‘lines’) into an array of data structures with
‘numcols’ elements (one data structure for each column) see *note
Arrays of datasets::. The total number of rows in the table is
also put into the memory that ‘numrows’ points to.
‘lines’ is a list of strings with each node representing one line
(including the new-line character), see *note List of strings::.
It will mostly be the output of ‘gal_txt_stdin_read’, which is used
to read the program's input as separate lines from the standard
input (see below). Note that ‘filename’ and ‘lines’ are mutually
exclusive and one of them must be ‘NULL’.
This function is just for column information. Therefore it only
stores meta-data like column name, units and comments. No actual
data (contents of the columns for example, the ‘array’ or ‘dsize’
elements) will be allocated by this function. This is a low-level
function particular to reading tables in plain text format. To be
generic, it is recommended to use ‘gal_table_info’ which will allow
getting information from a variety of table formats based on the
filename (see *note Table input output::).
-- Function:
gal_data_t *
gal_txt_table_read (char *filename, gal_list_str_t *lines,
size_t numrows, gal_data_t *colinfo, gal_list_sizet_t
*indexll, size_t minmapsize, int quietmmap)
Read the columns given in the list ‘indexll’ from a plain text file
(‘filename’) or list of strings (‘lines’), into a linked list of
data structures (see *note List of size_t:: and *note List of
gal_data_t::). If the necessary space for each column is larger
than ‘minmapsize’, do not keep it in the RAM, but in a file on the
HDD/SSD. For more one ‘minmapsize’ and ‘quietmmap’, see the
description under the same name in *note Generic data container::.
‘lines’ is a list of strings with each node representing one line
(including the new-line character), see *note List of strings::.
It will mostly be the output of ‘gal_txt_stdin_read’, which is used
to read the program's input as separate lines from the standard
input (see below). Note that ‘filename’ and ‘lines’ are mutually
exclusive and one of them must be ‘NULL’.
Note that this is a low-level function, so the output data list is
the inverse of the input indices linked list. It is recommended to
use ‘gal_table_read’ for generic reading of tables in any format,
see *note Table input output::.
-- Function:
gal_data_t *
gal_txt_image_read (char *filename, gal_list_str_t *lines,
size_t minmapsize, int quietmmap)
Read the 2D plain text dataset in file (‘filename’) or list of
strings (‘lines’) into a dataset and return the dataset. If the
necessary space for the image is larger than ‘minmapsize’, do not
keep it in the RAM, but in a file on the HDD/SSD. For more on
‘minmapsize’ and ‘quietmmap’, see the description under the same
name in *note Generic data container::.
‘lines’ is a list of strings with each node representing one line
(including the new-line character), see *note List of strings::.
It will mostly be the output of ‘gal_txt_stdin_read’, which is used
to read the program's input as separate lines from the standard
input (see below). Note that ‘filename’ and ‘lines’ are mutually
exclusive and one of them must be ‘NULL’.
-- Function:
gal_list_str_t *
gal_txt_stdin_read (long timeout_microsec)
Read the complete standard input and return a list of strings with
each line (including the new-line character) as one node of that
list. If the standard input is already filled (for example,
connected to another program's output with a pipe), then this
function will parse the whole stream.
If Standard input is not pre-configured and the _first line_ is
typed/written in the terminal before ‘timeout_microsec’
micro-seconds, it will continue parsing until reaches an
end-of-file character ( after a new-line on the keyboard)
with no time limit. If nothing is entered before
‘timeout_microsec’ micro-seconds, it will return ‘NULL’.
All the functions that can read plain text tables will accept a
filename as well as a list of strings (intended to be the output of
this function for using Standard input). The reason for keeping
the standard input is that once something is read from the standard
input, it is hard to put it back. We often need to read a text
file several times: once to count how many columns it has and which
ones are requested, and another time to read the desired columns.
So it easier to keep it all in allocated memory and pass it on from
the start for each round.
-- Function:
gal_list_str_t *
gal_txt_read_to_list (char *filename)
Read the contents of the given plain-text file and put each word
(separated by a SPACE character, into a new node of the output
list. The order of nodes in the output is the same as the input.
Any new-line character at the end of a word is removed in the
output list.
-- Function:
void
gal_txt_write (gal_data_t *cols, struct gal_fits_list_key_t
**keylist, gal_list_str_t *comment, char *filename, uint8_t
colinfoinstdout, int tab0_img1, int freekeys)
Write ‘cols’ in a plain text file ‘filename’ (table when
‘tab0_img1==0’ and image when ‘tab0_img1==1’). ‘cols’ may have one
or two dimensions which determines the output:
1D
‘cols’ is treated as a column and a list of datasets (see
*note List of gal_data_t::): every node in the list is written
as one column in a table.
2D
‘cols’ is a two dimensional array, it cannot be treated as a
list (only one 2D array can currently be written to a text
file). So if ‘cols->next!=NULL’ the next nodes in the list
are ignored and will not be written.
This is a low-level function for tables. It is recommended to use
‘gal_table_write’ for generic writing of tables in a variety of
formats, see *note Table input output::.
It is possible to add two types of metadata to the printed table:
comments and keywords. Each string in the list given to ‘comments’
will be printed into the file as a separate line, starting with
‘#’. Keywords have a more specific and computer-parsable format
and are passed through ‘keylist’. Each keyword is also printed in
one line, but with the format below. Because of the various
components in a keyword, it is thus necessary to use the
‘gal_fits_list_key_t’ data structure. For more, see *note FITS
header keywords::.
# [key] NAME: VALUE / [UNIT] KEYWORD COMMENT.
If ‘filename’ already exists this function will abort with an error
and will not write over the existing file. Before calling this
function make sure if the file exists or not. If ‘comments!=NULL’,
a ‘#’ will be put at the start of each node of the list of strings
and will be written in the file before the column meta-data in
‘filename’ (see *note List of strings::).
When ‘filename==NULL’, the column information will be printed on
the standard output (command-line). When ‘colinfoinstdout!=0’ and
‘filename==NULL’ (columns are printed in the standard output), the
dataset metadata will also printed in the standard output. When
printing to the standard output, the column information can be
piped into another program for further processing and thus the
meta-data (lines starting with a ‘#’) must be ignored. In such
cases, you only print the column values by passing ‘0’ to
‘colinfoinstdout’.
12.3.12.2 TIFF files (‘tiff.h’)
...............................
Outside of astronomy, the TIFF standard is arguably the most commonly
used format to store high-precision data/images. Unlike FITS however,
the TIFF standard only supports images (not tables), but like FITS, it
has support for all standard data types (see *note Numeric data types::)
which is the primary reason other fields use it.
Another similarity of the TIFF and FITS standards is that TIFF
supports multiple images in one file. The TIFF standard calls each one
of these images (and their accompanying meta-data) a 'directory'
(roughly equivalent to the FITS extensions). Unlike FITS however, the
directories can only be identified by their number (counting from zero),
recall that in FITS you can also use the extension name to identify it.
The functions described here allow easy reading (and later writing)
of TIFF files within Gnuastro or for users of Gnuastro's libraries.
Currently only reading is supported, but if you are interested, please
get in touch with us.
-- Function:
int
gal_tiff_name_is_tiff (char *name)
Return ‘1’ if ‘name’ has a TIFF suffix. This can be used to make
sure that a given input file is TIFF. See ‘gal_tiff_suffix_is_tiff’
for a list of recognized suffixes.
-- Function:
int
gal_tiff_suffix_is_tiff (char *name)
Return ‘1’ if ‘suffix’ is a recognized TIFF suffix. The recognized
suffixes are ‘tif’, ‘tiff’, ‘TIFF’ and ‘TIFF’.
-- Function:
size_t
gal_tiff_dir_string_read (char *string)
Return the number within ‘string’ as a ‘size_t’ number to identify
a TIFF directory. Note that the directories start counting from
zero.
-- Function:
gal_data_t *
gal_tiff_read (char *filename, size_t dir, size_t minmapsize,
int quietmmap)
Read the ‘dir’ directory within the TIFF file ‘filename’ and return
the contents of that TIFF directory as ‘gal_data_t’. If the
directory's image contains multiple channels, the output will be a
list (see *note List of gal_data_t::).
-- Function:
void
gal_tiff_write (gal_data_t *in, char *filename, int widthinpx,
int heightinpix, int bitspersample, int numimg)
Write the given dataset (‘in’) into ‘filename’ (a TIFF file) with
the specified image width in pixels (‘widthinpix’),height in pixels
(‘heightinpix’), bits per sample (‘bitspersample’), and number of
images (‘numimg’).
12.3.12.3 JPEG files (‘jpeg.h’)
...............................
The JPEG file format is one of the most common formats for storing and
transferring images, recognized by almost all image rendering and
processing programs. In particular, because of its lossy compression
algorithm, JPEG files can have low volumes, making it used heavily on
the internet. For more on this file format, and a comparison with
others, please see *note Recognized file formats::.
For scientific purposes, the lossy compression and very limited
dynamic range (8-bit integers) make JPEG very unattractive for storing
of valuable data. However, because of its commonality, it will
inevitably be needed in some situations. The functions here can be used
to read and write JPEG images into Gnuastro's *note Generic data
container::. If the JPEG file has more than one color channel, each
channel is treated as a separate node in a list of datasets (see *note
List of gal_data_t::).
-- Function:
int
gal_jpeg_name_is_jpeg (char *name)
Return ‘1’ if ‘name’ has a JPEG suffix. This can be used to make
sure that a given input file is JPEG. See ‘gal_jpeg_suffix_is_jpeg’
for a list of recognized suffixes.
-- Function:
int
gal_jpeg_suffix_is_jpeg (char *name)
Return ‘1’ if ‘suffix’ is a recognized JPEG suffix. The recognized
suffixes are ‘.jpg’, ‘.JPG’, ‘.jpeg’, ‘.JPEG’, ‘.jpe’, ‘.jif’,
‘.jfif’ and ‘.jfi’.
-- Function:
gal_data_t *
gal_jpeg_read (char *filename, size_t minmapsize, int
quietmmap)
Read the JPEG file ‘filename’ and return the contents as
‘gal_data_t’. If the directory's image contains multiple
colors/channels, the output will be a list with one node per
color/channel (see *note List of gal_data_t::).
-- Function:
void
gal_jpeg_write (gal_data_t *in, char *filename, uint8_t
quality, float widthincm)
Write the given dataset (‘in’) into ‘filename’ (a JPEG file). If
‘in’ is a list, then each node in the list will be a color channel,
therefore there can only be 1, 3 or 4 nodes in the list. If the
number of nodes is different, then this function will abort the
program with a message describing the cause. The lossy JPEG
compression level can be set through ‘quality’ which is a value
between 0 and 100 (inclusive, 100 being the best quality). The
display width of the JPEG file in units of centimeters (to suggest
to viewers/users, only a meta-data) can be set through ‘widthincm’.
12.3.12.4 EPS files (‘eps.h’)
.............................
The Encapsulated PostScript (EPS) format is commonly used to store
images (or individual/single-page parts of a document) in the PostScript
documents. For a more complete introduction, please see *note
Recognized file formats::. To provide high quality graphics, the
Postscript language is a vectorized format, therefore pixels (elements
of a "rasterized" format) are not defined in their context.
To display rasterized images, PostScript does allow arrays of pixels.
However, since the over-all EPS file may contain many vectorized
elements (for example, borders, text, or other lines over the text) and
interpreting them is not trivial or necessary within Gnuastro's scope,
Gnuastro only provides some functions to write a dataset (in the
‘gal_data_t’ format, see *note Generic data container::) into EPS.
-- Macro: GAL_EPS_MARK_COLNAME_TEXT
-- Macro: GAL_EPS_MARK_COLNAME_FONT
-- Macro: GAL_EPS_MARK_COLNAME_XPIX
-- Macro: GAL_EPS_MARK_COLNAME_YPIX
-- Macro: GAL_EPS_MARK_COLNAME_SHAPE
-- Macro: GAL_EPS_MARK_COLNAME_COLOR
-- Macro: GAL_EPS_MARK_COLNAME_SIZE1
-- Macro: GAL_EPS_MARK_COLNAME_SIZE2
-- Macro: GAL_EPS_MARK_COLNAME_ROTATE
-- Macro: GAL_EPS_MARK_COLNAME_FONTSIZE
-- Macro: GAL_EPS_MARK_COLNAME_LINEWIDTH
Name of column that the required property will be read from.
-- Macro: GAL_EPS_MARK_DEFAULT_SHAPE
-- Macro: GAL_EPS_MARK_DEFAULT_COLOR
-- Macro: GAL_EPS_MARK_DEFAULT_SIZE1
-- Macro: GAL_EPS_MARK_DEFAULT_SIZE2
-- Macro: GAL_EPS_MARK_DEFAULT_SIZE2_ELLIPSE
-- Macro: GAL_EPS_MARK_DEFAULT_ROTATE
-- Macro: GAL_EPS_MARK_DEFAULT_LINEWIDTH
-- Macro: GAL_EPS_MARK_DEFAULT_FONT
-- Macro: GAL_EPS_MARK_DEFAULT_FONTSIZE
Default values for the various mark properties. These constants
will be used if the caller has not provided any of the given
property.
-- Function:
int
gal_eps_name_is_eps (char *name)
Return ‘1’ if ‘name’ has an EPS suffix. This can be used to make
sure that a given input file is EPS. See ‘gal_eps_suffix_is_eps’
for a list of recognized suffixes.
-- Function:
int
gal_eps_suffix_is_eps (char *name)
Return ‘1’ if ‘suffix’ is a recognized EPS suffix. The recognized
suffixes are ‘.eps’, ‘.EPS’, ‘.epsf’, ‘.epsi’.
-- Function:
void
gal_eps_to_pt (float widthincm, size_t *dsize, size_t
*w_h_in_pt)
Given a specific width in centimeters (‘widthincm’ and the number
of he dataset's pixels in each dimension (‘dsize’) calculate the
size of he output in PostScript points. The output values are
written in the ‘w_h_in_pt’ array (which has to be allocated before
calling this unction). The first element in ‘w_h_in_pt’ is the
width and the second is the height of the image.
-- Function:
uint8_t
gal_eps_shape_name_to_id (char *name)
Return the shape ID of a mark from its name (which is not
case-sensitive).
-- Function:
uint8_t
gal_eps_shape_id_to_name (uint8_t id)
Return the shape name from its ID.
-- Function:
void
gal_eps_write (gal_data_t *in, char *filename, float
widthincm, uint32_t borderwidth, uint8_t bordercolor, int hex,
int dontoptimize, int forps, gal_data_t *marks)
Write the ‘in’ dataset into an EPS file called ‘filename’. ‘in’
has to be an unsigned 8-bit character type ‘GAL_TYPE_UINT8’, see
*note Numeric data types::). The desired width of the image in
human/non-pixel units can be set with he ‘widthincm’ argument. If
‘borderwidth’ is non-zero, it is interpreted as the width (in
points) of a solid black border around the mage. A border can
helpful when importing the EPS file into a document. The color of
the border can be set with ‘bordercolor’, use the macros in *note
Color functions::. If ‘forpdf’ is not zero, the output can be
imported into a Postscript file directly (not as an "encapsulated"
postscript, which is the default).
EPS files are plain-text (can be opened/edited in a text editor),
therefore there are different encodings to store the data (pixel
values) within them. Gnuastro supports the Hexadecimal and ASCII85
encoding. ASCII85 is more efficient (producing small file sizes),
so it is the default encoding. To use Hexadecimal encoding, set
‘hex’ to a non-zero value.
By default, when the dataset only has two values, this function
will use the PostScript optimization that allows setting the pixel
values per bit, not byte (*note Recognized file formats::). This
can greatly help reduce the file size. However, when
‘dontoptimize!=0’, this optimization is disabled: even though there
are only two values (is binary), the difference between them does
not correspond to the full contrast of black and white.
If ‘marks!=NULL’, it is assumed to contain multiple columns of
information to draw marks over the background image. The multiple
columns are a linked list of 1D ‘gal_data_t’ of the same size
(number of rows) that are connected to each other through the
‘next’ element (this is the same format that Gnuastro's library
uses for tables, see *note Table input output:: or *note Library
demo - reading and writing table columns::).
The macros defined above that have the format of
‘GAL_EPS_MARK_COLNAME_*’ show all the possible columns that you can
provide in this linked list. Only the two coordinate columns are
mandatory (‘GAL_EPS_MARK_COLNAME_XPIX’ and
‘GAL_EPS_MARK_COLNAME_YPIX’. If any of the other properties is not
in the linked list, then the default properties of the
‘GAL_EPS_MARK_DEFAULT_*’ macros will be used (also defined above.
The columns are identified based on the ‘name’ element of
Gnuastro's generic data structure (see *note Generic data
container::). The names must have the pre-defined names of the
‘GAL_EPS_MARK_COLNAME_*’ macros (case sensitive). Therefore, the
order of columns in the list is irrelevant!
12.3.12.5 PDF files (‘pdf.h’)
.............................
The portable document format (PDF) has arguably become the most common
format used for distribution of documents. In practice, a PDF file is
just a compiled PostScript file. For a more complete introduction,
please see *note Recognized file formats::. To provide high quality
graphics, the PDF is a vectorized format, therefore pixels (elements of
a "rasterized" format) are not defined in their context. As a result,
similar to *note EPS files::, Gnuastro only writes datasets to a PDF
file, not vice-versa.
-- Function:
int
gal_pdf_name_is_pdf (char *name)
Return ‘1’ if ‘name’ has an PDF suffix. This can be used to make
sure that a given input file is PDF. See ‘gal_pdf_suffix_is_pdf’
for a list of recognized suffixes.
-- Function:
int
gal_pdf_suffix_is_pdf (char *name)
Return ‘1’ if ‘suffix’ is a recognized PDF suffix. The recognized
suffixes are ‘.pdf’ and ‘.PDF’.
-- Function:
void
gal_pdf_write (gal_data_t *in, char *filename, float
widthincm, uint32_t borderwidth, uint8_t bordercolor, int
dontoptimize, gal_data_t *marks)
Write the ‘in’ dataset into an EPS file called ‘filename’. ‘in’
has to be an unsigned 8-bit character type (‘GAL_TYPE_UINT8’, see
*note Numeric data types::). The desired width of the image in
human/non-pixel units can be set with the ‘widthincm’ argument. If
‘borderwidth’ is non-zero, it is interpreted as the width (in
points) of a solid black border around the image. A border can
helpful when importing the PDF file into a document. The color of
the border can be set with ‘bordercolor’, use the macros in *note
Color functions::.
This function is just a wrapper for the ‘gal_eps_write’ function in
*note EPS files::. After making the EPS file, Ghostscript (with a
version of 9.10 or above, see *note Optional dependencies::) will
be used to compile the EPS file to a PDF file. Therefore if
Ghostscript does not exist, does not have the proper version, or
fails for any other reason, the EPS file will remain. It can be
used to find the cause, or use another converter or PostScript
compiler.
By default, when the dataset only has two values, this function
will use the PostScript optimization that allows setting the pixel
values per bit,not byte (*note Recognized file formats::). This
can greatly help reduce the file size. However, when
‘dontoptimize!=0’, this optimization is disabled: even though there
are only two values (is binary), the difference between them does
not correspond to the full contrast of black and white.
If ‘marks!=NULL’, it is assumed to contain information on how to
draw marks over the image. This is directly fed to the
‘gal_eps_write’ function, so for more on how to provide the mark
information, see the description of ‘gal_eps_write’ in *note EPS
files::.
12.3.13 World Coordinate System (‘wcs.h’)
-----------------------------------------
The FITS standard defines the world coordinate system (WCS) as a
mechanism to associate physical values to positions within a dataset.
For example, it can be used to convert pixel coordinates in an image to
celestial coordinates like the right ascension and declination. The
functions in this section are mainly just wrappers over CFITSIO, WCSLIB
and GSL library functions to help in common applications.
[*Tread safety*] Since WCSLIB version 5.18 (released in January
2018), most WCSLIB functions are thread safe(1). Gnuastro has
high-level functions to easily spin-off threads and speed up your
programs. For a fully working example see *note Library demo -
multi-threaded operation::. However you still need to be cautious in
the following scenarios below.
• Many users or operating systems may still use an older version.
• The ‘wcsprm’ structure of WCSLIB is not thread-safe: you can't use
the same pointer on multiple threads. For example, if you use
‘gal_wcs_img_to_world’ simultaneously on multiple threads, you
shouldn't pass the same ‘wcsprm’ structure pointer. You can use
‘gal_wcs_copy’ to keep and use separate copies the main structure
within each thread, and later free the copies with ‘gal_wcs_free’.
The full set of functions and global constants that are defined by
Gnuastro's ‘gnuastro/wcs.h’ are described below.
-- Global integer: GAL_WCS_DISTORTION_TPD
-- Global integer: GAL_WCS_DISTORTION_SIP
-- Global integer: GAL_WCS_DISTORTION_TPV
-- Global integer: GAL_WCS_DISTORTION_DSS
-- Global integer: GAL_WCS_DISTORTION_WAT
-- Global integer: GAL_WCS_DISTORTION_INVALID
Gnuastro identifiers of the various WCS distortion conventions, for
more, see Calabretta et al. (2004, preprint)(2). Among these, SIP
is a prior distortion, the rest other are sequent distortions. TPD
is a superset of all these, hence it has both prior and sequeal
distortion coefficients. More information is given in the
documentation of ‘dis.h’, from the WCSLIB manual(3).
-- Global integer: GAL_WCS_COORDSYS_EQB1950
-- Global integer: GAL_WCS_COORDSYS_EQJ2000
-- Global integer: GAL_WCS_COORDSYS_ECB1950
-- Global integer: GAL_WCS_COORDSYS_ECJ2000
-- Global integer: GAL_WCS_COORDSYS_GALACTIC
-- Global integer: GAL_WCS_COORDSYS_SUPERGALACTIC
-- Global integer: GAL_WCS_COORDSYS_INVALID
Recognized WCS coordinate systems in Gnuastro. ‘EQ’ and ‘EC’ stand
for the EQuatorial and ECliptic coordinate systems. In the
equatorial and ecliptic coordinates, ‘B1950’ stands for the
Besselian 1950 epoch and ‘J2000’ stands for the Julian 2000 epoch.
-- Global integer: GAL_WCS_LINEAR_MATRIX_PC
-- Global integer: GAL_WCS_LINEAR_MATRIX_CD
-- Global integer: GAL_WCS_LINEAR_MATRIX_INVALID
Identifiers of the linear transformation matrix: either in the
‘PCi_j’ or the ‘CDi_j’ formalism. For more, see the description of
‘--wcslinearmatrix’ in *note Input output options::.
-- Global integer: GAL_WCS_PROJECTION_AZP
-- Global integer: GAL_WCS_PROJECTION_SZP
-- Global integer: GAL_WCS_PROJECTION_TAN
-- Global integer: GAL_WCS_PROJECTION_STG
-- Global integer: GAL_WCS_PROJECTION_SIN
-- Global integer: GAL_WCS_PROJECTION_ARC
-- Global integer: GAL_WCS_PROJECTION_ZPN
-- Global integer: GAL_WCS_PROJECTION_ZEA
-- Global integer: GAL_WCS_PROJECTION_AIR
-- Global integer: GAL_WCS_PROJECTION_CYP
-- Global integer: GAL_WCS_PROJECTION_CEA
-- Global integer: GAL_WCS_PROJECTION_CAR
-- Global integer: GAL_WCS_PROJECTION_MER
-- Global integer: GAL_WCS_PROJECTION_SFL
-- Global integer: GAL_WCS_PROJECTION_PAR
-- Global integer: GAL_WCS_PROJECTION_MOL
-- Global integer: GAL_WCS_PROJECTION_AIT
-- Global integer: GAL_WCS_PROJECTION_COP
-- Global integer: GAL_WCS_PROJECTION_COE
-- Global integer: GAL_WCS_PROJECTION_COD
-- Global integer: GAL_WCS_PROJECTION_COO
-- Global integer: GAL_WCS_PROJECTION_BON
-- Global integer: GAL_WCS_PROJECTION_PCO
-- Global integer: GAL_WCS_PROJECTION_TSC
-- Global integer: GAL_WCS_PROJECTION_CSC
-- Global integer: GAL_WCS_PROJECTION_QSC
-- Global integer: GAL_WCS_PROJECTION_HPX
-- Global integer: GAL_WCS_PROJECTION_XPH
The various types of recognized FITS WCS projections; for more
details see *note Align pixels with WCS considering distortions::.
-- Macro: GAL_WCS_FLTERROR
Limit of rounding for floating point errors.
-- Function:
int
gal_wcs_distortion_name_to_id (char *name)
Convert the given string (assumed to be a FITS-standard,
string-based distortion identifier) to a Gnuastro's integer-based
distortion identifier (one of the ‘GAL_WCS_DISTORTION_*’ macros
defined above). The sting-based distortion identifiers have three
characters and are all in capital letters.
-- Function:
int
gal_wcs_distortion_name_from_id (int id)
Convert the given Gnuastro integer-based distortion identifier (one
of the ‘GAL_WCS_DISTORTION_*’ macros defined above) to the
string-based distortion identifier) of the FITS standard. The
sting-based distortion identifiers have three characters and are
all in capital letters.
-- Function:
int
gal_wcs_coordsys_name_to_id (char *name)
Convert the given string to Gnuastro's integer-based WCS coordinate
system identifier (one of the ‘GAL_WCS_COORDSYS_*’, listed above).
The expected strings can be seen in the description of the
‘--wcscoordsys’ option of the Fits program, see *note Keyword
inspection and manipulation::.
-- Function:
int
gal_wcs_distortion_name_to_id (char *name)
Convert the given string (assumed to be a FITS-standard,
string-based distortion identifier) to a Gnuastro's integer-based
distortion identifier (one of the ‘GAL_WCS_DISTORTION_*’ macros
defined above). The sting-based distortion identifiers have three
characters and are all in capital letters.
-- Function:
int
gal_wcs_projection_name_from_id (int id)
Convert the given Gnuastro integer-based projection identifier (one
of the ‘GAL_WCS_PROJECTION_*’ macros defined above) to the
string-based distortion identifier) of the FITS standard. The
string-based projection identifiers have three characters and are
all in capital letters. For a description of the various
projections, see *note Align pixels with WCS considering
distortions::.
-- Function:
int
gal_wcs_projection_name_to_id (char *name)
Convert the given string (assumed to be a FITS-standard,
string-based projection identifier) to a Gnuastro's integer-based
projection identifier (one of the ‘GAL_WCS_PROJECTION_*’ macros
defined above). The string-based projection identifiers have three
characters and are all in capital letters. For a description of
the various projections, see *note Align pixels with WCS
considering distortions::.
-- Function:
struct wcsprm *
gal_wcs_create (double *crpix, double *crval, double *cdelt,
double *pc, char **cunit, char **ctype, size_t ndim, int
linearmatrix)
Given all the most common standard components of the WCS standard,
construct a ‘struct wcsprm’, initialize and set it for future
processing. See the FITS WCS standard for more on these keywords.
All the arrays must have ‘ndim’ elements with them except for ‘pc’
which should have ‘ndim*ndim’ elements (a square matrix). Also,
‘cunit’ and ‘ctype’ are arrays of strings. If
‘GAL_WCS_LINEAR_MATRIX_CD’ is passed to ‘linearmatrix’ then the
output WCS structure will have a CD matrix (even though you have
given a PC and CDELT matrix as input to this function). Otherwise,
the output will have a PC and CDELT matrix (which is the
recommended format by WCSLIB).
#include
#include
#include
int
main(void)
{
int status;
size_t ndim=2;
struct wcsprm *wcs;
double crpix[]={50, 50};
double pc[]={-1, 0, 0, 1};
double cdelt[]={0.4, 0.4};
double crval[]={178.23, 36.98};
char *cunit[]={"deg", "deg"};
char *ctype[]={"RA---TAN", "DEC--TAN"};
int linearmatrix = GAL_WCS_LINEAR_MATRIX_PC;
/* Allocate and fill the 'wcsprm' structure. */
wcs=gal_wcs_create(crpix, crval, cdelt, pc, cunit,
ctype, ndim, linearmatrix);
printf("WCS structure created.\n");
/*... Add any operation with the WCS structure here ...*/
/* Free the WCS structure. */
gal_wcs_free(wcs);
printf("WCS structure freed.\n");
/* Return successfully. */
return EXIT_SUCCESS;
}
-- Function:
struct wcsprm *
gal_wcs_read_fitsptr (fitsfile *fptr, int linearmatrix, size_t
hstartwcs, size_t hendwcs, int *nwcs)
Return the WCSLIB ‘wcsprm’ structure that is read from the CFITSIO
‘fptr’ pointer to an opened FITS file. With older WCSLIB versions
(in particular below version 5.18) this function may not be
thread-safe.
Also put the number of coordinate representations found into the
space that ‘nwcs’ points to. To read the WCS structure directly
from a filename, see ‘gal_wcs_read’ below. After processing has
finished, you should free the WCS structure that this function
returns with ‘gal_wcs_free’.
The ‘linearmatrix’ argument takes one of three values: ‘0’,
‘GAL_WCS_LINEAR_MATRIX_PC’ and ‘GAL_WCS_LINEAR_MATRIX_CD’. It will
determine the format of the WCS when it is later written to file
with ‘gal_wcs_write’ or ‘gal_wcs_write_in_fitsptr’ (which is called
by ‘gal_fits_img_write’) So if you do not want to write the WCS
into a file later, just give it a value of ‘0’. For more on the
difference between these modes, see the description of
‘--wcslinearmatrix’ in *note Input output options::.
If you do not want to search the full FITS header for WCS-related
FITS keywords (for example, due to conflicting keywords), but only
a specific range of the header keywords you can use the ‘hstartwcs’
and ‘hendwcs’ arguments to specify the keyword number range
(counting from zero). If ‘hendwcs’ is larger than ‘hstartwcs’,
then only keywords in the given range will be checked. Hence, to
ignore this feature (and search the full FITS header), give both
these arguments the same value.
If the WCS information could not be read from the FITS file, this
function will return a ‘NULL’ pointer and put a zero in ‘nwcs’. A
WCSLIB error message will also be printed in ‘stderr’ if there was
an error.
This function is just a wrapper over WCSLIB's ‘wcspih’ function
which is not thread-safe. Therefore, be sure to not call this
function simultaneously (over multiple threads).
-- Function:
struct wcsprm *
gal_wcs_read (char *filename, char *hdu, int linearmatrix,
size_t hstartwcs, size_t hendwcs, int *nwcs, char
*hdu_option_name)
[*Not thread-safe*] Return the WCSLIB structure that is read from
the HDU/extension ‘hdu’ of the file ‘filename’. Also put the
number of coordinate representations found into the space that
‘nwcs’ points to. Please see ‘gal_wcs_read_fitsptr’ for more. For
more on ‘hdu_option_name’ see the description of ‘gal_array_read’
in *note Array input output::.
After processing has finished, you should free the WCS structure
that this function returns with ‘gal_wcs_free’.
-- Function:
void
gal_wcs_free (struct wcsprm *wcs)
Free the contents _and_ the space that ‘wcs’ points to. WCSLIB's
‘wcsfree’ function only frees the contents of the ‘wcsprm’
structure, not the actual pointer. However, Gnuastro's ‘wcsprm’
creation and reading functions allocate the structure also. This
higher-level function therefore simplifies the processing. A
complete working example is given in the description of
‘gal_wcs_create’.
-- Function:
char *
gal_wcs_dimension_name (struct wcsprm *wcs, size_t dimension)
Return an allocated string array (that should be freed later)
containing the first part of the ‘CTYPEi’ FITS keyword (which
contains the dimension name in the FITS standard). For example, if
‘CTYPE1’ is ‘RA---TAN’, the string that function returns will be
‘RA’. Recall that the second component of ‘CTYPEi’ contains the
type of projection.
-- Function:
char *
gal_wcs_write_wcsstr (struct wcsprm *wcs, int *nkeyrec)
Return an allocated string which contains the respective FITS
keywords for the given WCS structure into it. The number of
keywords is written in the space pointed by ‘nkeyrec’. Each FITS
keyword is 80 characters wide (according to the FITS standard), and
the next one is placed immediately after it, so the full string has
‘80*nkeyrec’ bytes. The output of this function can later be
written into an opened FITS file using ‘gal_fits_key_write_wcsstr’
(see *note FITS header keywords::).
-- Function:
void
gal_wcs_write (struct wcsprm *wcs, char *filename, char
*extname, gal_fits_list_key_t *keylist, int freekeys)
Write the given WCS structure into the second extension of an empty
FITS header. The first/primary extension will be empty like the
default format of all Gnuastro outputs. When ‘extname!=NULL’ it
will be used as the FITS extension name. Any set of extra headers
can also be written through the ‘keylist’ list. If ‘freekeys!=0’
then the list of keywords will be freed after they are written.
-- Function:
void
gal_wcs_write_in_fitsptr (fitsfile *fptr, struct wcsprm *wcs)
Convert the input ‘wcs’ structure (keeping the WCS
programmatically) into FITS keywords and write them into the given
FITS file pointer. This is a relatively low-level function which
assumes the FITS file has already been opened with CFITSIO. If you
just want to write the WCS into an empty file, you can use
‘gal_wcs_write’ (which internally calls this function after
creating the FITS file and later closes it safely).
-- Function:
struct wcsprm *
gal_wcs_copy (struct wcsprm *wcs)
Return a fully allocated (independent) copy of ‘wcs’.
-- Function:
struct wcsprm *
gal_wcs_copy_new_crval (struct wcsprm *wcs, double *crval)
Return a fully allocated (independent) copy of ‘wcs’ with a new set
of ‘CRVAL’ values. WCSLIB keeps a lot of extra information within
‘wcsprm’ and for optimizations, those extra information are used in
its calculations. Therefore, if you want to change parameters like
the reference point's sky coordinate values (‘CRVAL’), simply
changing the values in ‘wcs->crval[0]’ or ‘wcs->crval[1]’ will not
affect WCSLIB's calculations; you need to call this function.
-- Function:
void
gal_wcs_remove_dimension (struct wcsprm *wcs, size_t fitsdim)
Remove the given FITS dimension from the given ‘wcs’ structure.
-- Function:
void
gal_wcs_on_tile (gal_data_t *tile)
Create a WCSLIB ‘wcsprm’ structure for ‘tile’ using WCS parameters
of the tile's allocated block dataset, see *note Tessellation
library:: for the definition of tiles. If ‘tile’ already has a WCS
structure, this function will not do anything.
In many cases, tiles are created for internal/low-level processing.
Hence for performance reasons, when creating the tiles they do not
have any WCS structure. When needed, this function can be used to
add a WCS structure to each tile tile by copying the WCS structure
of its block and correcting the reference point's coordinates
within the tile.
-- Function:
double *
gal_wcs_warp_matrix (struct wcsprm *wcs)
Return the Warping matrix of the given WCS structure as an array of
double precision floating points. This will be the final matrix,
irrespective of the type of storage in the WCS structure. Recall
that the FITS standard has several methods to store the matrix.
The output is an allocated square matrix with each side equal to
the number of dimensions.
-- Function:
void
gal_wcs_clean_small_errors (struct wcsprm *wcs)
Errors can make small differences between the pixel-scale elements
(‘CDELT’) and can also lead to extremely small values in the ‘PC’
matrix. With this function, such errors will be "cleaned" as
follows: 1) if the maximum difference between the ‘CDELT’ elements
is smaller than the reference error, it will be set to the mean
value. When the FITS keyword ‘CRDER’ (optional) is defined it will
be used as a reference, if not the default value is
‘GAL_WCS_FLTERROR’. 2) If any of the PC elements differ from 0, 1
or -1 by less than ‘GAL_WCS_FLTERROR’, they will be rounded to the
respective value.
-- Function:
void
gal_wcs_decompose_pc_cdelt (struct wcsprm *wcs)
Decompose the ‘PCi_j’ and ‘CDELTi’ elements of ‘wcs’. According to
the FITS standard, in the ‘PCi_j’ WCS formalism, the rotation
matrix elements $m_{ij}$ are encoded in the ‘PCi_j’ keywords and
the scale factors are encoded in the ‘CDELTi’ keywords. There is
also another formalism (the ‘CDi_j’ formalism) which merges the two
into one matrix.
However, WCSLIB's internal operations are apparently done in the
‘PCi_j’ formalism. So its outputs are also all in that format by
default. When the input is a ‘CDi_j’, WCSLIB will still read the
matrix directly into the ‘PCi_j’ matrix and the ‘CDELTi’ values are
set to ‘1’ (one). This function is designed to correct such
issues: after it is finished, the ‘CDELTi’ values in ‘wcs’ will
correspond to the pixel scale, and the ‘PCi_j’ will correction show
the rotation.
-- Function:
void
gal_wcs_to_cd (struct wcsprm *wcs)
Make sure that the WCS structure's ‘PCi_j’ and ‘CDi_j’ keywords
have the same value and that the ‘CDELTi’ keywords have a value of
1.0. Also, set the ‘wcs->altlin=2’ (for the ‘CDi_j’ formalism).
With these changes ‘gal_wcs_write_in_fitsptr’ (and thus
‘gal_wcs_write’ and ‘gal_fits_img_write’ and its derivatives) will
have an output file in the format of ‘CDi_j’.
-- Function:
int
gal_wcs_coordsys_identify (struct wcsprm *wcs)
Read the given WCS structure and return its coordinate system as
one of Gnuastro's WCS coordinate system identifiers (the macros
‘GAL_WCS_COORDSYS_*’, listed above).
-- Function:
struct wcsprm *
gal_wcs_coordsys_convert (struct wcsprm *inwcs, int
coordsysid)
Return a newly allocated WCS structure with the ‘coordsysid’
coordinate system identifier. The Gnuastro WCS distortion
identifiers are defined in the ‘GAL_WCS_COORDSYS_*’ macros
mentioned above. Since the returned dataset is newly allocated, if
you do not need the original dataset after this, use the WCSLIB
library function ‘wcsfree’ to free the input, for example,
‘wcsfree(inwcs)’.
-- Function:
void
gal_wcs_coordsys_convert_points (int sys1, double *lng1_d,
double *lat1_d, int sys2, double *lng2_d, double *lat2_d,
size_t number)
Convert the input set of longitudes (‘lng1_d’, in degrees) and
latitudes (‘lat1_d’, in degrees) within a recognized coordinate
system (‘sys1’; one of the ‘GAL_WCS_COORDSYS_*’ macros above) into
an output coordinate system (‘sys2’). The output values are
written in ‘lng2_d’ and ‘lng2_d’. The total number of points
should be given in ‘number’. If you want the operation to be done
in place (without allocating a new dataset), give the same pointers
to the coordinate arguments.
-- Function:
void
gal_wcs_coordsys_sys1_ref_in_sys2 (int sys1, int sys2, double
*lng2, double *lat2)
Return the longitude and latitude of the reference point (on the
equator) of the first coordinate system (‘sys1’) within the second
system (‘sys2’). Coordinate systems are identified by the
‘GAL_WCS_COORDSYS_*’ macros above.
-- Function:
int
gal_wcs_distortion_identify (struct wcsprm *wcs)
Returns the Gnuastro identifier for the distortion of the input WCS
structure. The returned value is one of the ‘GAL_WCS_DISTORTION_*’
macros defined above. When the input pointer to a structure is
‘NULL’, or it does not contain a distortion, the returned value
will be ‘GAL_WCS_DISTORTION_INVALID’.
-- Function:
struct wcsprm *
gal_wcs_distortion_convert(struct wcsprm *inwcs, int
outdisptype, size_t *fitsize)
Return a newly allocated WCS structure, where the distortion is
implemented in a different standard, identified by the identifier
‘outdisptype’. The Gnuastro WCS distortion identifiers are defined
in the ‘GAL_WCS_DISTORTION_*’ macros mentioned above.
The available conversions in this function will grow. Currently it
only supports converting TPV to SIP and vice versa, following the
recipe of Shupe et al. (2012)(4). Please get in touch with us if
you need other types of conversions.
For some conversions, direct analytical conversions do not exist.
It is thus necessary to model and fit the two types. In such
cases, it is also necessary to specify the ‘fitsize’ array that is
the size of the array along each C-ordered dimension, so you can
simply pass the ‘dsize’ element of your ‘gal_data_t’ dataset, see
*note Generic data container::. Currently this is only necessary
when converting TPV to SIP. For other conversions you may simply
pass a ‘NULL’ pointer.
For example, if you want to convert the TPV coefficients of your
input ‘image.fits’ to SIP coefficients, you can use the following
functions (which are also available as a command-line operation in
*note Fits::).
int nwcs;
gal_data_t *data=gal_fits_img_read("image.fits", "1", -1, 1, NULL);
inwcs=gal_wcs_read("image.fits", "1", 0, 0, 0, &nwcs, NULL);
data->wcs=gal_wcs_distortion_convert(inwcs, GAL_WCS_DISTORTION_TPV,
NULL);
wcsfree(inwcs);
gal_fits_img_write(data, "tpv.fits", NULL, 0);
-- Function:
double
gal_wcs_angular_distance_deg (double r1, double d1, double r2,
double d2)
Return the angular distance (in degrees) between a point located at
(‘r1’, ‘d1’) to (‘r2’, ‘d2’). All input coordinates are in
degrees. The distance (along a great circle) on a sphere between
two points is calculated with the equation below.
$$\cos(d)=\sin(d_1)\sin(d_2)+\cos(d_1)\cos(d_2)\cos(r_1-r_2)$$
However, since the pixel scales are usually very small numbers,
this function will not use that direct formula. It will be use the
Haversine formula (https://en.wikipedia.org/wiki/Haversine_formula)
which is better considering floating point errors:
$${\sin^2(d)\over 2}=\sin^2\left( {d_1-d_2\over 2} \right)+\cos(d_1)\cos(d_2)\sin^2\left( {r_1-r_2\over 2} \right)$$
-- Function:
void
gal_wcs_box_vertices_from_center (double ra_center, double
dec_center, double ra_delta, double dec_delta, double *out)
Calculate the vertices of a rectangular box given the central RA
and Dec and delta of each. The vertice coordinates are written in
the space that ‘out’ points to (assuming it has space for eight
‘double’s).
Given the spherical nature of the coordinate system, the vertice
lengths can't be calculated with a simple addition/subtraction.
For the declination, a simple addition/subtraction is enough.
Also, on the equator (where the RA is defined), a simple
addition/subtraction along the RA is fine. However, at other
declinations, the new RA after a shift needs special treatment,
such that close to the poles, a shift of 1 degree can correspond to
a new RA that is much more distant than the original RA. Assuming a
point at Right Ascension (RA) and Declination of $\alpha$ and
$\delta$, a shift of $R$ degrees along the positive RA direction
corresponds to a right ascension of
$\alpha+\frac{R}{\cos(\delta)}$. For more, see the description of
‘box-vertices-on-sphere’ in *note Coordinate and border
operators::.
The 8 coordinates of the 4 vertices of the box are written in the
order below. Where "bottom" corresponds to a lower declination and
"top" to higher declination, "left" corresponds to a larger RA and
"right" corresponds to lower RA.
out[0]: bottom-left RA
out[1]: bottom-left Dec
out[2]: bottom-right RA
out[3]: bottom-right Dec
out[4]: top-right RA
out[5]: top-right Dec
out[6]: top-left RA
out[7]: top-left Dec
-- Function:
double *
gal_wcs_pixel_scale (struct wcsprm *wcs)
Return the pixel scale for each dimension of ‘wcs’ in degrees. The
output is an allocated array of double precision floating point
type with one element for each dimension. If it is not successful,
this function will return ‘NULL’.
-- Function:
double
gal_wcs_pixel_area_arcsec2 (struct wcsprm *wcs)
Return the pixel area of ‘wcs’ in arc-second squared. This only
works when the input dataset has at least two dimensions and the
units of the first two dimensions (‘CUNIT’ keywords) are ‘deg’ (for
degrees). In other cases, this function will return a NaN.
-- Function:
int
gal_wcs_coverage (char *filename, char *hdu, size_t *ondim,
double **ocenter, double **owidth, double **omin, double
**omax, char *hdu_option_name)
Find the sky coverage of the image HDU (‘hdu’) within ‘filename’.
The number of dimensions is written into ‘ndim’, and space for the
various output arrays is internally allocated and filled with the
respective values. Therefore you need to free them afterwards.
For more on ‘hdu_option_name’ see the description of
‘gal_array_read’ in *note Array input output::.
Currently this function only supports images that are less than 180
degrees in width (which is usually the case!). This requirement
has been necessary to account for images that cross the RA=0 hour
circle on the sky. Please get in touch with us at
if you have an image that is larger
than 180 degrees so we try to find a solution based on need.
-- Function:
gal_data_t *
gal_wcs_world_to_img (gal_data_t *coords, struct wcsprm *wcs,
int inplace)
Convert the linked list of world coordinates in ‘coords’ to a
linked list of image coordinates given the input WCS structure.
‘coords’ must be a linked list of data structures of float64
('double') type, see*note Linked lists:: and *note List of
gal_data_t::. The top (first popped/read) node of the linked list
must be the first WCS coordinate (RA in an image usually) etc.
Similarly, the top node of the output will be the first image
coordinate (in the FITS standard). In case WCSLIB fails to convert
any of the coordinates (for example, the RA of one coordinate is
given as 400!), the respective element in the output will be
written as NaN.
If ‘inplace’ is zero, then the output will be a newly allocated
list and the input list will be untouched. However, if ‘inplace’
is non-zero, the output values will be written into the input's
already allocated array and the returned pointer will be the same
pointer to ‘coords’ (in other words, you can ignore the returned
value). Note that in the latter case, only the values will be
changed, things like units or name (if present) will be untouched.
-- Function:
gal_data_t *
gal_wcs_img_to_world (gal_data_t *coords, struct wcsprm *wcs,
int inplace)
Convert the linked list of image coordinates in ‘coords’ to a
linked list of world coordinates given the input WCS structure.
See the description of ‘gal_wcs_world_to_img’ for more details.
---------- Footnotes ----------
(1)
(2)
(3)
(4) Proc. of SPIE Vol. 8451 84511M-1.
, also available at
.
12.3.14 Arithmetic on datasets (‘arithmetic.h’)
-----------------------------------------------
When the dataset's type and other information are already known, any
programming language (including C) provides some very good tools for
various operations (including arithmetic operations like addition) on
the dataset with a simple loop. However, as an author of a program,
making assumptions about the type of data, its dimensions and other
basic characteristics will come with a large processing burden.
For example, if you always read your data as double precision
floating points for a simple operation like addition with an integer
constant, you will be wasting a lot of CPU and memory when the input
dataset is ‘int32’ type for example, (see *note Numeric data types::).
This overhead may be small for small images, but as you scale your
process up and work with hundred/thousands of files that can be very
large, this overhead will take a significant portion of the processing
power. The functions and macros in this section are designed precisely
for this purpose: to allow you to do any of the defined operations on
any dataset with no overhead (in the native type of the dataset).
Gnuastro's Arithmetic program uses the functions and macros of this
section, so please also have a look at the *note Arithmetic:: program
and in particular *note Arithmetic operators:: for a better description
of the operators discussed here.
The main function of this library is ‘gal_arithmetic’ that is
described below. It can take an arbitrary number of arguments as
operands (depending on the operator, similar to ‘printf’). Its first
two arguments are integers specifying the flags and operator. So first
we will review the constants for the recognized flags and operators and
discuss them, then introduce the actual function.
-- Macro: GAL_ARITHMETIC_FLAG_INPLACE
-- Macro: GAL_ARITHMETIC_FLAG_FREE
-- Macro: GAL_ARITHMETIC_FLAG_NUMOK
-- Macro: GAL_ARITHMETIC_FLAG_ENVSEED
-- Macro: GAL_ARITHMETIC_FLAG_QUIET
-- Macro: GAL_ARITHMETIC_FLAGS_BASIC
Bitwise flags to pass onto ‘gal_arithmetic’ (see below). To pass
multiple flags, use the bitwise OR operator. For example, if you
pass ‘GAL_ARITHMETIC_FLAG_INPLACE | GAL_ARITHMETIC_FLAG_NUMOK’,
then the operation will be done in-place (without allocating a new
array), and a single number will also be acceptable (that will be
applied to all the pixels). Each flag is described below:
‘GAL_ARITHMETIC_FLAG_INPLACE’
Do the operation in-place (in the input dataset, thus
modifying it) to improve CPU and memory usage. If this flag
is used, after ‘gal_arithmetic’ finishes, the input dataset
will be modified. It is thus useful if you have no more need
for the input after the operation.
‘GAL_ARITHMETIC_FLAG_FREE’
Free (all the) input dataset(s) after the operation is done.
Hence the inputs are no longer usable after ‘gal_arithmetic’.
‘GAL_ARITHMETIC_FLAG_NUMOK’
It is acceptable to use a number and an array together. For
example, if you want to add all the pixels in an image with a
single number you can pass this flag to avoid having to
allocate a constant array the size of the image (with all the
pixels having the same number).
‘GAL_ARITHMETIC_FLAG_ENVSEED’
Use the pre-defined environment variable for setting the
random number generator seed when an operator needs it (for
example, ‘mknoise-sigma’). For more on random number
generation in Gnuastro see *note Generating random numbers::.
‘GAL_ARITHMETIC_FLAG_QUIET’
Do not print any warnings or messages for operators that may
benefit from it. For example, by default the ‘mknoise-sigma’
operator prints the random number generator function and seed
that it used (in case the user wants to reproduce this result
later). By activating this bit flag to the call, that extra
information is not printed on the command-line.
‘GAL_ARITHMETIC_FLAGS_BASIC’
A wrapper for activating the three "basic" operations that are
commonly necessary together: ‘GAL_ARITHMETIC_FLAG_INPLACE’,
‘GAL_ARITHMETIC_FLAG_FREE’ and ‘GAL_ARITHMETIC_FLAG_NUMOK’.
-- Macro: GAL_ARITHMETIC_OP_PLUS
-- Macro: GAL_ARITHMETIC_OP_MINUS
-- Macro: GAL_ARITHMETIC_OP_MULTIPLY
-- Macro: GAL_ARITHMETIC_OP_DIVIDE
-- Macro: GAL_ARITHMETIC_OP_LT
-- Macro: GAL_ARITHMETIC_OP_LE
-- Macro: GAL_ARITHMETIC_OP_GT
-- Macro: GAL_ARITHMETIC_OP_GE
-- Macro: GAL_ARITHMETIC_OP_EQ
-- Macro: GAL_ARITHMETIC_OP_NE
-- Macro: GAL_ARITHMETIC_OP_AND
-- Macro: GAL_ARITHMETIC_OP_OR
Binary operators (requiring two operands) that accept datasets of
any recognized type (see *note Numeric data types::). When
‘gal_arithmetic’ is called with any of these operators, it expects
two datasets as arguments. For a full description of these
operators with the same name, see *note Arithmetic operators::.
The first dataset/operand will be put on the left of the operator
and the second will be put on the right. The output type of the
first four is determined from the input types (largest type of the
inputs). The rest (which are all conditional operators) will
output a binary ‘uint8_t’ (or ‘unsigned char’) dataset with values
of either ‘0’ (zero) or ‘1’ (one).
-- Macro: GAL_ARITHMETIC_OP_NOT
The logical NOT operator. When ‘gal_arithmetic’ is called with
this operator, it only expects one operand (dataset), since this is
a unary operator. The output is ‘uint8_t’ (or ‘unsigned char’)
dataset of the same size as the input. Any non-zero element in the
input will be ‘0’ (zero) in the output and any ‘0’ (zero) will have
a value of ‘1’ (one).
-- Macro: GAL_ARITHMETIC_OP_ISBLANK
A unary operator with output that is ‘1’ for any element in the
input that is blank, and ‘0’ for any non-blank element. When
‘gal_arithmetic’ is called with this operator, it will only expect
one input dataset. The output dataset will have ‘uint8_t’ (or
‘unsigned char’) type.
‘gal_arithmetic’ with this operator is just a wrapper for the
‘gal_blank_flag’ function of *note Library blank values:: and this
operator is just included for completeness in arithmetic
operations. So in your program, it might be easier to just call
‘gal_blank_flag’.
-- Macro: GAL_ARITHMETIC_OP_WHERE
The three-operand _where_ operator thoroughly discussed in *note
Arithmetic operators::. When ‘gal_arithmetic’ is called with this
operator, it will only expect three input datasets: the first
(which is the same as the returned dataset) is the array that will
be modified. The second is the condition dataset (that must have a
‘uint8_t’ or ‘unsigned char’ type), and the third is the value to
be used if condition is non-zero.
As a result, note that the order of operands when calling
‘gal_arithmetic’ with ‘GAL_ARITHMETIC_OP_WHERE’ is the opposite of
running Gnuastro's Arithmetic program with the ‘where’ operator
(see *note Arithmetic::). This is because the latter uses the
reverse-Polish notation which is not necessary when calling a
function (see *note Reverse polish notation::).
-- Macro: GAL_ARITHMETIC_OP_SQRT
-- Macro: GAL_ARITHMETIC_OP_LOG
-- Macro: GAL_ARITHMETIC_OP_LOG10
Unary operator functions for calculating the square root
($\sqrt{i}$), $ln(i)$ and $log(i)$ mathematical operators on each
element of the input dataset. The returned dataset will have a
floating point type, but its precision is determined from the
input: if the input is a 64-bit floating point, the output will
also be 64-bit. Otherwise, the returned dataset will be 32-bit
floating point: you do not gain precision by using these operators,
but you gain in operating speed if you use the sufficient
precision. See *note Numeric data types:: for more on the
precision of floating point numbers to help in selecting your
required floating point precision.
If you want your output to be 64-bit floating point but your input
is a different type, you can convert the input to a 64-bit floating
point type with ‘gal_data_copy_to_new_type’ or
‘gal_data_copy_to_new_type_free’(see *note Copying datasets::).
Alternatively, you can use the ‘GAL_ARITHMETIC_OP_TO_FLOAT64’
operators in the arithmetic library.
-- Macro: GAL_ARITHMETIC_OP_SIN
-- Macro: GAL_ARITHMETIC_OP_COS
-- Macro: GAL_ARITHMETIC_OP_TAN
-- Macro: GAL_ARITHMETIC_OP_ASIN
-- Macro: GAL_ARITHMETIC_OP_ACOS
-- Macro: GAL_ARITHMETIC_OP_ATAN
-- Macro: GAL_ARITHMETIC_OP_ATAN2
Trigonometric functions (and their inverse). All the angles,
either inputs or outputs, are in units of degrees.
-- Macro: GAL_ARITHMETIC_OP_SINH
-- Macro: GAL_ARITHMETIC_OP_COSH
-- Macro: GAL_ARITHMETIC_OP_TANH
-- Macro: GAL_ARITHMETIC_OP_ASINH
-- Macro: GAL_ARITHMETIC_OP_ACOSH
-- Macro: GAL_ARITHMETIC_OP_ATANH
Hyperbolic functions (and their inverse).
-- Macro: GAL_ARITHMETIC_OP_RA_TO_DEGREE
-- Macro: GAL_ARITHMETIC_OP_DEC_TO_DEGREE
-- Macro: GAL_ARITHMETIC_OP_DEGREE_TO_RA
-- Macro: GAL_ARITHMETIC_OP_DEGREE_TO_DEC
Unary operators to convert between degrees (as a single floating
point number) to the sexagesimal Right Ascension and Declination
format (as strings, respectively in the format of ‘_h_m_s’ and
‘_d_m_s’). The first two operators expect a string operand (in the
sexagesimal formats mentioned above, but also in the ‘_:_:_’) and
will return a double-precision floating point operand. The latter
two are the opposite.
-- Macro: GAL_ARITHMETIC_OP_COUNTS_TO_MAG
-- Macro: GAL_ARITHMETIC_OP_MAG_TO_COUNTS
-- Macro: GAL_ARITHMETIC_OP_MAG_TO_SB
-- Macro: GAL_ARITHMETIC_OP_SB_TO_MAG
-- Macro: GAL_ARITHMETIC_OP_COUNTS_TO_JY
-- Macro: GAL_ARITHMETIC_OP_JY_TO_COUNTS
-- Macro: GAL_ARITHMETIC_OP_MAG_TO_JY
-- Macro: GAL_ARITHMETIC_OP_JY_TO_MAG
-- Macro: GAL_ARITHMETIC_OP_MAG_TO_NANOMAGGY
-- Macro: GAL_ARITHMETIC_OP_NANOMAGGY_TO_MAG
Binary operators for converting brightness and surface brightness
units to and from each other. The first operand to all of them are
the values in the input unit (left of the ‘-TO-’, for example
counts in ‘COUNTS_TO_MAG’). The second popped operand is the zero
point (right of the ‘-TO-’, for example magnitudes in
‘COUNTS_TO_MAG’). The exceptions are the operators that involve
surface brightness (those with ‘SB’). For the surface brightness
related operators, the second popped operand is the area in units
of arcsec$^2$ and the third popped operand is the final unit.
-- Macro: GAL_ARITHMETIC_OP_COUNTS_TO_SB
-- Macro: GAL_ARITHMETIC_OP_SB_TO_COUNTS
Operators for converting counts to surface brightness and
vice-versa. These operators take three operands: 1) the input
dataset in units of counts or surface brightness (depending on the
operator), 2) the zero point, 3) the area in units of arcsec$^2$.
-- Macro: GAL_ARITHMETIC_OP_AU_TO_PC
-- Macro: GAL_ARITHMETIC_OP_PC_TO_AU
-- Macro: GAL_ARITHMETIC_OP_LY_TO_PC
-- Macro: GAL_ARITHMETIC_OP_PC_TO_LY
-- Macro: GAL_ARITHMETIC_OP_LY_TO_AU
-- Macro: GAL_ARITHMETIC_OP_AU_TO_LY
Unary operators to convert various distance units to and from each
other: Astronomical Units (AU), Parsecs (PC) and Light years (LY).
-- Macro: GAL_ARITHMETIC_OP_MINVAL
-- Macro: GAL_ARITHMETIC_OP_MAXVAL
-- Macro: GAL_ARITHMETIC_OP_NUMBERVAL
-- Macro: GAL_ARITHMETIC_OP_SUMVAL
-- Macro: GAL_ARITHMETIC_OP_MEANVAL
-- Macro: GAL_ARITHMETIC_OP_STDVAL
-- Macro: GAL_ARITHMETIC_OP_MEDIANVAL
Unary operand statistical operators that will return a single value
for datasets of any size. These are just wrappers around similar
functions in *note Statistical operations:: and are included in
‘gal_arithmetic’ only for completeness (to use easily in *note
Arithmetic::). In your programs, it will probably be easier if you
use those ‘gal_statistics_’ functions directly.
-- Macro: GAL_ARITHMETIC_OP_UNIQUE
-- Macro: GAL_ARITHMETIC_OP_NOBLANK
Unary operands that will remove some elements from the input
dataset. The first will return the unique elements, and the second
will return the non-blank elements. Due to the removal of
elements, the dimensionality of the output will be lost.
These are just wrappers over the ‘gal_statistics_unique’ and
‘gal_blank_remove’. These are just wrappers around similar
functions in *note Statistical operations:: and are included in
‘gal_arithmetic’ only for completeness (to use easily in *note
Arithmetic::). In your programs, it will probably be easier if you
use those ‘gal_statistics_’ functions directly.
-- Macro: GAL_ARITHMETIC_OP_ABS
Unary operand absolute-value operator.
-- Macro: GAL_ARITHMETIC_OP_MIN
-- Macro: GAL_ARITHMETIC_OP_MAX
-- Macro: GAL_ARITHMETIC_OP_NUMBER
-- Macro: GAL_ARITHMETIC_OP_SUM
-- Macro: GAL_ARITHMETIC_OP_MEAN
-- Macro: GAL_ARITHMETIC_OP_STD
-- Macro: GAL_ARITHMETIC_OP_MEDIAN
Multi-operand statistical operations. When ‘gal_arithmetic’ is
called with any of these operators, it will expect only a single
operand that will be interpreted as a list of datasets (see *note
List of gal_data_t::). These operators can work on multiple
threads using the ‘numthreads’ argument. See the discussion under
the ‘min’ operator in *note Arithmetic operators::.
The output will be a single dataset with each of its elements
replaced by the respective statistical operation on the whole list.
The type of the output is determined from the operator
(irrespective of the input type): for ‘GAL_ARITHMETIC_OP_MIN’ and
‘GAL_ARITHMETIC_OP_MAX’, it will be the same type as the input, for
‘GAL_ARITHMETIC_OP_NUMBER’, the output will be ‘GAL_TYPE_UINT32’
and for the rest, it will be ‘GAL_TYPE_FLOAT32’.
-- Macro: GAL_ARITHMETIC_OP_QUANTILE
Similar to the operands above (including ‘GAL_ARITHMETIC_MIN’),
except that when ‘gal_arithmetic’ is called with these operators,
it requires two arguments. The first is the list of datasets like
before, and the second is the 1-element dataset with the quantile
value. The output type is the same as the inputs.
-- Macro: GAL_ARITHMETIC_OP_SIGCLIP_STD
-- Macro: GAL_ARITHMETIC_OP_SIGCLIP_MEAN
-- Macro: GAL_ARITHMETIC_OP_SIGCLIP_MEDIAN
-- Macro: GAL_ARITHMETIC_OP_SIGCLIP_NUMBER
Similar to the operands above (including ‘GAL_ARITHMETIC_MIN’),
except that when ‘gal_arithmetic’ is called with these operators,
it requires two arguments. The first is the list of datasets like
before, and the second is the 2-element list of $\sigma$-clipping
parameters. The first element in the parameters list is the
multiple of sigma and the second is the termination criteria (see
*note Sigma clipping::). The output type of
‘GAL_ARITHMETIC_OP_SIGCLIP_NUMBER’ will be ‘GAL_TYPE_UINT32’ and
for the rest it will be ‘GAL_TYPE_FLOAT32’.
-- Macro: GAL_ARITHMETIC_OP_MKNOISE_SIGMA
-- Macro: GAL_ARITHMETIC_OP_MKNOISE_POISSON
-- Macro: GAL_ARITHMETIC_OP_MKNOISE_UNIFORM
Add noise to the input dataset. These operators take two
arguments: the first is the input data set (can have any
dimensionality or number of elements. The second argument is the
noise specifier (a single element, of any type): for a fixed-sigma
noise, it is the Gaussian standard deviation, for the Poisson
noise, it is the background (see *note Photon counting noise::) and
for the uniform distribution it is the width of the interval around
each element of the input dataset.
By default, a separate random number generator seed will be used on
each separate run of these operators. Therefore two identical runs
on the same input will produce different results. You can get
reproducible results by setting the ‘GAL_RNG_SEED’ environment
variable and activating the ‘GAL_ARITHMETIC_FLAG_ENVSEED’ flag.
For more on random number generation in Gnuastro, see *note
Generating random numbers::.
By default these operators will print the random number generator
function and seed (in case the user wants to reproduce the result
later), but this can be disabled by activating the bit-flag
‘GAL_ARITHMETIC_FLAG_QUIET’ described above.
-- Macro: GAL_ARITHMETIC_OP_RANDOM_FROM_HIST
-- Macro: GAL_ARITHMETIC_OP_RANDOM_FROM_HIST_RAW
Select random values from a custom distribution (defined by a
histogram). For more, see the description of the respective
operators in *note Generating random numbers::.
-- Macro: GAL_ARITHMETIC_OP_STITCH
Stitch a list of input datasets along the requested dimension. See
the description of the ‘stitch’ operator in Arithmetic (*note
Dimensionality changing operators::).
-- Macro: GAL_ARITHMETIC_OP_POW
Binary operator to-power operator. When ‘gal_arithmetic’ is called
with any of these operators, it will expect two operands: raising
the first by the second (returning a floating point, inputs can be
integers).
-- Macro: GAL_ARITHMETIC_OP_BITAND
-- Macro: GAL_ARITHMETIC_OP_BITOR
-- Macro: GAL_ARITHMETIC_OP_BITXOR
-- Macro: GAL_ARITHMETIC_OP_BITLSH
-- Macro: GAL_ARITHMETIC_OP_BITRSH
-- Macro: GAL_ARITHMETIC_OP_MODULO
Binary integer-only operand operators. These operators are only
defined on integer data types. When ‘gal_arithmetic’ is called
with any of these operators, it will expect two operands: the first
is put on the left of the operator and the second on the right.
The ones starting with ‘BIT’ are the respective bitwise operators
in C and ‘MODULO’ is the modulo/remainder operator. For a
discussion on these operators, please see *note Arithmetic
operators::.
The output type is determined from the input types and C's internal
conversions: it is strongly recommended that both inputs have the
same type (any integer type), otherwise the bitwise behavior will
be determined by your compiler.
-- Macro: GAL_ARITHMETIC_OP_BITNOT
The unary bitwise NOT operator. When ‘gal_arithmetic’ is called
with any of these operators, it will expect one operand of an
integer type and preform the bitwise NOT operation on it. The
output will have the same type as the input.
-- Macro: GAL_ARITHMETIC_OP_TO_UINT8
-- Macro: GAL_ARITHMETIC_OP_TO_INT8
-- Macro: GAL_ARITHMETIC_OP_TO_UINT16
-- Macro: GAL_ARITHMETIC_OP_TO_INT16
-- Macro: GAL_ARITHMETIC_OP_TO_UINT32
-- Macro: GAL_ARITHMETIC_OP_TO_INT32
-- Macro: GAL_ARITHMETIC_OP_TO_UINT64
-- Macro: GAL_ARITHMETIC_OP_TO_INT64
-- Macro: GAL_ARITHMETIC_OP_TO_FLOAT32
-- Macro: GAL_ARITHMETIC_OP_TO_FLOAT64
Unary type-conversion operators. When ‘gal_arithmetic’ is called
with any of these operators, it will expect one operand and convert
it to the requested type. Note that with these operators,
‘gal_arithmetic’ is just a wrapper over the
‘gal_data_copy_to_new_type’ or ‘gal_data_copy_to_new_type_free’
that are discussed in ‘Copying datasets’. It accepts these
operators only for completeness and easy usage in *note
Arithmetic::. So in your programs, it might be preferable to
directly use those functions.
-- Macro: GAL_ARITHMETIC_OP_E
-- Macro: GAL_ARITHMETIC_OP_C
-- Macro: GAL_ARITHMETIC_OP_G
-- Macro: GAL_ARITHMETIC_OP_H
-- Macro: GAL_ARITHMETIC_OP_AU
-- Macro: GAL_ARITHMETIC_OP_LY
-- Macro: GAL_ARITHMETIC_OP_PI
-- Macro: GAL_ARITHMETIC_OP_AVOGADRO
-- Macro: GAL_ARITHMETIC_OP_FINESTRUCTURE
Return the respective mathematical constant. For their description
please see *note Constants::. The constant values are taken from
the GNU Scientific Library's headers (defined in ‘gsl/gsl_math.h’).
-- Macro: GAL_ARITHMETIC_OP_BOX_AROUND_ELLIPSE
Return the width (along horizontal) and height (along vertical) of
a box that encompasses an ellipse with the same center point. For
more on the three input operands to this operator see the
description of ‘box-around-ellipse’. This function returns two
datasets as a ‘gal_data_t’ linked list. The top element of the
list is the height and its next element is the width.
-- Macro: GAL_ARITHMETIC_OP_BOX_VERTICES_ON_SPHERE
Return the vertices of a (possibly rectangular) box on a sphere,
given its center RA, Dec and the width of the box along the two
dimensions. It will take the spherical nature of the coordinate
system into account (for more, see the description of
‘gal_wcs_box_vertices_from_center’ in *note World Coordinate
System::). This function returns 8 datasets as a ‘gal_data_t’
linked list in the following order: bottom-left RA, bottom-left
Dec, bottom-right RA, bottom-right Dec, top-right RA, top-right
Dec, top-left RA, top-left Dec.
-- Macro: GAL_ARITHMETIC_OP_MAKENEW
Create a new, zero-valued dataset with an unsigned 8-bit data type.
The length along each dimension of the dataset should be given as a
single list of ‘gal_data_t’s. The number of dimensions is derived
from the number of nodes in the list and the length along each
dimension is the single-valued element within that list. Just note
that the list should be in the reverse of the desired dimensions.
-- Macro: GAL_ARITHMETIC_OP_MAKENEW
Given a dataset and a constant,
-- Macro: GAL_ARITHMETIC_OPSTR_LOADCOL_HDU
-- Macro: GAL_ARITHMETIC_OPSTR_LOADCOL_FILE
-- Macro: GAL_ARITHMETIC_OPSTR_LOADCOL_PREFIX
-- Macro: GAL_ARITHMETIC_OPSTR_LOADCOL_HDU_LEN
-- Macro: GAL_ARITHMETIC_OPSTR_LOADCOL_FILE_LEN
-- Macro: GAL_ARITHMETIC_OPSTR_LOADCOL_PREFIX_LEN
Constant components of the ‘load-col-’ operator (see *note Loading
external columns::). These are just fixed strings (and their
lengths) that are placed in between the various components of that
operator to allow choosing a certain column of a certain HDU of a
certain file.
-- Macro: GAL_ARITHMETIC_OP_INDEX
-- Macro: GAL_ARITHMETIC_OP_COUNTER
-- Macro: GAL_ARITHMETIC_OP_INDEXONLY
-- Macro: GAL_ARITHMETIC_OP_COUNTERONLY
Return a dataset with the same number of elements and
dimensionality as the first (and only!) input dataset. But each
output pixel's value will be replaced by its index (counting from
0) or counter (counting from 1). Note that the
‘GAL_ARITHMETIC_OP_INDEX’ and ‘GAL_ARITHMETIC_OP_INDEXONLY’
operators are identical within the library (same for the counter
operators). They are given separate macros here to help the
higher-level callers to manage their inputs separately (see *note
Size and position operators::).
-- Macro: GAL_ARITHMETIC_OP_SIZE
Size operator that will return a single value for datasets of any
kind. When ‘gal_arithmetic’ is called with this operator, it
requires two arguments. The first is the dataset, and the second
is a single integer value. The output type is a single integer.
-- Macro: GAL_ARITHMETIC_OP_SWAP
Return the first dataset, but with the second dataset being placed
in the ‘next’ element of the first. This is useful to swap the
operators on the stacks of the higher-level programs that call the
arithmetic library.
-- Macro: GAL_ARITHMETIC_OP_EQB1950_TO_EQJ2000
-- Macro: GAL_ARITHMETIC_OP_EQB1950_TO_ECB1950
-- Macro: GAL_ARITHMETIC_OP_EQB1950_TO_ECJ2000
-- Macro: GAL_ARITHMETIC_OP_EQB1950_TO_GALACTIC
-- Macro: GAL_ARITHMETIC_OP_EQB1950_TO_SUPERGALACTIC
-- Macro: GAL_ARITHMETIC_OP_EQJ2000_TO_EQB1950
-- Macro: GAL_ARITHMETIC_OP_EQJ2000_TO_ECB1950
-- Macro: GAL_ARITHMETIC_OP_EQJ2000_TO_ECJ2000
-- Macro: GAL_ARITHMETIC_OP_EQJ2000_TO_GALACTIC
-- Macro: GAL_ARITHMETIC_OP_EQJ2000_TO_SUPERGALACTIC
-- Macro: GAL_ARITHMETIC_OP_ECB1950_TO_EQB1950
-- Macro: GAL_ARITHMETIC_OP_ECB1950_TO_EQJ2000
-- Macro: GAL_ARITHMETIC_OP_ECB1950_TO_ECJ2000
-- Macro: GAL_ARITHMETIC_OP_ECB1950_TO_GALACTIC
-- Macro: GAL_ARITHMETIC_OP_ECB1950_TO_SUPERGALACTIC
-- Macro: GAL_ARITHMETIC_OP_ECJ2000_TO_EQB1950
-- Macro: GAL_ARITHMETIC_OP_ECJ2000_TO_EQJ2000
-- Macro: GAL_ARITHMETIC_OP_ECJ2000_TO_ECB1950
-- Macro: GAL_ARITHMETIC_OP_ECJ2000_TO_GALACTIC
-- Macro: GAL_ARITHMETIC_OP_ECJ2000_TO_SUPERGALACTIC
-- Macro: GAL_ARITHMETIC_OP_GALACTIC_TO_EQB1950
-- Macro: GAL_ARITHMETIC_OP_GALACTIC_TO_EQJ2000
-- Macro: GAL_ARITHMETIC_OP_GALACTIC_TO_ECB1950
-- Macro: GAL_ARITHMETIC_OP_GALACTIC_TO_ECJ2000
-- Macro: GAL_ARITHMETIC_OP_GALACTIC_TO_SUPERGALACTIC
-- Macro: GAL_ARITHMETIC_OP_SUPERGALACTIC_TO_EQB1950
-- Macro: GAL_ARITHMETIC_OP_SUPERGALACTIC_TO_EQJ2000
-- Macro: GAL_ARITHMETIC_OP_SUPERGALACTIC_TO_ECB1950
-- Macro: GAL_ARITHMETIC_OP_SUPERGALACTIC_TO_ECJ2000
-- Macro: GAL_ARITHMETIC_OP_SUPERGALACTIC_TO_GALACTIC
Operators that convert recognized celestial coordinates to and from
each other. They all take two operands and return two
‘gal_data_t’s (as a list). For more on celestial coordinate
conversion, see *note Coordinate conversion operators::.
-- Function:
gal_data_t *
gal_arithmetic (int operator, size_t numthreads, int flags,
...)
Apply the requested arithmetic operator on the operand(s). The
_operator_ is identified through the macros above (that start with
‘GAL_ARITHMETIC_OP_’). The number of necessary operands (number of
arguments to replace '‘...’' in the declaration of this function,
above) depends on the operator and is described under each
operator, above. Each operand has a type of '‘gal_data_t *’' (see
last paragraph with example).
If the operator can work on multiple threads, the number of threads
can be specified with ‘numthreads’. When the operator is
single-threaded, ‘numthreads’ will be ignored. Special conditions
can also be specified with the ‘flag’ operator (a bit-flag with
bits described above, for example, ‘GAL_ARITHMETIC_FLAG_INPLACE’ or
‘GAL_ARITHMETIC_FLAG_FREE’).
‘gal_arithmetic’ is a multi-argument function (like C's ‘printf’).
In other words, the number of necessary arguments is not fixed and
depends on the value to ‘operator’. Below, you can see a minimal,
fully working example, showing how different operators need
different numbers of arguments.
#include
#include
#include
#include
int
main(void)
{
/* Define the datasets and flag. */
gal_data_t *in1, *in2, *out1, *out2;
int flag=GAL_ARITHMETIC_FLAGS_BASIC;
/* Read the input images. */
in1=gal_fits_img_read("image1.fits", "1", -1, 1, NULL);
in2=gal_fits_img_read("image2.fits", "1", -1, 1, NULL);
/* Take the logarithm (base-e) of the first input. */
out1=gal_arithmetic(GAL_ARITHMETIC_OP_LOG, 1, flag, in1);
/* Add the second input with the logarithm of the first. */
out2=gal_arithmetic(GAL_ARITHMETIC_OP_PLUS, 1, flag, in2, out1);
/* Write the output into a file. */
gal_fits_img_write(out2, "out.fits", NULL, 0);
/* Clean up. Due to the in-place flag (in
* 'GAL_ARITHMETIC_FLAGS_BASIC'), 'out1' and 'out2' point to the
* same array in memory and due to the freeing flag, any input
* dataset(s) that were not returned have been freed internally
* by 'gal_arithmetic'. Therefore it is only necessary to free
* 'out2': all other allocated spaces have been freed internally.
* before reaching this point. */
gal_data_free(out2);
/* Return control back to the OS (saying that we succeeded). */
return EXIT_SUCCESS;
}
As you see above, you can feed the returned dataset from one call
of ‘gal_arithmetic’ to another call. The advantage of using
‘gal_arithmetic’ (as opposed to manually writing a ‘for’ or ‘while’
loop and doing the operation with the ‘+’ operator and ‘log()’
function yourself), is that you do not have to worry about the type
of the input data (for a list of acceptable data types in Gnuastro,
see *note Library data types::). Arithmetic will automatically
deal with the data types internally and choose the best output type
depending on the operator.
-- Function:
int
gal_arithmetic_set_operator (char *string, size_t
*num_operands)
Return the operator macro/code that corresponds to ‘string’. The
number of operands that it needs are written into the space that
‘*num_operands’ points to. If the string could not be interpreted
as an operator, this function will return
‘GAL_ARITHMETIC_OP_INVALID’.
This function will check ‘string’ with the fixed human-readable
names (using ‘strcmp’) for the operators and return the two
numbers. Note that ‘string’ must only contain the single operator
name and nothing else (not even any extra white space).
-- Function:
char *
gal_arithmetic_operator_string (int operator)
Return the human-readable standard string that corresponds to the
given operator. For example, when the input is
‘GAL_ARITHMETIC_OP_PLUS’ or ‘GAL_ARITHMETIC_OP_MEAN’, the strings
‘+’ or ‘mean’ will be returned.
-- Function:
gal_data_t *
gal_arithmetic_load_col (char *str, int searchin, int
ignorecase, size_t minmapsize, int quietmmap)
Return the column that corresponds to the identifier in the input
string (‘str’). ‘str’ is expected to be in the format of the
‘load-col-’ operator (see *note Loading external columns::). This
function will extract the column identifier, the file name and the
HDU (if necessary) from the string, read the requested column in
memory and return it.
See *note Table input output:: for the macros that can be given to
‘searchin’ and ‘ignorecase’ and *note Generic data container:: for
the definitions of ‘minmapsize’ and ‘quietmmap’.
12.3.15 Tessellation library (‘tile.h’)
---------------------------------------
In many contexts, it is desirable to slice the dataset into subsets or
tiles (overlapping or not). In such a way that you can work on each
tile independently. One method would be to copy that region to a
separate allocated space, but in many contexts this is not necessary and
in fact can be a big burden on CPU/Memory usage. The ‘block’ pointer in
Gnuastro's *note Generic data container:: is defined for such
situations: where allocation is not necessary. You just want to read
the data or write to it independently (or in coordination with) other
regions of the dataset. Added with parallel processing, this can
greatly improve the time/memory consumption.
See the figure below for example: assume the ‘larger’ dataset is a
contiguous block of memory that you are interpreting as a 2D array. But
you only want to work on the smaller ‘tile’ region.
larger
---------------------------------
| |
| tile |
| ---------- |
| | | |
| |_ | |
| |*| | |
| ---------- |
| tile->block = larger |
|_ |
|*| |
---------------------------------
To use ‘gal_data_t’'s ‘block’ concept, you allocate a ‘gal_data_t
*tile’ which is initialized with the pointer to the first element in the
sub-array (as its ‘array’ argument). Note that this is not necessarily
the first element in the larger array. You can set the size of the tile
along with the initialization as you please. Recall that, when given a
non-‘NULL’ pointer as ‘array’, ‘gal_data_initialize’ (and thus
‘gal_data_alloc’) do not allocate any space and just uses the given
pointer for the new ‘array’ element of the ‘gal_data_t’. So your ‘tile’
data structure will not be pointing to a separately allocated space.
After the allocation is done, you just point ‘tile->block’ to the
‘larger’ dataset which hosts the full block of memory. Where relevant,
Gnuastro's library functions will check the ‘block’ pointer of their
input dataset to see how to deal with dimensions and increments so they
can always remain within the tile. The tools introduced in this section
are designed to help in defining and working with tiles that are created
in this manner.
Since the block structure is defined as a pointer, arbitrary levels
of tessellation/grid-ing are possible (‘tile->block’ may itself be a
tile in an even larger allocated space). Therefore, just like a
linked-list (see *note Linked lists::), it is important to have the
‘block’ pointer of the largest (allocated) dataset set to ‘NULL’.
Normally, you will not have to worry about this, because
‘gal_data_initialize’ (and thus ‘gal_data_alloc’) will set the ‘block’
element to ‘NULL’ by default, just remember not to change it. You can
then only change the ‘block’ element for the tiles you define over the
allocated space.
Below, we will first review constructs for *note Independent tiles::
and then define the current approach to fully tessellating a dataset (or
covering every pixel/data-element with a non-overlapping tile grid in
*note Tile grid::. This approach to dealing with parts of a larger
block was inspired from a similarly named concept in the GNU Scientific
Library (GSL), see its "Vectors and Matrices" chapter for their
implementation.
12.3.15.1 Independent tiles
...........................
The most general application of tiles is to treat each independently,
for example they may overlap, or they may not cover the full image.
This section provides functions to help in checking/inspecting such
tiles. In *note Tile grid:: we will discuss functions that
define/work-with a tile grid (where the tiles do not overlap and fully
cover the input dataset). Therefore, the functions in this section are
general and can be used for the tiles produced by that section also.
-- Function:
void
gal_tile_start_coord (gal_data_t *tile, size_t *start_coord)
Calculate the starting coordinates of a tile in its allocated block
of memory and write them in the memory that ‘start_coord’ points to
(which must have ‘tile->ndim’ elements).
-- Function:
void
gal_tile_start_end_coord (gal_data_t *tile, size_t *start_end,
int rel_block)
Put the starting and ending (end point is not inclusive)
coordinates of ‘tile’ into the ‘start_end’ array. It is assumed
that a space of ‘2*tile->ndim’ has been already allocated (static
or dynamic) for ‘start_end’ before this function is called.
‘rel_block’ (or relative-to-block) is only relevant when ‘tile’ has
an intermediate tile between it and the allocated space (like a
channel, see ‘gal_tile_full_two_layers’). If it does not
(‘tile->block’ points the allocated dataset), then the value to
‘rel_block’ is irrelevant.
When ‘tile->block’ is itself a larger block and ‘rel_block’ is set
to 0, then the starting and ending positions will be based on the
position within ‘tile->block’, not the allocated space.
-- Function:
void *
gal_tile_start_end_ind_inclusive (gal_data_t *tile, gal_data_t
*work, size_t *start_end_inc)
Put the indices of the first/start and last/end pixels (inclusive)
in a tile into the ‘start_end’ array (that must have two elements).
NOTE: this function stores the index of each point, not its
coordinates. It will then return the pointer to the start of the
tile in the ‘work’ data structure (which does not have to be equal
to ‘tile->block’.
The outputs of this function are defined to make it easy to parse
over an n-dimensional tile. For example, this function is one of
the most important parts of the internal processing of in
‘GAL_TILE_PARSE_OPERATE’ function-like macro that is described
below.
-- Function:
gal_data_t *
gal_tile_series_from_minmax (gal_data_t *block, size_t
*minmax, size_t number)
Construct a list of tile(s) given coordinates of the minimum and
maximum of each tile. The minimum and maximums are assumed to be
inclusive and in C order (slowest dimension first). The returned
pointer is an allocated ‘gal_data_t’ array that can later be freed
with ‘gal_data_array_free’ (see *note Arrays of datasets::).
Internally, each element of the output array points to the next
element, so the output may also be treated as a list of datasets
(see *note List of gal_data_t::) and passed onto the other
functions described in this section.
The array keeping the minimum and maximum coordinates for each tile
must have the following format. So in total ‘minmax’ must have
‘2*ndim*number’ elements.
| min0_d0 | min0_d1 | max0_d0 | max0_d1 | ...
... | minN_d0 | minN_d1 | maxN_d0 | maxN_d1 |
-- Function:
gal_data_t *
gal_tile_block (gal_data_t *tile)
Return the dataset that contains ‘tile’'s allocated block of
memory. If tile is immediately defined as part of the allocated
block, then this is equivalent to ‘tile->block’. However, it is
possible to have multiple layers of tiles (where ‘tile->block’ is
itself a tile). So this function is the most generic way to get to
the actual allocated dataset.
-- Function:
size_t
gal_tile_block_increment (gal_data_t *block, size_t *tsize,
size_t num_increment, size_t *coord)
Return the increment necessary to start at the next contiguous
patch memory associated with a tile. ‘block’ is the allocated
block of memory and ‘tsize’ is the size of the tile along every
dimension. If ‘coord’ is ‘NULL’, it is ignored. Otherwise, it
will contain the coordinate of the start of the next contiguous
patch of memory.
This function is intended to be used in a loop and ‘num_increment’
is the main variable to this function. For the first time you call
this function, it should be ‘1’. In subsequent calls (while you
are parsing a tile), it should be increased by one.
-- Function:
gal_data_t *
gal_tile_block_write_const_value (gal_data_t *tilevalues,
gal_data_t *tilesll, int withblank, int initialize)
Write a constant value for each tile over the area it covers in an
allocated dataset that is the size of ‘tile’'s allocated block of
memory (found through ‘gal_tile_block’ described above). The
arguments to this function are:
‘tilevalues’
This must be an array that has the same number of elements as
the nodes in in ‘tilesll’ and in the same order that 'tilesll'
elements are parsed (from top to bottom, see *note Linked
lists::). As a result the array's number of dimensions is
irrelevant, it will be parsed contiguously.
‘tilesll’
The list of input tiles (see *note List of gal_data_t::).
Internally, it might be stored as an array (for example, the
output of ‘gal_tile_series_from_minmax’ described above), but
this function does not care, it will parse the ‘next’ elements
to go to the next tile. This function will not pop-from or
free the ‘tilesll’, it will only parse it from start to end.
‘withblank’
If the block containing the tiles has blank elements, those
blank elements will be blank in the output of this function
also, hence the array will be initialized with blank values
when this option is called (see below).
‘initialize’
Initialize the allocated space with blank values before
writing in the constant values. This can be useful when the
tiles do not cover the full allocated block.
-- Function:
gal_data_t *
gal_tile_block_check_tiles (gal_data_t *tilesll)
Make a copy of the memory block and fill it with the index of each
tile in ‘tilesll’ (counting from 0). The non-filled areas will
have blank values. The output dataset will have a type of
‘GAL_TYPE_INT32’ (see *note Library data types::).
This function can be used when you want to check the coverage of
each tile over the allocated block of memory. It is just a wrapper
over the ‘gal_tile_block_write_const_value’ (with ‘withblank’ set
to zero).
-- Function:
void *
gal_tile_block_relative_to_other (gal_data_t *tile, gal_data_t
*other)
Return the pointer corresponding to the start of the region covered
by ‘tile’ over the ‘other’ dataset. See the examples in
‘GAL_TILE_PARSE_OPERATE’ for some example applications of this
function.
-- Function:
void
gal_tile_block_blank_flag (gal_data_t *tilell, size_t
numthreads)
Check if each tile in the list has blank values and update its
‘flag’ to mark this check and its result (see *note Generic data
container::). The operation will be done on ‘numthreads’ threads.
-- Function-like macro: GAL_TILE_PARSE_OPERATE (IN, OTHER, PARSE_OTHER,
CHECK_BLANK, OP)
Parse ‘IN’ (which can be a tile or a fully allocated block of
memory) and do the ‘OP’ operation on it. ‘OP’ can be any
combination of C expressions. If ‘OTHER!=NULL’, ‘OTHER’ will be
interpreted as a dataset and this macro will allow access to its
element(s) and it can optionally be parsed while parsing over ‘IN’.
If ‘OTHER’ is a fully allocated block of memory (not a tile), then
the same region that is covered by ‘IN’ within its own block will
be parsed (the same starting pixel with the same number of pixels
in each dimension). Hence, in this case, the blocks of ‘OTHER’ and
‘IN’ must have the same size. When ‘OTHER’ is a tile it must have
the same size as ‘IN’ and parsing will start from its starting
element/pixel. Also, the respective allocated blocks of ‘OTHER’
and ‘IN’ (if different) may have different sizes. Using ‘OTHER’
(along with ‘PARSE_OTHER’), this function-like macro will thus
enable you to parse and define your own operation on two fixed size
regions in one or two blocks of memory. In the latter case, they
may have different numeric data types, see *note Numeric data
types::).
The input arguments to this macro are explained below, the expected
type of each argument are also written following the argument name:
‘IN (gal_data_t)’
Input dataset, this can be a tile or an allocated block of
memory.
‘OTHER (gal_data_t)’
Dataset (‘gal_data_t’) to parse along with ‘IN’. It can be
‘NULL’. In that case, ‘o’ (see description of ‘OP’ below)
will be ‘NULL’ and should not be used. If ‘PARSE_OTHER’ is
zero, only its first element will be used and the size of this
dataset is irrelevant.
When ‘OTHER’ is a block of memory, it has to have the same
size as the allocated block of ‘IN’. When it s a tile, it has
to have the same size as ‘IN’.
‘PARSE_OTHER (int)’
Parse the other dataset along with the input. When this is
non-zero and ‘OTHER!=NULL’, then the ‘o’ pointer will be
incremented to cover the ‘OTHER’ tile at the same rate as ‘i’,
see description of ‘OP’ for ‘i’ and ‘o’.
‘CHECK_BLANK (int)’
If it is non-zero, then the input will be checked for blank
values and ‘OP’ will only be called when we are not on a blank
element.
‘OP’
Operator: this can be any number of C expressions. This macro
is going to define a ‘itype *i’ variable which will increment
over each element of the input array/tile. ‘itype’ will be
replaced with the C type that corresponds to the type of
‘INPUT’. As an example, if ‘INPUT’'s type is
‘GAL_DATA_UINT16’ or ‘GAL_DATA_FLOAT32’, ‘i’ will be defined
as ‘uint16’ or ‘float’ respectively.
This function-like macro will also define an ‘otype *o’ which
you can use to access an element of the ‘OTHER’ dataset (if
‘OTHER!=NULL’). ‘o’ will correspond to the type of ‘OTHER’
(similar to ‘itype’ and ‘INPUT’ discussed above). If
‘PARSE_OTHER’ is non-zero, then ‘o’ will also be incremented
to the same index element but in the other array. You can use
these along with any other variable you define before this
macro to process the input and/or the other.
All variables within this function-like macro begin with
‘tpo_’ except for the three variables listed below.
Therefore, as long as you do not start the names of your
variables with this prefix everything will be fine. Note that
‘i’ (and possibly ‘o’) will be incremented once by this
function-like macro, so do not increment them within ‘OP’.
‘i’
Pointer to the element of ‘INPUT’ that is being parsed
with the proper type.
‘o’
Pointer to the element of ‘OTHER’ that is being parsed
with the proper type. ‘o’ can only be used if
‘OTHER!=NULL’ and it will be parsed/incremented if
‘PARSE_OTHER’ is non-zero.
‘b’
Blank value in the type of ‘INPUT’.
You can use a given tile (‘tile’ on a dataset that it was not
initialized with but has the same size, let's call it ‘new’) with
the following steps:
void *tarray;
gal_data_t *tblock;
/* `tile->block' must be corrected AFTER `tile->array'. */
tarray = tile->array;
tblock = tile->block;
tile->array = gal_tile_block_relative_to_other(tile, new);
tile->block = new;
/* Parse and operate over this region of the `new' dataset. */
GAL_TILE_PARSE_OPERATE(tile, NULL, 0, 0, {
YOUR_PROCESSING;
});
/* Reset `tile->block' and `tile->array'. */
tile->array=tarray;
tile->block=tblock;
You can work on the same region of another block in one run of this
function-like macro. To do that, you can make a fake tile and pass
that as the ‘OTHER’ argument. Below is a demonstration, ‘tile’ is
the actual tile that you start with and ‘new’ is the other block of
allocated memory.
size_t zero=0;
gal_data_t *faketile;
/* Allocate the fake tile, these can be done outside a loop
* (over many tiles). */
faketile=gal_data_alloc(NULL, new->type, 1, &zero,
NULL, 0, -1, 1, NULL, NULL, NULL);
free(faketile->array); /* To keep things clean. */
free(faketile->dsize); /* To keep things clean. */
faketile->block = new;
faketile->ndim = new->ndim;
/* These can be done in a loop (over many tiles). */
faketile->size = tile->size;
faketile->dsize = tile->dsize;
faketile->array = gal_tile_block_relative_to_other(tile, new);
/* Do your processing.... in a loop (over many tiles). */
GAL_TILE_PARSE_OPERATE(tile, faketile, 1, 1, {
YOUR_PROCESSING_EXPRESSIONS;
});
/* Clean up (outside the loop). */
faketile->array=NULL;
faketile->dsize=NULL;
gal_data_free(faketile);
12.3.15.2 Tile grid
...................
One very useful application of tiles is to completely cover an input
dataset with tiles. Such that you know every pixel/data-element of the
input image is covered by only one tile. The constructs in this section
allow easy definition of such a tile structure. They will create lists
of tiles that are also usable by the general tools discussed in *note
Independent tiles::.
As discussed in *note Tessellation::, (mainly raw) astronomical
images will mostly require two layers of tessellation, one for amplifier
channels which all have the same size and another (smaller tile-size)
tessellation over each channel. Hence, in this section we define a
general structure to keep the main parameters of this two-layer
tessellation and help in benefiting from it.
-- Type (C struct): gal_tile_two_layer_params
The general structure to keep all the necessary parameters for a
two-layer tessellation.
struct gal_tile_two_layer_params
{
/* Inputs */
size_t *tilesize; /*******************************/
size_t *numchannels; /* These parameters have to be */
float remainderfrac; /* filled manually before */
uint8_t workoverch; /* calling the functions in */
uint8_t checktiles; /* this section. */
uint8_t oneelempertile; /*******************************/
/* Internal parameters. */
size_t ndim;
size_t tottiles;
size_t tottilesinch;
size_t totchannels;
size_t *channelsize;
size_t *numtiles;
size_t *numtilesinch;
char *tilecheckname;
size_t *permutation;
size_t *firsttsize;
/* Tile and channel arrays (which are also lists). */
gal_data_t *tiles;
gal_data_t *channels;
};
-- Function:
size_t *
gal_tile_full (gal_data_t *input, size_t *regular, float
remainderfrac, gal_data_t **out, size_t multiple, size_t
**firsttsize)
Cover the full dataset with (mostly) identical tiles and return the
number of tiles created along each dimension. The regular tile
size (along each dimension) is determined from the ‘regular’ array.
If ‘input’'s size is not an exact multiple of ‘regular’ for each
dimension, then the tiles touching the edges in that dimension will
have a different size to fully cover every element of the input
(depending on ‘remainderfrac’).
The output is an array with the same dimensions as ‘input’ which
contains the number of tiles along each dimension. See *note
Tessellation:: for a description of its application in Gnuastro's
programs and ‘remainderfrac’, just note that this function defines
only one layer of tiles.
This is a low-level function (independent of the
‘gal_tile_two_layer_params’ structure defined above). If you want
a two-layer tessellation, directly call ‘gal_tile_full_two_layers’
that is described below. The input arguments to this function are:
‘input’
The main dataset (allocated block) which you want to create a
tessellation over (only used for its sizes). So ‘input’ may
be a tile also.
‘regular’
The size of the regular tiles along each of the input's
dimensions. So it must have the same number of elements as
the dimensions of ‘input’ (or ‘input->ndim’).
‘remainderfrac’
The significant fraction of the remainder space to see if it
should be split into two and put on both sides of a dimension
or not. This is thus only relevant ‘input’ length along a
dimension is not an exact multiple of the regular tile size
along that dimension. See *note Tessellation:: for a more
thorough discussion.
‘out’
Pointer to the array of data structures that will keep all the
tiles (see *note Arrays of datasets::). If ‘*out==NULL’, then
the necessary space to keep all the tiles will be allocated.
If not, then all the tile information will be filled from the
dataset that ‘*out’ points to, see ‘multiple’ for more.
‘multiple’
When ‘*out==NULL’ (and thus will be allocated by this
function), allocate space for ‘multiple’ times the number of
tiles needed. This can be very useful when you have several
more identically sized ‘inputs’, and you want all their tiles
to be allocated (and thus indexed) together, even though they
have different ‘block’ datasets (that then link to one
allocated space). See the definition of channels in *note
Tessellation:: and ‘gal_tile_full_two_layers’ below.
‘firsttsize’
The size of the first tile along every dimension. This is
only different from the regular tile size when ‘regular’ is
not an exact multiple of ‘input’'s length along every
dimension. This array is allocated internally by this
function.
-- Function:
void
gal_tile_full_sanity_check (char *filename, char *hdu,
gal_data_t *input, struct gal_tile_two_layer_params *tl)
Make sure that the input parameters (in ‘tl’, short for two-layer)
correspond to the input dataset. ‘filename’ and ‘hdu’ are only
required for error messages. Also, allocate and fill the
‘tl->channelsize’ array.
-- Function:
void
gal_tile_full_two_layers (gal_data_t *input, struct
gal_tile_two_layer_params *tl)
Create the two layered tessellation in ‘tl’. The general set of
steps you need to take to define the two-layered tessellation over
an image can be seen in the example code below.
gal_data_t *input;
struct gal_tile_two_layer_params tl;
char *filename="input.fits", *hdu="1";
/* Set all the inputs shown in the structure definition. */
...
/* Read the input dataset. */
input=gal_fits_img_read(filename, hdu, -1, 1, NULL);
/* Do a sanity check and preparations. */
gal_tile_full_sanity_check(filename, hdu, input, &tl);
/* Build the two-layer tessellation*/
gal_tile_full_two_layers(input, &tl);
/* `tl.tiles' and `tl.channels' are now a lists of tiles.*/
-- Function:
void
gal_tile_full_permutation (struct gal_tile_two_layer_params
*tl)
Make a permutation to allow the conversion of tile location in
memory to its location in the full input dataset and put it in
‘tl->permutation’. If a permutation has already been defined for
the tessellation, this function will not do anything. If
permutation will not be necessary (there is only one channel or one
dimension), then this function will not do anything
(‘tl->permutation’ must have been initialized to ‘NULL’).
When there is only one channel OR one dimension, the tiles are
allocated in memory in the same order that they represent the input
data. However, to make channel-independent processing possible in
a generic way, the tiles of each channel are allocated
contiguously. So, when there is more than one channel AND more
than one dimension, the index of the tile does not correspond to
its position in the grid covering the input dataset.
The example below may help clarify: assume you have a 6x6
tessellation with two channels in the horizontal and one in the
vertical. On the left you can see how the tile IDs correspond to
the input dataset. NOTE how '03' is on the second row, not on the
first after '02'. On the right, you can see how the tiles are
stored in memory (and shown if you simply write the array into a
FITS file for example).
Corresponding to input In memory
---------------------- --------------
15 16 17 33 34 35 30 31 32 33 34 35
12 13 14 30 31 32 24 25 26 27 28 29
09 10 11 27 28 29 18 19 20 21 22 23
06 07 08 24 25 26 <-- 12 13 14 15 16 17
03 04 05 21 22 23 06 07 08 09 10 11
00 01 02 18 19 20 00 01 02 03 04 05
As a result, if your values are stored in same order as the tiles,
and you want them in over-all memory (for example, to save as a
FITS file), you need to permute the values:
gal_permutation_apply(values, tl->permutation);
If you have values over-all and you want them in tile-order, you
can apply the inverse permutation:
gal_permutation_apply_inverse(values, tl->permutation);
Recall that this is the definition of permutation in this context:
permute: IN_ALL[ i ] = IN_MEMORY[ perm[i] ]
inverse: IN_ALL[ perm[i] ] = IN_MEMORY[ i ]
-- Function:
void
gal_permutation_apply_onlydim0 (gal_data_t *input, size_t
*permutation)
Similar to ‘gal_permutation_apply’, but when the dataset is
2-dimensional, permute each row (dimension 1 in C) as one element.
In other words, only permute along dimension 0. The ‘permutation’
array should therefore only have ‘input->dsize[0]’ elements.
-- Function:
void
gal_tile_full_values_write (gal_data_t *tilevalues, struct
gal_tile_two_layer_params *tl, int withblank, char *filename,
gal_fits_list_key_t *keys, int freekeys)
Write one value for each tile into a file. It is important to note
that the values in ‘tilevalues’ must be ordered in the same manner
as the tiles, so ‘tilevalues->array[i]’ is the value that should be
given to ‘tl->tiles[i]’. The ‘tl->permutation’ array must have
been initialized before calling this function with
‘gal_tile_full_permutation’.
If ‘withblank’ is non-zero, then block structure of the tiles will
be checked and all blank pixels in the block will be blank in the
final output file also.
-- Function:
gal_data_t *
gal_tile_full_values_smooth (gal_data_t *tilevalues, struct
gal_tile_two_layer_params *tl, size_t width, size_t
numthreads)
Smooth the given values with a flat kernel of the given ‘width’.
This cannot be done manually because if ‘tl->workoverch==0’, tiles
in different channels must not be mixed/smoothed. Also the tiles
are contiguous within the channel, not within the image, see the
description under ‘gal_tile_full_permutation’.
-- Function:
size_t
gal_tile_full_id_from_coord (struct gal_tile_two_layer_params
*tl, size_t *coord)
Return the ID of the tile that corresponds to the coordinates
‘coord’. Having this ID, you can use the ‘tl->tiles’ array to get
to the proper tile or read/write a value into an array that has one
value per tile.
-- Function:
void
gal_tile_full_free_contents (struct gal_tile_two_layer_params
*tl)
Free all the allocated arrays within ‘tl’.
12.3.16 Bounding box (‘box.h’)
------------------------------
Functions related to reporting the bounding box of certain inputs are
declared in ‘gnuastro/box.h’. All coordinates in this header are in the
FITS format (first axis is the horizontal and the second axis is
vertical).
-- Function:
void
gal_box_bound_ellipse_extent (double a, double b, double
theta_deg, double *extent)
Return the maximum extent along each dimension of the given ellipse
from the center of the ellipse. Therefore this is half the extent
of the box in each dimension. ‘a’ is the ellipse semi-major axis,
‘b’ is the semi-minor axis, ‘theta_deg’ is the position angle in
degrees. The extent in each dimension is in floating point format
and stored in ‘extent’ which must already be allocated before this
function.
-- Function:
void
gal_box_bound_ellipse (double a, double b, double theta_deg,
long *width)
Any ellipse can be enclosed into a rectangular box. This function
will write the height and width of that box where ‘width’ points
to. It assumes the center of the ellipse is located within the
central pixel of the box. ‘a’ is the ellipse semi-major axis
length, ‘b’ is the semi-minor axis, ‘theta_deg’ is the position
angle in degrees. The ‘width’ array will contain the output size
in long integer type. ‘width[0]’, and ‘width[1]’ are the number of
pixels along the first and second FITS axis. Since the ellipse
center is assumed to be in the center of the box, all the values in
‘width’ will be an odd integer.
-- Function:
void
gal_box_bound_ellipsoid_extent (double *semiaxes, double
*euler_deg, double *extent)
Return the maximum extent along each dimension of the given
ellipsoid from its center. Therefore this is half the extent of
the box in each dimension. The semi-axis lengths of the ellipsoid
must be present in the 3 element ‘semiaxis’ array. The ‘euler_deg’
array contains the three ellipsoid Euler angles in degrees. For a
description of the Euler angles, see description of
‘gal_box_bound_ellipsoid’ below. The extent in each dimension is
in floating point format and stored in ‘extent’ which must already
be allocated before this function.
-- Function:
void
gal_box_bound_ellipsoid (double *semiaxes, double *euler_deg,
long *width)
Any ellipsoid can be enclosed into a rectangular volume/box. The
purpose of this function is to give the integer size/width of that
box. The semi-axes lengths of the ellipse must be in the
‘semiaxes’ array (with three elements). The major axis length must
be the first element of ‘semiaxes’. The only other condition is
that the next two semi-axes must both be smaller than the first.
The orientation of the major axis is defined through three proper
Euler angles (ZXZ order in degrees) that are given in the
‘euler_deg’ array. The ‘width’ array will contain the output size
in long integer type (in FITS axis order). Since the ellipsoid
center is assumed to be in the center of the box, all the values in
‘width’ will be an odd integer.
The proper Euler angles can be defined in many ways (which axes to
rotate about). For a full description of the Euler angles, please
see Wikipedia (https://en.wikipedia.org/wiki/Euler_angles). Here
we adopt the ZXZ (or $Z_1X_2Z_3$) proper Euler angles were the
first rotation is done around the Z axis, the second one about the
(rotated) X axis and the third about the (rotated) Z axis.
-- Function:
void
gal_box_border_from_center (double center, size_t ndim, long
*width, long *fpixel, long *lpixel)
Given the center coordinates in ‘center’ and the ‘width’ (along
each dimension) of a box, return the coordinates of the first
(‘fpixel’) and last (‘lpixel’) pixels. All arrays must have ‘ndim’
elements (one for each dimension).
-- Function:
void
gal_box_border_rotate_around_center (long *fpixel, long
*lpixel, size_t ndim, float rotate_deg)
Modify the input first and last pixels (‘fpixel’ and ‘lpixel’, that
you can estimate with ‘gal_box_border_from_center’) to account for
the given rotation (in units of degrees) in 2D (currently ‘ndim’
can only have a value of ‘2’).
-- Function:
int
gal_box_overlap (long *naxes, long *fpixel_i, long *lpixel_i,
long *fpixel_o, long *lpixel_o, size_t ndim)
An ‘ndim’-dimensional dataset of size ‘naxes’ (along each
dimension, in FITS order) and a box with first and last (inclusive)
coordinate of ‘fpixel_i’ and ‘lpixel_i’ is given. This box does
not necessarily have to lie within the dataset, it can be outside
of it, or only partially overlap. This function will change the
values of ‘fpixel_i’ and ‘lpixel_i’ to exactly cover the overlap in
the input dataset's coordinates.
This function will return 1 if there is an overlap and 0 if there
is not. When there is an overlap, the coordinates of the first and
last pixels of the overlap will be put in ‘fpixel_o’ and
‘lpixel_o’.
12.3.17 Polygons (‘polygon.h’)
------------------------------
Polygons are commonly necessary in image processing. For example, in
Crop they are used for cutting out non-rectangular regions of a image
(see *note Crop::), and in Warp, for mapping different pixel grids over
each other (see *note Warp::).
Polygons come in two classes: convex and concave (or generally,
non-convex!), see below for a demonstration. Convex polygons are those
where all inner angles are less than 180 degrees. By contrast, a convex
polygon is one where an inner angle may be more than 180 degress.
Concave Polygon Convex Polygon
D --------C D------------- C
\ | E / |
\E | \ |
/ | \ |
A--------B A ----------B
In all the functions here the vertices (and points) are defined as an
array. So a polygon with 4 vertices will be identified with an array of
8 elements with the first two elements keeping the 2D coordinates of the
first vertice and so on.
-- Macro: GAL_POLYGON_MAX_CORNERS
The largest number of vertices a polygon can have in this library.
-- Macro: GAL_POLYGON_ROUND_ERR
We have to consider floating point round-off errors when dealing
with polygons. For example, we will take ‘A’ as the maximum of ‘A’
and ‘B’ when ‘A>B-GAL_POLYGON_ROUND_ERR’.
-- Function:
void
gal_polygon_vertices_sort_convex (double *in, size_t n, size_t
*ordinds)
We have a simple polygon (that can result from projection, so its
edges do not collide or it does not have holes) and we want to
order its corners in an anticlockwise fashion. This is necessary
for clipping it and finding its area later. The input vertices can
have practically any order.
The input (‘in’) is an array containing the coordinates (two
values) of each vertice. ‘n’ is the number of corners. So ‘in’
should have ‘2*n’ elements. The output (‘ordinds’) is an array
with ‘n’ elements specifying the indices in order. This array must
have been allocated before calling this function. The indexes are
output for more generic usage, for example, in a homographic
transform (necessary in warping an image, see *note Linear warping
basics::), the necessary order of vertices is the same for all the
pixels. In other words, only the positions of the vertices change,
not the way they need to be ordered. Therefore, this function
would only be necessary once.
As a summary, the input is unchanged, only ‘n’ values will be put
in the ‘ordinds’ array. Such that calling the input coordinates in
the following fashion will give an anti-clockwise order when there
are 4 vertices:
1st vertice: in[ordinds[0]*2], in[ordinds[0]*2+1]
2nd vertice: in[ordinds[1]*2], in[ordinds[1]*2+1]
3rd vertice: in[ordinds[2]*2], in[ordinds[2]*2+1]
4th vertice: in[ordinds[3]*2], in[ordinds[3]*2+1]
The implementation of this is very similar to the Graham scan in
finding the Convex Hull. However, in projection we will never have
a concave polygon (the left condition below, where this algorithm
will get to E before D), we will always have a convex polygon
(right case) or E will not exist! This is because we are always
going to be calculating the area of the overlap between a
quadrilateral and the pixel grid or the quadrilateral itself.
The ‘GAL_POLYGON_MAX_CORNERS’ macro is defined so there will be no
need to allocate these temporary arrays separately. Since we are
dealing with pixels, the polygon cannot really have too many
vertices.
-- Function:
int
gal_polygon_is_convex (double *v, size_t n)
Returns ‘1’ if the polygon is convex with vertices defined by ‘v’
and ‘0’ if it is a concave polygon. Note that the vertices of the
polygon should be sorted in an anti-clockwise manner.
-- Function:
double
gal_polygon_area_flat (double *v, size_t n)
Find the area of a polygon with vertices defined in ‘v’ on a
euclidian (flat) coordinate system. ‘v’ points to an array of
doubles which keep the positions of the vertices such that ‘v[0]’
and ‘v[1]’ are the positions of the first vertice to be considered.
-- Function:
double
gal_polygon_area_sky (double *v, size_t n)
Find the area of a polygon with vertices defined in ‘v’ on a
celestial coordinate system. This is a coordinate system where the
first coordinate goes from 0 to 360 (increasing to the right),
while the second coordinate ranges from -90 to +90 (on the poles).
‘v’ points to an array of doubles which keep the positions of the
vertices such that ‘v[0]’ and ‘v[1]’ are the positions of the first
vertice to be considered.
This function uses an approximation to account for the curvature of
the sky and the different nature of spherical coordinates with
respect to the flat coordinate system. Bug 64617
(https://savannah.gnu.org/bugs/index.php?64617) has been defined in
Gnuastro to address this problem. Please check that bug in case it
has been fixed. Until this bug is fixed, here are some tips:
• Subtract the RA and Dec of all the vertice coordinates from a
constant so the center of the polygon falls on (RA, Dec) of
(180,0). The sphere has a similar nature everywhere on it, so
shifting the polygon vertices will not change its area; this
also removes issues with the RA=0 or RA=360 coordinate and
decrease issues caused by RA depending on declination.
• These approximations should not cause any statistically
significant error on normal (less than a few degrees) scales.
But it won't hurt to do a small sanity check for your
particular usage scenario.
• Any help (even in the mathematics of the problem; not
necessary programming) would be appreciated (we didn't have
time to derive the necessary equations), so if you have some
background in this and can prepare the mathematical
description of the problem, please get in touch.
-- Function:
int
gal_polygon_is_inside (double *v, double *p, size_t n)
Returns ‘0’ if point ‘p’ in inside a polygon, either convex or
concave. The vertices of the polygon are defined by ‘v’ and ‘0’
otherwise, they have to be ordered in an anti-clockwise manner.
This function uses the winding number algorithm
(https://en.wikipedia.org/wiki/Point_in_polygon#Winding_number_algorithm),
to check the points. Note that this is a generic function (working
on both concave and convex polygons, so if you know before-hand
that your polygon is convex, it is much more efficient to use
‘gal_polygon_is_inside_convex’.
-- Function:
int
gal_polygon_is_inside_convex (double *v, double *p, size_t n)
Return ‘1’ if the point ‘p’ is within the polygon whose vertices
are defined by ‘v’. The polygon is assumed to be convex, for a
more generic function that deals with concave and convex polygons,
see ‘gal_polygon_is_inside’. Note that the vertices of the polygon
have to be sorted in an anti-clock-wise manner.
-- Function:
int
gal_polygon_ppropin (double *v, double *p, size_t n)
Similar to ‘gal_polygon_is_inside_convex’, except that if the point
‘p’ is on one of the edges of a polygon, this will return ‘0’.
-- Function:
int
gal_polygon_is_counterclockwise (double *v, size_t n)
Returns ‘1’ if the sorted polygon has a counter-clockwise
orientation and ‘0’ otherwise. This function uses the concept of
"winding", which defines the relative order in which the vertices
of a polygon are listed to determine the orientation of vertices.
For complex polygons (where edges, or sides, intersect), the most
significant orientation is returned. In a complex polygon, when
the alternative windings are equal (for example, an ‘8’-shape) it
will return ‘1’ (as if it was counter-clockwise). Note that the
polygon vertices have to be sorted before calling this function.
-- Function:
int
gal_polygon_to_counterclockwise (double *v, size_t n)
Arrange the vertices of the sorted polygon in place, to be in a
counter-clockwise direction. If the input polygon already has a
counter-clockwise direction it will not touch the input. The
return value is ‘1’ on successful execution. This function is just
a wrapper over ‘gal_polygon_is_counterclockwise’, and will reverse
the order of the vertices when necessary.
-- Function:
void
gal_polygon_clip (double *s, size_t n, double *c, size_t m,
double *o, size_t *numcrn)
Clip (find the overlap of) two polygons. This function uses the
Sutherland-Hodgman
(https://en.wikipedia.org/wiki/Sutherland%E2%80%93Hodgman_algorithm)
polygon clipping algorithm. Note that the vertices of both
polygons have to be sorted in an anti-clock-wise manner.
The Pseudocode from Wikipedia:
List outputList = subjectPolygon;
for (Edge clipEdge in clipPolygon) do
List inputList = outputList;
outputList.clear();
Point S = inputList.last;
for (Point E in inputList) do
if (E inside clipEdge) then
if (S not inside clipEdge) then
outputList.add(ComputeIntersection(S,E,clipEdge));
end if
outputList.add(E);
else if (S inside clipEdge) then
outputList.add(ComputeIntersection(S,E,clipEdge));
end if
S = E;
done
done
The difference is that we are not using lists, but arrays to keep
polygon vertices. The two polygons are called Subject ‘s’ and Clip
‘c’ with ‘n’ and ‘m’ vertices respectively. The output is stored
in ‘o’ and the number of elements in the output are stored in what
‘*numcrn’ (for number of corners) points to.
-- Function:
void
gal_polygon_vertices_sort (double *vertices, size_t n, size_t
*ordinds)
Sort the indices of the un-ordered ‘vertices’ array to a
counter-clockwise polygon in the already allocated space of
‘ordinds’. It is assumed that there are ‘n’ vertices, and thus
that ‘vertices’ contains ‘2*n’ elements where the two coordinates
of the first vertice occupy the first two elements of the array and
so on.
The polygon can be both concave and convex (see the start of this
section). However, note that for concave polygons there is no
unique sort from an un-ordered set of vertices. So after this
function you may want to use ‘gal_polygon_is_convex’ and print a
warning to check the output if the polygon was concave.
Note that the contents of the ‘vertices’ array are left untouched
by this function. If you want to write the ordered vertice
coordinates in another array with the same size, you can use a loop
like this:
for(i=0;i
#include /* qsort is defined in stdlib.h. */
#include
int
main (void)
{
size_t s[4]={0, 1, 2, 3};
float f[4]={1.3,0.2,1.8,0.1};
gal_qsort_index_single=f;
qsort(s, 4, sizeof(size_t), gal_qsort_index_single_float_d);
printf("%zu, %zu, %zu, %zu\n", s[0], s[1], s[2], s[3]);
return EXIT_SUCCESS;
}
The output will be: ‘2, 0, 1, 3’.
-- Function:
int
gal_qsort_index_single_TYPE_i (const void *a, const void *b)
Similar to ‘gal_qsort_index_single_TYPE_d’, but will sort the
indexes such that the values of ‘gal_qsort_index_single’ can be
parsed in increasing order.
-- Function:
int
gal_qsort_index_multi_d (const void *a, const void *b)
When passed to ‘qsort’ with an array of ‘gal_qsort_index_multi’,
this function will sort the array based on the values of the given
indices. The sorting will be ordered according to the ‘values’
pointer of ‘gal_qsort_index_multi’. Note that ‘values’ must point
to the same place in all the structures of the
‘gal_qsort_index_multi’ array.
This function is only useful when the indices of multiple arrays on
multiple threads are to be sorted. If your program is single
threaded, or all the indices belong to a single array (sorting
different sub-sets of indices in a single array on multiple
threads), it is recommended to use ‘gal_qsort_index_single_d’.
-- Function:
int
gal_qsort_index_multi_i (const void *a, const void *b)
Similar to ‘gal_qsort_index_multi_d’, but the result will be sorted
in increasing order (first element will have the smallest value).
12.3.19 K-d tree (‘kdtree.h’)
-----------------------------
K-d tree is a space-partitioning binary search tree for organizing
points in a k-dimensional space. They are a very useful data structure
for multidimensional searches like range searches and nearest neighbor
searches. For a more formal and complete introduction see the Wikipedia
page (https://en.wikipedia.org/wiki/K-d_tree).
Each non-leaf node in a k-d tree divides the space into two parts,
known as half-spaces. To select the top/root node for partitioning, we
find the median of the points and make a hyperplane normal to the first
dimension. The points to the left of this space are represented by the
left subtree of that node and points to the right of the space are
represented by the right subtree. This is then repeated for all the
points in the input, thus associating a "left" and "right" branch for
each input point.
Gnuastro uses the standard algorithms of the k-d tree with one small
difference that makes it much more memory and CPU optimized. The set of
input points that define the tree nodes are given as a list of
Gnuastro's data container type, see *note List of gal_data_t::. Each
‘gal_data_t’ in the list represents the point's coordinate in one
dimension, and the first element in the list is the first dimension.
Hence the number of data values in each ‘gal_data_t’ (which must be
equal in all of them) represents the number of points. This is the same
format that Gnuastro's Table reading/writing functions read/write
columns in tables, see *note Table input output::.
The output k-d tree is a list of two ‘gal_data_t’s, representing the
input's row-number (or index, counting from 0) of the left and right
subtrees of each row. Each ‘gal_data_t’ thus has the same number of
rows (or points) as the input, but only containing integers with a type
of ‘uint32_t’ (unsigned 32-bit integer). If a node has no left, or
right subtree, then ‘GAL_BLANK_UINT32’ will be used. Below you can see
the simple tree for 2D points from Wikipedia. The input point
coordinates are represented as two input ‘gal_data_t’s (‘X’ and ‘Y’,
where ‘X->next=Y’ and ‘Y->next=NULL’). If you had three dimensional
points, you could define an extra ‘gal_data_t’ such that ‘Y->next=Z’ and
‘Z->next=NULL’. The output is always a list of two ‘gal_data_t’s, where
the first one contains the index of the left sub-tree in the input, and
the second one, the index of the right subtree. The index of the root
node (‘0’ in the case below(1)) is also returned as a single number.
INDEX INPUT OUTPUT K-D Tree
(as guide) X --> Y LEFT --> RIGHT (visualized)
---------- ------- -------------- ------------------
0 5 4 1 2 (5,4)
1 2 3 BLANK 4 / \
2 7 2 5 3 (2,3) \
3 9 6 BLANK BLANK \ (7,2)
4 4 7 BLANK BLANK (4,7) / \
5 8 1 BLANK BLANK (8,1) (9,6)
This format is therefore scalable to any number of dimensions: the
number of dimensions are determined from the number of nodes in the
input list of ‘gal_data_t’s (for example, using ‘gal_list_data_number’).
In Gnuastro's k-d tree implementation, there are thus no special
structures to keep every tree node (which would take extra memory and
would need to be moved around as the tree is being created). Everything
is done internally on the index of each point in the input dataset: the
only thing that is flipped/sorted during tree creation is the index to
the input row for any number of dimensions. As a result, Gnuastro's k-d
tree implementation is very memory and CPU efficient and its two output
columns can directly be written into a standard table (without having to
define any special binary format).
-- Function:
gal_data_t *
gal_kdtree_create (gal_data_t *coords_raw, size_t *root)
Create a k-d tree in a bottom-up manner (from leaves to the root).
This function returns two ‘gal_data_t’s connected as a list, see
description above. The first dataset contains the indexes of left
and right nodes of the subtrees for each input node. The index of
the root node is written into the memory that ‘root’ points to.
‘coords_raw’ is the list of the input points (one ‘gal_data_t’ per
dimension, see above). If the input dataset has no data
(‘coords_raw->size==0’), this function will return a ‘NULL’
pointer.
For example, assume you have the simple set of points below (from
the visualized example at the start of this section) in a
plain-text file called ‘coordinates.txt’:
$ cat coordinates.txt
5 4
2 3
7 2
9 6
4 7
8 1
With the program below, you can calculate the kd-tree, and write it
in a FITS file (while keeping the root index as a FITS keyword
inside of it).
#include
#include
#include
int
main (void)
{
gal_data_t *input, *kdtree;
char kdtreefile[]="kd-tree.fits";
char inputfile[]="coordinates.txt";
/* To write the root within the saved file. */
size_t root;
char *unit="index";
char *keyname="KDTROOT";
gal_fits_list_key_t *keylist=NULL;
char *comment="k-d tree root index (counting from 0).";
/* Read the input table. Note: this assumes the table only
* contains your input point coordinates (one column for each
* dimension). If it contains more columns with other properties
* for each point, you can specify which columns to read by
* name or number, see the documentation of 'gal_table_read'. */
input=gal_table_read(inputfile, "1", NULL, NULL,
GAL_TABLE_SEARCH_NAME, 0, -1, 0, NULL);
/* Construct a k-d tree. The index of root is stored in `root` */
kdtree=gal_kdtree_create(input, &root);
/* Write the k-d tree to a file and write root index and input
* name as FITS keywords ('gal_table_write' frees 'keylist').*/
gal_fits_key_list_title_add(&keylist, "k-d tree parameters", 0);
gal_fits_key_write_filename("KDTIN", inputfile, &keylist, 0, 1);
gal_fits_key_list_add_end(&keylist, GAL_TYPE_SIZE_T, keyname, 0,
&root, 0, comment, 0, unit, 0);
gal_table_write(kdtree, &keylist, NULL, GAL_TABLE_FORMAT_BFITS,
kdtreefile, "kdtree", 0, 1);
/* Clean up and return. */
gal_list_data_free(input);
gal_list_data_free(kdtree);
return EXIT_SUCCESS;
}
You can inspect the saved k-d tree FITS table with Gnuastro's *note
Table:: (first command below), and you can see the keywords
containing the root index with *note Fits:: (second command below):
asttable kd-tree.fits
astfits kd-tree.fits -h1
-- Function:
size_t
gal_kdtree_nearest_neighbour (gal_data_t *coords_raw,
gal_data_t *kdtree, size_t root, double *point, double
*least_dist)
Returns the index of the nearest input point to the query point
(‘point’, assumed to be an array with same number of elements as
‘gal_data_t’s in ‘coords_raw’). The distance between the query
point and its nearest neighbor is stored in the space that
‘least_dist’ points to. This search is efficient due to the
constant checking for the presence of possible best points in other
branches. If it is not possible for the other branch to have a
better nearest neighbor, that branch is not searched.
As an example, let's use the k-d tree that was created in the
example of ‘gal_kdtree_create’ (above) and find the nearest row to
a given coordinate (‘point’). This will be a very common scenario,
especially in large and multi-dimensional datasets where the k-d
tree creation can take long and you do not want to re-create the
k-d tree every time. In the ‘gal_kdtree_create’ example output, we
also wrote the k-d tree root index as a FITS keyword (‘KDTROOT’),
so after loading the two table data (input coordinates and k-d
tree), we will read the root from the FITS keyword. This is a very
simple example, but the scalability is clear: for example, it is
trivial to parallelize (see *note Library demo - multi-threaded
operation::).
#include
#include
#include
int
main (void)
{
/* INPUT: desired point. */
double point[2]={8.9,5.9};
/* Same as example in description of 'gal_kdtree_create'. */
gal_data_t *input, *kdtree;
char kdtreefile[]="kd-tree.fits";
char inputfile[]="coordinates.txt";
/* Processing variables of this function. */
char kdtreehdu[]="1";
double *in_x, *in_y, least_dist;
size_t root, nkeys=1, nearest_index;
gal_data_t *rkey, *keysll=gal_data_array_calloc(nkeys);
/* Read the input coordinates, see comments in example of
* 'gal_kdtree_create' for more. */
input=gal_table_read(inputfile, "1", NULL, NULL,
GAL_TABLE_SEARCH_NAME, 0, -1, 0, NULL);
/* Read the k-d tree contents (created before). */
kdtree=gal_table_read(kdtreefile, "1", NULL, NULL,
GAL_TABLE_SEARCH_NAME, 0, -1, 0, NULL);
/* Read the k-d tree root index from the header keyword.
* See example in description of 'gal_fits_key_read_from_ptr'.*/
keysll[0].name="KDTROOT";
keysll[0].type=GAL_TYPE_SIZE_T;
gal_fits_key_read(kdtreefile, kdtreehdu, keysll, 0, 0, NULL);
keysll[0].name=NULL; /* Since we did not allocate it. */
rkey=gal_data_copy_to_new_type(&keysll[0], GAL_TYPE_SIZE_T);
root=((size_t *)(rkey->array))[0];
/* Find the nearest neighbour of the point. */
nearest_index=gal_kdtree_nearest_neighbour(input, kdtree, root,
point, &least_dist);
/* Print the results. */
in_x=input->array;
in_y=input->next->array;
printf("(%g, %g): nearest is (%g, %g), with a distance of %g\n",
point[0], point[1], in_x[nearest_index],
in_y[nearest_index], least_dist);
/* Clean up and return. */
gal_data_free(rkey);
gal_list_data_free(input);
gal_list_data_free(kdtree);
gal_data_array_free(keysll, nkeys, 1);
return EXIT_SUCCESS;
}
---------- Footnotes ----------
(1) This example input table is the same as the example in Wikipedia
(as of December 2020). However, on the Wikipedia output, the root node
is (7,2), not (5,4). The difference is primarily because there are 6
rows and the median element of an even number of elements can vary by
integer calculation strategies. Here we use 0-based indexes for finding
median and round to the smaller integer.
12.3.20 Permutations (‘permutation.h’)
--------------------------------------
Permutation is the technical name for re-ordering of values. The need
for permutations occurs a lot during (mainly low-level) processing. To
do permutation, you must provide two inputs: an array of values (that
you want to re-order in place) and a permutation array which contains
the new index of each element (let's call it ‘perm’). The diagram below
shows the input array before and after the re-ordering.
permute: AFTER[ i ] = BEFORE[ perm[i] ] i = 0 .. N-1
inverse: AFTER[ perm[i] ] = BEFORE[ i ] i = 0 .. N-1
The functions here are a re-implementation of the GNU Scientific
Library's ‘gsl_permute’ function. The reason we did not use that
function was that it uses system-specific types (like ‘long’ and ‘int’)
which can have different widths on different systems, hence are not
easily convertible to Gnuastro's fixed width types (see *note Numeric
data types::). There is also a separate function for each type, heavily
using macros to allow a ‘base’ function to work on all the types. Thus
it is hard to read/understand. Hence, Gnuastro contains a re-write of
their steps in a new type-agnostic method which is a single function
that can work on any type.
As described in GSL's source code and manual, this implementation
comes from Donald Knuth's _Art of computer programming_ book, in the
"Sorting and Searching" chapter of Volume 3 (3rd ed). Exercise 10 of
Section 5.2 defines the problem and in the answers, Knuth describes the
solution. So if you are interested, please have a look there for more.
We are in contact with the GSL developers and in the future(1) we
will submit these implementations to GSL. If they are finally
incorporated there, we will delete this section in future versions.
-- Function:
void
gal_permutation_check (size_t *permutation, size_t size)
Print how ‘permutation’ will re-order an array that has ‘size’
elements for each element in one one line.
-- Function:
void
gal_permutation_apply (gal_data_t *input, size_t *permutation)
Apply ‘permutation’ on the ‘input’ dataset (can have any type), see
above for the definition of permutation.
-- Function:
void
gal_permutation_apply_inverse (gal_data_t *input, size_t
*permutation)
Apply the inverse of ‘permutation’ on the ‘input’ dataset (can have
any type), see above for the definition of permutation.
-- Function:
void
gal_permutation_transpose_2d (gal_data_t *input)
Transpose an input 2D matrix into a new dataset. If the input is
not a square, this function will change the ‘input->array’ element
to a newly allocated array (the old one will be freed internally).
Therefore, in case you have already stored ‘input->array’ for other
usage _before_ this function, and the input is not a square, be
sure to update the previously stored pointer if the input is not a
square.
---------- Footnotes ----------
(1) Gnuastro's Task 14497 (http://savannah.gnu.org/task/?14497). If
this task is still "postponed" when you are reading this and you are
interested to help, your contributions would be very welcome. Both
Gnuastro and GSL developers are very busy, hence both would appreciate
your help.
12.3.21 Matching (‘match.h’)
----------------------------
Matching is often necessary when two measurements of the same points
have been done using different instruments (or hardware), different
software or different configurations of the same software. In other
words, you have two catalogs or tables, and each has N columns
containing the N-dimensional "coordinate" values of each point. Each
table can have other columns too, for example, one can have magnitudes
in one filter, and another can have morphology measurements.
The matching functions here will use the coordinate columns of the
two tables to find a permutation for each, and the total number of
matched rows ($N_{match}$). This will enable you to match by the
positions if you like. At a higher level, you can apply the permutation
to the magnitude or morphology columns to merge the catalogs over the
$N_{match}$ rows. The input and output data formats of the functions
are the some and described below before the actual functions. Each
function also has extra arguments due to the particular algorithm it
uses for the matching.
The two inputs of the functions (‘coord1’ and ‘coord2’) must be *note
List of gal_data_t::. Each ‘gal_data_t’ node in ‘coord1’ or ‘coord2’
should be a single dimensional dataset (column in a table) and all the
nodes (in each) must have the same number of elements (rows). In other
words, each column can be visualized as having the coordinates of each
point in its respective dimension. The dimensions of the coordinates is
determined by the number of ‘gal_data_t’ nodes in the two input lists
(which must be equal). The number of rows (or the number of elements in
each ‘gal_data_t’) in the columns of ‘coord1’ and ‘coord2’ can (and,
usually will!) be different. In summary, these functions will be happy
if you use ‘gal_table_read’ to read the two coordinate columns from a
file, see *note Table input output::.
The functions below return a simply-linked list of three 1D datasets
(see *note List of gal_data_t::), let's call the returned dataset ‘ret’.
The first two (‘ret’ and ‘ret->next’) are permutations. In other words,
the ‘array’ elements of both have a type of ‘size_t’, see *note
Permutations::. The third node (‘ret->next->next’) is the calculated
distance for that match and its array has a type of ‘double’. The
number of matches will be put in the space pointed by the ‘nummatched’
argument. If there was not any match, this function will return ‘NULL’.
The two permutations can be applied to the rows of the two inputs:
the first one (‘ret’) should be applied to the rows of the table
containing ‘coord1’ and the second one (‘ret->next’) to the table
containing ‘coord2’. After applying the returned permutations to the
inputs, the top ‘nummatched’ elements of both will match with each
other. The ordering of the rest of the elements is undefined (depends
on the matching function used). The third node is the distances between
the respective match (which may be elliptical distance, see discussion
of "aperture" below).
The functions will not simply return the nearest neighbor as a match.
This is because the nearest neighbor may be too far to be a meaningful!
They will check the distance between the nearest neighbor of each point
and only return a match if it is within an acceptable N-dimensional
distance (or "aperture"). The matching aperture is defined by the
‘aperture’ array that is an input argument to the functions.
If several points of one catalog lie within this aperture of a point
in the other catalog, the nearest is defined as the match. In a 2D
situation (where the input lists have two nodes), for the most generic
case, ‘aperture’ must have three elements: the major axis length, axis
ratio and position angle (see *note Defining an ellipse and
ellipsoid::). If ‘aperture[1]==1’, the aperture will be a circle of
radius ‘aperture[0]’ and the third value will not be used. When the
aperture is an ellipse, distances between the points are also calculated
in the respective elliptical distances ($r_{el}$ in *note Defining an
ellipse and ellipsoid::).
*Output permutations ignore internal sorting*: the output
permutations will correspond to the initial inputs. Therefore, even
when ‘inplace!=0’ (and this function re-arranges the inputs in place),
the output permutation will correspond to original (possibly non-sorted)
inputs. The reason for this is that you rarely want to permute the
actual positional columns after the match. Usually, you also have other
columns (such as the magnitude and morphology) and you want to find how
they differ between the objects that match. Once you have the
permutations, they can be applied to those other columns (see *note
Permutations::) and the higher-level processing can continue. So if you
do not need the coordinate columns for the rest of your analysis, it is
better to set ‘inplace=1’.
-- Function:
gal_data_t *
gal_match_sort_based (gal_data_t *coord1, gal_data_t *coord2,
double *aperture, int sorted_by_first, int inplace, size_t
minmapsize, int quietmmap, size_t *nummatched)
Use a basic sort-based match to find the matching points of two
input coordinates. See the descriptions above on the format of the
inputs and outputs. To speed up the search, this function will
sort the input coordinates by their first column (first axis). If
_both_ are already sorted by their first column, you can avoid the
sorting step by giving a non-zero value to ‘sorted_by_first’.
When sorting is necessary and ‘inplace’ is non-zero, the actual
input columns will be sorted. Otherwise, an internal copy of the
inputs will be made, used (sorted) and later freed before
returning. Therefore, when ‘inplace==0’, inputs will remain
untouched, but this function will take more time and memory. If
internal allocation is necessary and the space is larger than
‘minmapsize’, the space will be not allocated in the RAM, but in a
file, see description of ‘--minmapsize’ and ‘--quietmmap’ in *note
Processing options::.
-- Function:
gal_data_t *
gal_match_kdtree (gal_data_t *coord1, gal_data_t *coord2,
gal_data_t *coord1_kdtree, size_t kdtree_root, double
*aperture, size_t numthreads, size_t minmapsize, int
quietmmap, size_t *nummatched)
Use the k-d tree concept for finding matches between two catalogs,
optionally in parallel (on ‘numthreads’ threads). The k-d tree of
the first input (‘coord1_kdtree’), and its root index
(‘kdtree_root’), should be constructed and found before calling
this function, to do this, you can use the ‘gal_kdtree_create’ of
*note K-d tree::. The desired ‘aperture’ array is the same as
‘gal_match_sort_based’ and described at the top of this section.
If ‘coord1_kdtree==NULL’, this function will return a ‘NULL’
pointer and write a value of ‘0’ in the space that ‘nummatched’
points to.
The final number of matches is returned in ‘nummatched’ and the
format of the returned dataset (three columns) is described above.
If internal allocation is necessary and the space is larger than
‘minmapsize’, the space will be not allocated in the RAM, but in a
file, see description of ‘--minmapsize’ and ‘--quietmmap’ in *note
Processing options::.
12.3.22 Statistical operations (‘statistics.h’)
-----------------------------------------------
After reading a dataset into memory from a file or fully simulating it
with another process, the most common processes that will be done on it
are statistical operations to let you quantify different aspects of the
data. the functions in this section describe Gnuastro's current set of
tools for this job. All these functions can work on any numeric data
type natively (see *note Numeric data types::) and can also work on
tiles over a dataset. Hence the inputs and outputs are in Gnuastro's
*note Generic data container::.
-- Macro: GAL_STATISTICS_SIG_CLIP_MAX_CONVERGE
The maximum number of clips, when $\sigma$-clipping should be done
by convergence. If the clipping does not converge before making
this many clips, all $\sigma$-clipping outputs will be NaN.
-- Macro: GAL_STATISTICS_MODE_GOOD_SYM
The minimum acceptable symmetricity of the mode calculation. If
the symmetricity of the derived mode is less than this value, all
the returned values by ‘gal_statistics_mode’ will have a value of
NaN.
-- Macro: GAL_STATISTICS_BINS_INVALID
-- Macro: GAL_STATISTICS_BINS_REGULAR
-- Macro: GAL_STATISTICS_BINS_IRREGULAR
Macros used to identify if the regularity of the bins when defining
bins.
-- Macro: GAL_STATISTICS_CLIP_OUTCOL_STD
-- Macro: GAL_STATISTICS_CLIP_OUTCOL_MAD
-- Macro: GAL_STATISTICS_CLIP_OUTCOL_MEAN
-- Macro: GAL_STATISTICS_CLIP_OUTCOL_MEDIAN
-- Macro: GAL_STATISTICS_CLIP_OUTCOL_NUMBER_USED
-- Macro: GAL_STATISTICS_CLIP_OUTCOL_NUMBER_CLIPS
Macros containing the index of the clipping outputs, see the
descriptions of ‘gal_statistics_clip_sigma’ below.
-- Macro: GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_STD
-- Macro: GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_MAD
-- Macro: GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_MEAN
Macros containing bit flags for optional clipping outputs, see the
descriptions of ‘gal_statistics_clip_sigma’ below.
-- Function:
gal_data_t *
gal_statistics_number (gal_data_t *input)
Return a single-element dataset with type ‘size_t’ which contains
the number of non-blank elements in ‘input’.
-- Function:
gal_data_t *
gal_statistics_minimum (gal_data_t *input)
Return a single-element dataset containing the minimum non-blank
value in ‘input’. The numerical datatype of the output is the same
as ‘input’.
-- Function:
gal_data_t *
gal_statistics_maximum (gal_data_t *input)
Return a single-element dataset containing the maximum non-blank
value in ‘input’. The numerical datatype of the output is the same
as ‘input’.
-- Function:
gal_data_t *
gal_statistics_sum (gal_data_t *input)
Return a single-element (‘double’ or ‘float64’) dataset containing
the sum of the non-blank values in ‘input’.
-- Function:
gal_data_t *
gal_statistics_mean (gal_data_t *input)
Return a single-element (‘double’ or ‘float64’) dataset containing
the mean of the non-blank values in ‘input’.
-- Function:
gal_data_t *
gal_statistics_std (gal_data_t *input)
Return a single-element (‘double’ or ‘float64’) dataset containing
the standard deviation of the non-blank values in ‘input’.
-- Function:
gal_data_t *
gal_statistics_mean_std (gal_data_t *input)
Return a two-element (‘double’ or ‘float64’) dataset containing the
mean and standard deviation of the non-blank values in ‘input’.
The first element of the returned dataset is the mean and the
second is the standard deviation.
This function will calculate both values in one pass over the
dataset. Hence when both the mean and standard deviation of a
dataset are necessary, this function is much more efficient than
calling ‘gal_statistics_mean’ and ‘gal_statistics_std’ separately.
-- Function:
double
gal_statistics_std_from_sums (double sum, double sump2, size_t
num)
Return the standard deviation from the values that can be obtained
in a single pass through the distribution: ‘sum’: the sum of the
elements, ‘sump2’: the sum of the power-of-2 of each element, and
‘num’: the number of elements.
This is a low-level function that is only useful after the
distribution of values has been parsed (and the three input
arguments are calculated). It is the lower-level function that is
used in functions like ‘gal_statistics_std’, or other components of
Gnuastro that measure the standard deviation (for example,
MakeCatalog's ‘--std’ column).
-- Function:
gal_data_t *
gal_statistics_median (gal_data_t *input, int inplace)
Return a single-element dataset containing the median of the
non-blank values in ‘input’. The numerical datatype of the output
is the same as ‘input’.
Calculating the median involves sorting the dataset and removing
blank values, for better performance (and less memory usage), you
can give a non-zero value to the ‘inplace’ argument. In this case,
the sorting and removal of blank elements will be done directly on
the input dataset. However, after this function the original
dataset may have changed (if it was not sorted or had blank
values).
-- Function:
gal_data_t *
gal_statistics_mad (gal_data_t *input, int inplace)
Return a single-element dataset with same type as input, containing
the median absolute deviation (MAD) of the non-blank values in
‘input’.
If ‘inplace==0’, the input dataset will remain untouched.
Otherwise, the MAD calculation will be done on the input dataset
without allocating a new one (its values will be changed after this
function). This is good when you do not need the input after this
function and avoid taking extra RAM and CPU.
-- Function:
gal_data_t *
gal_statistics_median_mad (gal_data_t *input, int inplace)
Return a two-element dataset with same type as input, containing
the median and median absolute deviation (MAD) of the non-blank
values in ‘input’.
If ‘inplace==0’, the input dataset will remain untouched.
Otherwise, the MAD calculation will be done on the input dataset
without allocating a new one (its values will be changed after this
function). This is good when you do not need the input after this
function and avoid taking extra RAM and CPU.
-- Function:
size_t
gal_statistics_quantile_index (size_t size, double quantile)
Return the index of the element that has a quantile of ‘quantile’
assuming the dataset has ‘size’ elements.
-- Function:
gal_data_t *
gal_statistics_quantile (gal_data_t *input, double quantile,
int inplace)
Return a single-element dataset containing the value with in a
quantile ‘quantile’ of the non-blank values in ‘input’. The
numerical datatype of the output is the same as ‘input’. See
‘gal_statistics_median’ for a description of ‘inplace’.
-- Function:
size_t
gal_statistics_quantile_function_index (gal_data_t *input,
gal_data_t *value, int inplace)
Return the index of the quantile function (inverse quantile) of
‘input’ at ‘value’. In other words, this function will return the
index of the nearest element (of a sorted and non-blank) ‘input’ to
‘value’. If the value is outside the range of the input, then this
function will return ‘GAL_BLANK_SIZE_T’.
-- Function:
gal_data_t *
gal_statistics_quantile_function (gal_data_t *input,
gal_data_t *value, int inplace)
Return a single-element dataset containing the quantile function of
the non-blank values in ‘input’ at ‘value’ (a single-element
dataset). The numerical data type is of the returned dataset is
‘float64’ (or ‘double’). In other words, this function will return
the quantile of ‘value’ in ‘input’. ‘value’ has to have the same
type as ‘input’. See ‘gal_statistics_median’ for a description of
‘inplace’.
When all elements are blank, the returned value will be NaN. If the
value is smaller than the input's smallest element, the returned
value will be negative infinity. If the value is larger than the
input's largest element, then the returned value will be positive
infinity
-- Function:
gal_data_t *
gal_statistics_unique (gal_data_t *input, int inplace)
Return a 1D dataset with the same numeric data type as the input,
but only containing its unique elements and without any (possible)
blank/NaN elements. Note that the input's number of dimensions is
irrelevant for this function. If ‘inplace’ is not zero, then the
unique values will over-write the allocated space of the input,
otherwise a new space will be allocated and the input will not be
touched.
-- Function:
int
gal_statistics_has_negative (gal_data_t *input)
Return ‘1’ if the input dataset contains a negative number and ‘0’
otherwise. If the dataset doesn't have a numeric type (as in a
string), this function will abort with, saying that it does not
recognize the file type.
-- Function:
gal_data_t *
gal_statistics_mode (gal_data_t *input, float mirrordist, int
inplace)
Return a four-element (‘double’ or ‘float64’) dataset that contains
the mode of the ‘input’ distribution. This function implements the
non-parametric algorithm to find the mode that is described in
Appendix C of Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664).
In short it compares the actual distribution and its "mirror
distribution" to find the mode. In order to be efficient, you can
determine how far the comparison goes away from the mirror through
the ‘mirrordist’ parameter (think of it as a multiple of
sigma/error). See ‘gal_statistics_median’ for a description of
‘inplace’.
The output array has the following elements (in the given order,
note that counting in C starts from 0).
array[0]: mode
array[1]: mode quantile.
array[2]: symmetricity.
array[3]: value at the end of symmetricity.
-- Function:
gal_data_t *
gal_statistics_mode_mirror_plots (gal_data_t *input,
gal_data_t *value, size_t numbins, int inplace, double
*mirror_val)
Make a mirrored histogram and cumulative frequency plot (with
‘numbins’) with the mirror distribution of the ‘input’ having a
value in ‘value’. If all the input elements are blank, or the
mirror value is outside the range of the input, this function will
return a ‘NULL’ pointer.
The output is a list of data structures (see *note List of
gal_data_t::): the first is the bins with one bin at the mirror
point, the second is the histogram with a maximum of one and the
third is the cumulative frequency plot (with a maximum of one).
-- Function:
int
gal_statistics_is_sorted (gal_data_t *input, int updateflags)
Return ‘0’ if the input is not sorted, if it is sorted, this
function will return ‘1’ and ‘2’ if it is increasing or decreasing,
respectively. This function will abort with an error if ‘input’
has zero elements and will return ‘1’ (sorted, increasing) when
there is only one element. This function will only look into the
dataset if the ‘GAL_DATA_FLAG_SORT_CH’ bit of ‘input->flag’ is ‘0’,
see *note Generic data container::.
When the flags do not indicate a previous check _and_ ‘updateflags’
is non-zero, this function will set the flags appropriately to
avoid having to re-check the dataset in future calls (this can be
very useful when repeated checks are necessary). When
‘updateflags==0’, this function has no side-effects on the dataset:
it will not toggle the flags.
If you want to re-check a dataset with the blank-value-check flag
already set (for example, if you have made changes to it), then
explicitly set the ‘GAL_DATA_FLAG_SORT_CH’ bit to zero before
calling this function. When there are no other flags, you can
simply set the flags to zero (with ‘input->flag=0’), otherwise you
can use this expression:
input->flag &= ~GAL_DATA_FLAG_SORT_CH;
-- Function:
void
gal_statistics_sort_increasing (gal_data_t *input)
Sort the input dataset (in place) in an increasing order and toggle
the sort-related bit flags accordingly.
-- Function:
void
gal_statistics_sort_decreasing (gal_data_t *input)
Sort the input dataset (in place) in a decreasing order and toggle
the sort-related bit flags accordingly.
-- Function:
gal_data_t *
gal_statistics_no_blank_sorted (gal_data_t *input, int
inplace)
Remove all the blanks and sort the input dataset. If ‘inplace’ is
non-zero this will happen on the input dataset (in the allocated
space of the input dataset). However, if ‘inplace’ is zero, this
function will allocate a new copy of the dataset and work on that.
Therefore if ‘inplace==0’, the input dataset will be modified.
This function uses the bit flags of the input, so if you have
modified the dataset, set ‘input->flag=0’ before calling this
function. Also note that ‘inplace’ is only for the dataset
elements. Therefore even when ‘inplace==0’, if the input is
already sorted _and_ has no blank values, then the flags will be
updated to show this.
If all the elements were blank, then the returned dataset's ‘size’
will be zero. This is thus a good parameter to check after calling
this function to see if there actually were any non-blank elements
in the input or not and take the appropriate measure. This can
help avoid strange bugs in later steps. The flags of a zero-sized
returned dataset will indicate that it has no blanks and is sorted
in an increasing order. Even if having blank values or being
sorted is not defined on a zero-element dataset, it is up to the
caller to choose what they will do with a zero-element dataset.
The flags have to be set after this function any way.
-- Function:
gal_data_t *
gal_statistics_regular_bins (gal_data_t *input, gal_data_t
*inrange, size_t numbins, double onebinstart)
Generate an array of regularly spaced elements as a 1D array
(column) of type ‘double’ (i.e., ‘float64’, it has to be double to
account for small differences on the bin edges). The input
arguments are described below
‘input’
The dataset you want to apply the bins to. This is only
necessary if the range argument is not complete, see below.
If ‘inrange’ has all the necessary information, you can pass a
‘NULL’ pointer for this.
‘inrange’
This dataset keeps the desired range along each dimension of
the input data structure, it has to be in ‘float’ (i.e.,
‘float32’) type.
• If you want the full range of the dataset (in any
dimensions, then just set ‘inrange’ to ‘NULL’ and the
range will be specified from the minimum and maximum
value of the dataset (‘input’ cannot be ‘NULL’ in this
case).
• If there is one element for each dimension in range, then
it is viewed as a quantile (Q), and the range will be: 'Q
to 1-Q'.
• If there are two elements for each dimension in range,
then they are assumed to be your desired minimum and
maximum values. When either of the two are NaN, the
minimum and maximum will be calculated for it.
‘numbins’
The number of bins: must be larger than 0.
‘onebinstart’
A desired value to start one bin. Note that with this option,
the bins will not start and end exactly on the given range
values, it will be slightly shifted to accommodate this
request (enough for the bin containing the value to start at
it). If you do not have any preference on where to start a
bin, set this to NAN.
-- Function:
gal_data_t *
gal_statistics_histogram (gal_data_t *input, gal_data_t *bins,
int normalize, int maxone)
Make a histogram of all the elements in the given dataset with bin
values that are defined in the ‘bins’ structure (see
‘gal_statistics_regular_bins’, they currently have to be equally
spaced). The returned histogram is a 1-D ‘gal_data_t’ of type
‘GAL_TYPE_FLOAT32’, with the same number of elements as ‘bins’.
For each bin, it will contain the number of input elements that
fell inside of that bin.
Let's write the center of the $i$th element of the bin array as
$b_i$, and the fixed half-bin width as $h$. Then element $j$ of
the input array ($in_j$) will be counted in $b_i$ if $(b_i-h) \le
in_j < (b_i+h)$. However, if $in_j$ is somewhere in the last bin,
the condition changes to $(b_i-h) \le in_j \le (b_i+h)$.
If ‘normalize!=0’, the histogram will be "normalized" such that the
sum of the counts column will be one. In other words, all the
counts in every bin will be divided by the total number of counts.
If ‘maxone!=0’, the histogram's maximum count will be 1. In other
words, the counts in every bin will be divided by the value of the
maximum. In both of these cases, the output dataset will have a
‘GAL_DATA_FLOAT32’ datatype.
-- Function:
gal_data_t *
gal_statistics_histogram2d (gal_data_t *input, gal_data_t
*bins)
This function is very similar to ‘gal_statistics_histogram’, but
will build a 2D histogram (count how many of the elements of
‘input’ are a within a 2D box. The bins comprising the first
dimension of the 2D box are defined by ‘bins’. The bins of the
second dimension are defined by ‘bins->next’ (‘bins’ is a *note
List of gal_data_t::). Both the ‘bin’ and ‘bin->next’ can be
created with ‘gal_statistics_regular_bins’.
This function returns a list of ‘gal_data_t’ with three
nodes/columns, so you can directly write them into a table (see
*note Table input output::). Assuming ‘bins’ has $N1$ bins and
‘bins->next’ has $N2$ bins, each node/column of the returned output
is a 1D array with $N1\times N2$ elements. The first and second
columns are the center of the 2D bin along the first and second
dimensions and have a ‘double’ data type. The third column is the
2D histogram (the number of input elements that have a value within
that 2D bin) and has a ‘uint32’ data type (see *note Numeric data
types::).
-- Function:
gal_data_t *
gal_statistics_cfp (gal_data_t *input, gal_data_t *bins, int
normalize)
Make a cumulative frequency plot (CFP) of all the elements in
‘input’ with bin values that are defined in the ‘bins’ structure
(see ‘gal_statistics_regular_bins’).
The CFP is built from the histogram: in each bin, the value is the
sum of all previous bins in the histogram. Thus, if you have
already calculated the histogram before calling this function, you
can pass it onto this function as the data structure in
‘bins->next’ (see ‘List of gal_data_t’). If ‘bin->next!=NULL’,
then it is assumed to be the histogram. If it is ‘NULL’, then the
histogram will be calculated internally and freed after the job is
finished.
When a histogram is given and it is normalized, the CFP will also
be normalized (even if the normalized flag is not set here): note
that a normalized CFP's maximum value is 1.
-- Function:
gal_data_t *
gal_statistics_concentration (gal_data_t *input, double width,
int inplace)
Return the concentration around the median for the input
distribution. For more on the algorithm and ‘width’, see the
description of ‘--concentration’ in *note Single value
measurements::.
If ‘inplace!=0’, then this function will use the actual allocated
space of the input data and will not internally allocate a new
dataset (which can have memory and CPU benefits); but will alter
(sort and remove blank elements from) your input dataset.
-- Function:
gal_data_t *
gal_statistics_clip_sigma (gal_data_t *input, float multip,
float param, float extrastats, int inplace, int quiet)
Apply $\sigma$-clipping on a given dataset and return a dataset
that contains the results. For a description of $\sigma$-clipping
see *note Sigma clipping::. ‘multip’ is the multiple of the
standard deviation (or $\sigma$, that is used to define outliers in
each round of clipping).
The role of ‘param’ is determined based on its value. If ‘param’
is larger than ‘1’ (one), it must be an integer and will be
interpreted as the number clips to do. If it is less than ‘1’
(one), it is interpreted as the tolerance level to stop the
iteration.
The returned dataset (let's call it ‘out’) contains a 6-element
array with type ‘GAL_TYPE_FLOAT32’. Through the
‘GAL_STATISTICS_CLIP_OUTCOL_*’ macros below, you can access any
particular measurement.
out=gal_statistics_clip_sigma(input, ....);
float *array=out->array;
array[ GAL_STATISTICS_CLIP_OUTCOL_NUMBER_USED ]
array[ GAL_STATISTICS_CLIP_OUTCOL_MEAN ]
array[ GAL_STATISTICS_CLIP_OUTCOL_STD ]
array[ GAL_STATISTICS_CLIP_OUTCOL_MEDIAN ]
array[ GAL_STATISTICS_CLIP_OUTCOL_MAD ]
array[ GAL_STATISTICS_CLIP_OUTCOL_NUMBER_CLIPS ]
However, note that all are not measured by default! Since the mean
and MAD are not necessary during sigma-clipping, if you want them,
you have to set the following two bit flags in the ‘extrastats’
argument as below.
int extrastats=0; /* To initialize all bits */
/* If you want the sigma-clipped MAD. */
extrastats |= GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_MAD;
/* If you want the sigma-clipped mean. */
extrastats |= GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_MEAN;
If the $\sigma$-clipping does not converge or all input elements
are blank, then this function will return NaN values for all the
elements above.
-- Function:
gal_data_t *
gal_statistics_clip_mad (gal_data_t *input, float multip,
float param, uint8_t extrastats, int inplace, int quiet)
Similar to ‘gal_statistics_clip_sigma’, but will do median absolute
deviation (MAD) based clipping, see *note MAD clipping::.
The only difference is that for this function the MAD is
automatically calculated during clipping. It is the mean and
standard deviation that will not be calculated unless requested
with the ‘GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_MEAN’ and
‘GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_STD’ bit flats respectively.
-- Function:
gal_data_t *
gal_statistics_outlier_bydistance (int pos1_neg0, gal_data_t
*input, size_t window_size, float sigma, float sigclip_multip,
float sigclip_param, int inplace, int quiet)
Find the first positive outlier (if ‘pos1_neg0!=0’) in the ‘input’
distribution. When ‘pos1_neg0==0’, the same algorithm goes to the
start of the dataset. The returned dataset contains a single
element: the first positive outlier. It is one of the dataset's
elements, in the same type as the input. If the process fails for
any reason (for example, no outlier was found), a ‘NULL’ pointer
will be returned.
All (possibly existing) blank elements are first removed from the
input dataset, then it is sorted. A sliding window of
‘window_size’ elements is parsed over the dataset. Starting from
the ‘window_size’-th element of the dataset, in the direction of
increasing values. This window is used as a reference. The first
element where the distance to the previous (sorted) element is
‘sigma’ units away from the distribution of distances in its window
is considered an outlier and returned by this function.
Formally, if we assume there are $N$ non-blank elements. They are
first sorted. Searching for the outlier starts on element $W$.
Let's take $v_i$ to be the $i$-th element of the sorted input (with
no blank values) and $m$ and $\sigma$ as the $\sigma$-clipped
median and standard deviation from the distances of the previous
$W$ elements (not including $v_i$). If the value given to ‘sigma’
is displayed with $s$, the $i$-th element is considered as an
outlier when the condition below is true.
$${(v_i-v_{i-1})-m\over \sigma}>s$$
The ‘sigclip_multip’ and ‘sigclip_param’ arguments specify the
properties of the $\sigma$-clipping (see *note Sigma clipping:: for
more). You see that by this definition, the outlier cannot be any
of the lower half elements. The advantage of this algorithm
compared to $\sigma$-clippign is that it only looks backwards (in
the sorted array) and parses it in one direction.
If ‘inplace!=0’, the removing of blank elements and sorting will be
done within the input dataset's allocated space. Otherwise, this
function will internally allocate (and later free) the necessary
space to keep the intermediate space that this process requires.
If ‘quiet!=0’, this function will report the parameters every time
it moves the window as a separate line with several columns. The
first column is the value, the second (in square brackets) is the
sorted index, the third is the distance of this element from the
previous one. The Fourth and fifth (in parenthesis) are the median
and standard deviation of the $\sigma$-clipped distribution within
the window and the last column is the difference between the third
and fourth, divided by the fifth.
-- Function:
gal_data_t *
gal_statistics_outlier_flat_cfp (gal_data_t *input, size_t
numprev, float sigclip_multip, float sigclip_param, float
thresh, size_t numcontig, int inplace, int quiet, size_t
*index)
Return the first element in the given dataset where the cumulative
frequency plot first becomes significantly flat for a sufficient
number of elements. The returned dataset only has one element
(with the same type as the input). If ‘index!=NULL’, the index
(counting from zero, after sorting the dataset and removing any
blanks) is written in the space that ‘index’ points to. If no
sufficiently flat portion is found, the returned pointer will be
‘NULL’.
The flatness on the cumulative frequency plot is defined like this
(see *note Histogram and Cumulative Frequency Plot::): on the
sorted dataset, for every point ($a_i$), we calculate
$d_i=a_{i+2}-a_{i-2}$. This done on the first $N$ elements (value
of ‘numprev’). After element $a_{N+2}$, we start estimating the
flatness as follows: for every element we use the $N$, $d_i$
measurements before it as the reference. Let's call this set $D_i$
for element $i$. The $\sigma$-clipped median ($m$) and standard
deviation ($s$) of $D_i$ are then calculated. The
$\sigma$-clipping can be configured with the two ‘sigclip_param’
and ‘sigclip_multip’ arguments.
Taking $t$ as the significance threshold (value to ‘thresh’), a
point is considered flat when $a_i>m+t\sigma$. But a single point
satisfying this condition will probably just be due to noise. To
make a more robust estimate, this significance/condition has to
hold for ‘numcontig’ contiguous elements after $a_i$. When this is
satisfied, $a_i$ is returned as the point where the distribution's
cumulative frequency plot becomes flat.
To get a good estimate of $m$ and $s$, it is thus recommended to
set ‘numprev’ as large as possible. However, be careful not to set
it too high: the checks in the paragraph above are not done on the
first ‘numprev’ elements and this function assumes the flatness
occurs after them. Also, be sure that the value to ‘numcontig’ is
much less than ‘numprev’, otherwise $\sigma$-clipping may not be
able to remove the immediate outliers in $D_i$ near the boundary of
the flat region.
When ‘quiet==0’, the basic measurements done on each element are
printed on the command-line (good for finding the best parameters).
When ‘inplace!=0’, the sorting and removal of blank elements is
done on the input dataset, so the input may be altered after this
function.
12.3.23 Fitting functions (‘fit.h’)
-----------------------------------
After doing a measurement, it is usually necessary to parameterize the
relation that has been found. The functions in this section are
wrappers over the GNU Scientific Library (GSL) Linear Least-Squares
Fitting (https://www.gnu.org/software/gsl/doc/html/lls.html), to make
them easily accessible using Gnuastro's *note Generic data container::.
The respective GSL function is mentioned under each function.
-- Global integer: GAL_FIT_INVALID
-- Global integer: GAL_FIT_LINEAR
-- Global integer: GAL_FIT_LINEAR_WEIGHTED
-- Global integer: GAL_FIT_LINEAR_NO_CONSTANT
-- Global integer: GAL_FIT_LINEAR_NO_CONSTANT_WEIGHTED
-- Global integer: GAL_FIT_POLYNOMIAL
-- Global integer: GAL_FIT_POLYNOMIAL_WEIGHTED
-- Global integer: GAL_FIT_POLYNOMIAL_NUMBER
Identifiers for the various types of fitting functions. These can
be used by the callers of these functions to select between various
fitting types. They can easily be converted to, and from, fixed
human-readable strings using the ‘gal_fit_name_*’ functions below.
The last one ‘GAL_FIT_ROBUST_NUMBER’ is the total number of
available fitting methods (can be used to add more macros in the
calling program and to avoid overlaps with existing codes).
-- Global integer: GAL_FIT_ROBUST_INVALID
-- Global integer: GAL_FIT_ROBUST_DEFAULT
-- Global integer: GAL_FIT_ROBUST_BISQUARE
-- Global integer: GAL_FIT_ROBUST_CAUCHY
-- Global integer: GAL_FIT_ROBUST_FAIR
-- Global integer: GAL_FIT_ROBUST_HUBER
-- Global integer: GAL_FIT_ROBUST_OLS
-- Global integer: GAL_FIT_ROBUST_WELSCH
-- Global integer: GAL_FIT_ROBUST_NUMBER
Identifiers for the various types of robust polynomial fitting
functions. For a description of each, see
.
The last one ‘GAL_FIT_ROBUST_NUMBER’ is the total number of
available functions (can be used to add more macros in the calling
program and to avoid overlaps with existing codes).
-- Function:
uint8_t
gal_fit_name_to_id (char *name)
Return the internal code of a standard human-readable name for the
various fitting functions. If the name is not recognized, the
returned value will be ‘GAL_FIT_INVALID’.
-- Function:
char *
gal_fit_name_from_id (uint8_t fitid)
Return a standard human-readable name for the fitting function
identified with the ‘fitid’ (read as "fitting ID"). If the fitting
ID couldn't be recognized, a NULL pointer is returned.
-- Function:
uint8_t
gal_fit_name_robust_to_id (char *name)
Return the internal code of a standard human-readable name for the
various robust fitting types. If the name is not recognized, the
returned value will be ‘GAL_FIT_INVALID’.
-- Function:
char *
gal_fit_name_robust_from_id (uint8_t robustid)
Return a standard human-readable name for the input robust fitting
type. If the fitting ID couldn't be recognized, a NULL pointer is
returned.
-- Function:
gal_data_t *
gal_fit_1d_linear (gal_data_t *xin, gal_data_t *yin,
gal_data_t *ywht)
Preform a 1D linear regression fit with a constant term(1) in the
form of $Y=c_0+c_1X$. The input ‘xin’ contains the independent
variable values and ‘yin’ contains the measured variable values for
each independent variable. When ‘ywht!=NULL’, it is assumed to
contain the "weight" of each Y measurement (if you don't have
weights on your measured values, simply set this to ‘NULL’). The
weight of each measurement is the inverse of its variance. For a
Gaussian error distribution with standard deviation $\sigma$, the
weight is therefore $1/\sigma^2$.
If any of the values in any of the inputs is blank (NaN in floating
point), the final fitted parameters will all be NaN. To remove rows
with a NaN/blank, you can use ‘gal_blank_remove_rows’ (which will
remove all rows with a blank values in any of the columns with a
single call).
The output is a single dataset with a ‘GAL_TYPE_FLOAT64’ type with
6 elements:
1. $c_0$: the constant in $Y=c_0+c_1X$.
2. $c_1$: the multiple in $Y=c_0+c_1X$.
3. First element of variance-covariance matrix.
4. Second and third (which are equal) elements of the
variance-covariance matrix.
5. Fourth element of the variance-covariance matrix.
6. The reduced $\chi^2$ of the fit.
-- Function:
gal_data_t *
gal_fit_1d_linear_no_constant (gal_data_t *xin, gal_data_t
*yin, gal_data_t *ywht)
Preform a 1D linear regression fit _without_ a constant term(2),
formally: $Y=c_1X$. The input ‘xin’ contains the independent
variable values and ‘yin’ contains the measured variable values for
each independent variable. When ‘ywht!=NULL’, it is assumed to
contain the "weight" of each Y measurement (if you don't have
weights on your measured values, simply set this to ‘NULL’). The
weight of each measurement is the inverse of its variance. For a
Gaussian error distribution with standard deviation $\sigma$, the
weight is therefore $1/\sigma^2$.
If any of the values in any of the inputs is blank (NaN in floating
point), the final fitted parameters will all be NaN. To remove rows
with a NaN/blank, you can use ‘gal_blank_remove_rows’ (which will
remove all rows with a blank values in any of the columns with a
single call).
The output is a single dataset with a ‘GAL_TYPE_FLOAT64’ type with
3 elements:
1. $c_1$: the multiple in $Y=c_0+c_1X$.
2. Variance of $c_1$.
3. The reduced $\chi^2$ of the fit.
-- Function:
gal_data_t *
gal_fit_1d_linear_estimate (gal_data_t *fit, gal_data_t *xin)
Given a linear least squares fit output (‘fit’), estimate the fit
on an arbitrary number of independent variable (horizontal axis, or
X, in an X-Y plot) within ‘xin’. ‘fit’ is assumed to be the output
of either ‘gal_fit_1d_linear’ or ‘gal_fit_1d_linear_no_constant’.
In case you haven't used those functions to obtain the constants
and covariance matrix elements, see the description of those
functions for the expected format of ‘fit’.
This function returns two columns (as a *note List of
gal_data_t::): The top node of the list is the estimated values at
the input X-axis positions, and the next node is the errors in the
estimation. Naturally, both have the same number of elements as
‘xin’. Being a list, helps in easily printing the output columns
to a table (see *note Table input output::).
-- Function:
gal_data_t *
gal_fit_1d_polynomial (gal_data_t *xin, gal_data_t *yin,
gal_data_t *ywht, size_t maxpower, double *redchisq)
Preform a 1D polynomial fit, formally:
$Y=c+0+c_1X+c_2X^2+\cdots+c_nX^n$ (using GSL's multi-parameter
regression(3)). The largest power of $X$ is determined with the
‘maxpower’ argument (which is $n$ in the equation above). The
reduced $\chi^2$ of the fit is written in the space that
‘*redchisq’ points to.
The input ‘xin’ contains the independent variable values and the
input ‘yin’ contains the measured variable values for each
independent variable. When ‘ywht!=NULL’, it is assumed to contain
the "weight" of each Y measurement (if you don't have weights on
your measured values, simply set this to ‘NULL’). The weight of
each measurement is the inverse of its variance. For a Gaussian
error distribution with standard deviation $\sigma$, the weight is
therefore $1/\sigma^2$.
If any of the values in any of the inputs is blank (NaN in floating
point), the final fitted parameters will all be NaN. To remove rows
with a NaN/blank, you can use ‘gal_blank_remove_rows’ (which will
remove all rows with a blank values in any of the columns with a
single call).
The output of this function is a list of two datasets, linked as a
list (as a *note List of gal_data_t::). Both have a
‘GAL_TYPE_FLOAT64’ type, and are described below (in order).
1. A one dimensional and contains $n+1$ elements (for the $n+1$
constants that have been found $(c_0, c_1, c_2, \cdots, c_n)$.
2. A two dimensional variance-covariance matrix with
$(n+1)\times(n+1)$ elements.
-- Function:
gal_data_t *
gal_fit_1d_polynomial_robust (gal_data_t *xin, gal_data_t
*yin, size_t maxpower, uint8_t robustid, double *redchisq)
Preform a 1D robust polynomial fit, formally:
$Y=c+0+c_1X+c_2X^2+\cdots+c_nX^n$ (using GSL's robust linear
regression(4)). See the description there for the details.
The inputs and outputs of this function are almost identical to
‘gal_fit_1d_polynomial’, with the difference that you need to
specify the function to reject outliers through the ‘robustid’
input argument. You can pass any of the ‘GAL_FIT_ROBUST_*’ codes
defined at the top of this section to this (the names are identical
to the names in GSL).
-- Function:
gal_data_t *
gal_fit_1d_polynomial_estimate (gal_data_t *fit, gal_data_t
*xin)
Given a 1D polynomial fit output (‘fit’), estimate the fit on an
arbitrary number of independent variable (horizontal axis, or X, in
an X-Y plot) within ‘xin’. ‘fit’ is assumed to be the output of
‘gal_fit_1d_polynomial’. In case you haven't used this function to
obtain the constants and covariance matrix, see the description of
that function for the expected format of ‘fit’.
This function returns two columns (as a *note List of
gal_data_t::): The top node of the list is the estimated values at
the input X-axis positions, and the next node is the errors in the
estimation. Naturally, both have the same number of elements as
‘xin’. Being a list, helps in easily printing the output columns
to a table (see *note Table input output::).
---------- Footnotes ----------
(1)
(2)
(3)
(4)
12.3.24 Binary datasets (‘binary.h’)
------------------------------------
Binary datasets only have two (usable) values: 0 (also known as
background) or 1 (also known as foreground). They are created after
some binary classification is applied to the dataset. The most common
is thresholding: for example, in an image, pixels with a value above the
threshold are given a value of 1 and those with a value less than the
threshold are assigned a value of 0.
Since there is only two values, in the processing of binary images,
you are usually concerned with the positioning of an element and its
vicinity (neighbors). When a dataset has more than one dimension,
multiple classes of immediate neighbors (that are touching the element)
can be defined for each data-element. To separate these different
classes of immediate neighbors, we define _connectivity_.
The classification is done by the distance from element center to the
neighbor's center. The nearest immediate neighbors have a connectivity
of 1, the second nearest class of neighbors have a connectivity of 2 and
so on. In total, the largest possible connectivity for data with ‘ndim’
dimensions is ‘ndim’. For example, in a 2D dataset, 4-connected
neighbors (that share an edge and have a distance of 1 pixel) have a
connectivity of 1. The other 4 neighbors that only share a vertice
(with a distance of $\sqrt{2}$ pixels) have a connectivity of 2.
Conventionally, the class of connectivity-2 neighbors also includes the
connectivity 1 neighbors, so for example, we call them 8-connected
neighbors in 2D datasets.
Ideally, one bit is sufficient for each element of a binary dataset.
However, CPUs are not designed to work on individual bits, the smallest
unit of memory addresses is a byte (containing 8 bits on modern CPUs).
Therefore, in Gnuastro, the type used for binary dataset is ‘uint8_t’
(see *note Numeric data types::). Although it does take 8-times more
memory, this choice offers much better performance and the some extra
(useful) features.
The advantage of using a full byte for each element of a binary
dataset is that you can also have other values (that will be ignored in
the processing). One such common "other" value in real datasets is a
blank value (to mark regions that should not be processed because there
is no data). The constant ‘GAL_BLANK_UINT8’ value must be used in these
cases (see *note Library blank values::). Another is some temporary
value(s) that can be given to a processed pixel to avoid having another
copy of the dataset as in ‘GAL_BINARY_TMP_VALUE’ that is described
below.
-- Macro: GAL_BINARY_TMP_VALUE
The functions described below work on a ‘uint8_t’ type dataset with
values of 1 or 0 (no other pixel will be touched). However, in
some cases, it is necessary to put temporary values in each element
during the processing of the functions. This temporary value has a
special meaning for the operation and will be operated on. So if
your input datasets have values other than 0 and 1 that you do not
want these functions to work on, be sure they are not equal to this
macro's value. Note that this value is also different from
‘GAL_BLANK_UINT8’, so your input datasets may also contain blank
elements.
-- Function:
gal_data_t *
gal_binary_erode (gal_data_t *input, size_t num, int
connectivity, int inplace)
Do ‘num’ erosions on the ‘connectivity’-connected neighbors of
‘input’ (see above for the definition of connectivity).
If ‘inplace’ is non-zero _and_ the input's type is
‘GAL_TYPE_UINT8’, then the erosion will be done within the input
dataset and the returned pointer will be ‘input’. Otherwise,
‘input’ is copied (and converted if necessary) to ‘GAL_TYPE_UINT8’
and erosion will be done on this new dataset which will also be
returned. This function will only work on the elements with a
value of 1 or 0. It will leave all the rest unchanged.
Erosion (inverse of dilation) is an operation in mathematical
morphology where each foreground pixel that is touching a
background pixel is flipped (changed to background). The
‘connectivity’ value determines the definition of "touching".
Erosion will thus decrease the area of the foreground regions by
one layer of pixels.
-- Function:
gal_data_t *
gal_binary_dilate (gal_data_t *input, size_t num, int
connectivity, int inplace)
Do ‘num’ dilations on the ‘connectivity’-connected neighbors of
‘input’ (see above for the definition of connectivity). For more
on ‘inplace’ and the output, see ‘gal_binary_erode’.
Dilation (inverse of erosion) is an operation in mathematical
morphology where each background pixel that is touching a
foreground pixel is flipped (changed to foreground). The
‘connectivity’ value determines the definition of "touching".
Dilation will thus increase the area of the foreground regions by
one layer of pixels.
-- Function:
gal_data_t *
gal_binary_open (gal_data_t *input, size_t num, int
connectivity, int inplace)
Do ‘num’ openings on the ‘connectivity’-connected neighbors of
‘input’ (see above for the definition of connectivity). For more
on ‘inplace’ and the output, see ‘gal_binary_erode’.
Opening is an operation in mathematical morphology which is defined
as erosion followed by dilation (see above for the definitions of
erosion and dilation). Opening will thus remove the outer
structure of the foreground. In this implementation, ‘num’
erosions are going to be applied on the dataset, then ‘num’
dilations.
-- Function:
gal_data_t *
gal_binary_number_neighbors (gal_data_t *input, int
connectivity, int inplace)
Return an image of the same size as the input, but where each
non-zero and non-blank input pixel is replaced with the number of
its non-zero and non-blank neighbors. The input dataset is assumed
to be binary (having an unsigned, 8-bit dataset). The neighbors
are defined through the ‘connectivity’ argument (see above) and if
‘inplace!=0’, then the output will be written into the input.
-- Function:
size_t
gal_binary_connected_components (gal_data_t *binary,
gal_data_t **out, int connectivity)
Return the number of connected components in ‘binary’ through the
breadth first search algorithm (finding all pixels belonging to one
component before going on to the next). Connection between two
pixels is defined based on the value to ‘connectivity’. ‘out’ is a
dataset with the same size as ‘binary’ with ‘GAL_TYPE_INT32’ type.
Every pixel in ‘out’ will have the label of the connected component
it belongs to. The labeling of connected components starts from 1,
so a label of zero is given to the input's background pixels.
When ‘*out!=NULL’ (its space is already allocated), it will be
cleared (to zero) at the start of this function. Otherwise, when
‘*out==NULL’, the necessary dataset to keep the output will be
allocated by this function.
‘binary’ must have a type of ‘GAL_TYPE_UINT8’, otherwise this
function will abort with an error. Other than blank pixels (with a
value of ‘GAL_BLANK_UINT8’ defined in *note Library blank
values::), all other non-zero pixels in ‘binary’ will be considered
as foreground (and will be labeled). Blank pixels in the input
will also be blank in the output.
-- Function:
gal_data_t *
gal_binary_connected_indexs(gal_data_t *binary, int
connectivity)
Build a ‘gal_data_t’ linked list, where each node of the list
contains an array with indices of the connected regions. Therefore
the arrays of each node can have a different size. Note that the
indices will only be calculated on the pixels with a value of 1 and
internally, it will temporarily change the values to 2 (and return
them back to 1 in the end).
-- Function:
gal_data_t *
gal_binary_connected_adjacency_matrix (gal_data_t *adjacency,
size_t *numconnected)
Find the number of connected labels and new labels based on an
adjacency matrix, which must be a square binary array (type
‘GAL_TYPE_UINT8’). The returned dataset is a list of new labels
for each old label. In other words, this function will find the
objects that are connected (possibly through a third object) and in
the output array, the respective elements for all input labels is
going to have the same value. The total number of connected labels
is put into the space that ‘numconnected’ points to.
An adjacency matrix defines connection between two labels. For
example, let's assume we have 5 labels and we know that labels 1
and 5 are connected to label 3, but are not connected with each
other. Also, labels 2 and 4 are not touching any other label. So
in total we have 3 final labels: one combined object (merged from
labels 1, 3, and 5) and the initial labels 2 and 4. The input
adjacency matrix would look like this (note the extra row and
column for a label 0 which is ignored):
INPUT OUTPUT
===== ======
in_lab 1 2 3 4 5 |
| numconnected = 3
0 0 0 0 0 0 |
in_lab 1 --> 0 0 0 1 0 0 |
in_lab 2 --> 0 0 0 0 0 0 | Returned: new labels for the
in_lab 3 --> 0 1 0 0 0 1 | 5 initial objects
in_lab 4 --> 0 0 0 0 0 0 | | 0 | 1 | 2 | 1 | 3 | 1 |
in_lab 5 --> 0 0 0 1 0 0 |
Although the adjacency matrix as used here is symmetric, currently
this function assumes that it is filled on both sides of the
diagonal.
-- Function:
gal_data_t *
gal_binary_connected_adjacency_list (gal_list_sizet_t
**listarr, size_t number, size_t minmapsize, int quietmmap,
size_t *numconnected)
Find the number of connected labels and new labels based on an
adjacency list. The output of this function is identical to that
of ‘gal_binary_connected_adjacency_matrix’. But the major
difference is that it uses a list of connected labels to each label
instead of a square adjacency matrix. This is done because when
the number of labels becomes very large (for example, on the scale
of 100,000), the adjacency matrix can consume more than 10GB of
RAM!
The input list has the following format: it is an array of pointers
to ‘gal_list_sizet_t *’ (or ‘gal_list_sizet_t **’). The array has
‘number’ elements and each ‘listarr[i]’ is a linked list of
‘gal_list_sizet_t *’. As a demonstration, the input of the same
example in ‘gal_binary_connected_adjacency_matrix’ would look like
below and the output of this function will be identical to there.
listarr[0] = NULL
listarr[1] = 3
listarr[2] = NULL
listarr[3] = 1 -> 5
listarr[4] = NULL
listarr[5] = 3
From this example, it is already clear that this method will
consume far less memory. But because it needs to parse lists (and
not easily jump between array elements), it can be slower. But in
scenarios where there are too many objects (that may exceed the
whole system's RAM+SWAP), this option is a good alternative and the
drop in processing speed is worth getting the job done.
Similar to ‘gal_binary_connected_adjacency_matrix’, this function
will write the final number of connected labels in ‘numconnected’.
But since it takes no ‘gal_data_t *’ argument (where it can inherit
the ‘minmapsize’ and ‘quietmmap’ parameters), it also needs these
as input. For more on ‘minmapsize’ and ‘quietmmap’, see *note
Memory management::.
-- Function:
gal_data_t *
gal_binary_holes_label (gal_data_t *input, int connectivity,
size_t *numholes)
Label all the holes in the foreground (non-zero elements in input)
as independent regions. Holes are background regions (zero-valued
in input) that are fully surrounded by the foreground, as defined
by ‘connectivity’. The returned dataset has a 32-bit signed
integer type with the size of the input. All holes in the input
will have labels/counters greater or equal to ‘1’. The rest of the
background regions will still have a value of ‘0’ and the initial
foreground pixels will have a value of ‘-1’. The total number of
holes will be written where ‘numholes’ points to.
-- Function:
void
gal_binary_holes_fill (gal_data_t *input, int connectivity,
size_t maxsize)
Fill all the holes (0 valued pixels surrounded by 1 valued pixels)
of the binary ‘input’ dataset. The connectivity of the holes can
be set with ‘connectivity’. Holes larger than ‘maxsize’ are not
filled. This function currently only works on a 2D dataset.
12.3.25 Labeled datasets (‘label.h’)
------------------------------------
A labeled dataset is one where each element/pixel has an integer label
(or counter). The label identifies the group/class that the element
belongs to. This form of labeling allows the higher-level study of all
pixels within a certain class.
For example, to detect objects/targets in an image/dataset, you can
apply a threshold to separate the noise from the signal (to detect
diffuse signal, a threshold is useless and more advanced methods are
necessary, for example *note NoiseChisel::). But the output of
detection is a binary dataset (which is just a very low-level labeling
of ‘0’ for noise and ‘1’ for signal).
The raw detection map is therefore hardly useful for any kind of
analysis on objects/targets in the image. One solution is to use a
connected-components algorithm (see ‘gal_binary_connected_components’ in
*note Binary datasets::). It is a simple and useful way to
separate/label connected patches in the foreground. This higher-level
(but still elementary) labeling therefore allows you to count how many
connected patches of signal there are in the dataset and is a major
improvement compared to the raw detection.
However, when your objects/targets are touching, the simple connected
components algorithm is not enough and a still higher-level labeling
mechanism is necessary. This brings us to the necessity of the
functions in this part of Gnuastro's library. The main inputs to the
functions in this section are already labeled datasets (for example,
with the connected components algorithm above).
Each of the labeled regions are independent of each other (the labels
specify different classes of targets). Therefore, especially in large
datasets, it is often useful to process each label on independent CPU
threads in parallel rather than in series. Therefore the functions of
this section actually use an array of pixel/element indices (belonging
to each label/class) as the main identifier of a region. Using indices
will also allow processing of overlapping labels (for example, in
deblending problems). Just note that overlapping labels are not yet
implemented, but planned. You can use ‘gal_label_indexs’ to generate
lists of indices belonging to separate classes from the labeled input.
-- Macro: GAL_LABEL_INIT
-- Macro: GAL_LABEL_RIVER
-- Macro: GAL_LABEL_TMPCHECK
Special negative integer values used internally by some of the
functions in this section. Recall that meaningful labels are
considered to be positive integers ($\geq1$). Zero is
conventionally kept for regions with no labels, therefore negative
integers can be used for any extra classification in the labeled
datasets.
-- Function:
gal_data_t *
gal_label_indexs (gal_data_t *labels, size_t numlabs, size_t
minmapsize, int quietmmap)
Return an array of ‘gal_data_t’ containers, each containing the
pixel indices of the respective label (see *note Generic data
container::). ‘labels’ contains the label of each element and has
to have an ‘GAL_TYPE_INT32’ type (see *note Library data types::).
Only positive (greater than zero) values in ‘labels’ will be
used/indexed, other elements will be ignored.
Meaningful labels start from ‘1’ and not ‘0’, therefore the output
array of ‘gal_data_t’ will contain ‘numlabs+1’ elements. The first
(zero-th) element of the output (‘indexs[0]’ in the example below)
will be initialized to a dataset with zero elements. This will
allow easy (non-confusing) access to the indices of each
(meaningful) label.
‘numlabs’ is the number of labels in the dataset. If it is given a
value of zero, then the maximum value in the input (largest label)
will be found and used. Therefore if it is given, but smaller than
the actual number of labels, this function may/will crash (it will
write in un-allocated space). ‘numlabs’ is therefore useful in a
highly optimized/checked environment.
For example, if the returned array is called ‘indexs’, then
‘indexs[10].size’ contains the number of elements that have a label
of ‘10’ in ‘labels’ and ‘indexs[10].array’ is an array (after
casting to ‘size_t *’) containing the indices of each one of those
elements/pixels.
By _index_ we mean the 1D position: the input number of dimensions
is irrelevant (any dimensionality is supported). In other words,
each element's index is the number of elements/pixels between it
and the dataset's first element/pixel. Therefore it is always
greater or equal to zero and stored in ‘size_t’ type.
-- Function:
size_t
gal_label_watershed (gal_data_t *values, gal_data_t *indexs,
gal_data_t *label, size_t *topinds, int min0_max1)
Use the watershed algorithm(1) to "over-segment" the pixels in the
‘indexs’ dataset based on values in the ‘values’ dataset.
Internally, each local extrema (maximum or minimum, based on
‘min0_max1’) and its surrounding pixels will be given a unique
label. For demonstration, see Figures 8 and 9 of Akhlaghi and
Ichikawa 2015 (http://arxiv.org/abs/1505.01664). If
‘topinds!=NULL’, it is assumed to point to an already allocated
space to write the index of each clump's local extrema, otherwise,
it is ignored.
The ‘values’ dataset must have a 32-bit floating point type
(‘GAL_TYPE_FLOAT32’, see *note Library data types::) and will only
be read by this function. ‘indexs’ must contain the indices of the
elements/pixels that will be over-segmented by this function and
have a ‘GAL_TYPE_SIZE_T’ type, see the description of
‘gal_label_indexs’, above. The final labels will be written in the
respective positions of ‘labels’, which must have a
‘GAL_TYPE_INT32’ type and be the same size as ‘values’.
When ‘indexs’ is already sorted, this function will ignore
‘min0_max1’. To judge if the dataset is sorted or not (by the
values the indices correspond to in ‘values’, not the actual
indices), this function will look into the bits of ‘indexs->flag’,
for the respective bit flags, see *note Generic data container::.
If ‘indexs’ is not already sorted, this function will sort it
according to the values of the respective pixel in ‘values’. The
increasing/decreasing order will be determined by ‘min0_max1’.
Note that if this function is called on multiple threads _and_
‘values’ points to a different array on each thread, this function
will not return a reasonable result. In this case, please sort
‘indexs’ prior to calling this function (see
‘gal_qsort_index_multi_d’ in *note Qsort functions::).
When ‘indexs’ is decreasing (increasing), or ‘min0_max1’ is ‘1’
(‘0’), local minima (maxima), are considered rivers (watersheds)
and given a label of ‘GAL_LABEL_RIVER’ (see above).
Note that rivers/watersheds will also be formed on the edges of the
labeled regions or when the labeled pixels touch a blank pixel.
Therefore this function will need to check for the presence of
blank values. To be most efficient, it is thus recommended to use
‘gal_blank_present’ (with ‘updateflag=1’) prior to calling this
function (see *note Library blank values::. Once the flag has been
set, no other function (including this one) that needs special
behavior for blank pixels will have to parse the dataset to see if
it has blank values any more.
If you are sure your dataset does not have blank values (by the
design of your software), to avoid an extra parsing of the dataset
and improve performance, you can set the two bits manually (see the
description of ‘flags’ in *note Generic data container::):
input->flag |= GAL_DATA_FLAG_BLANK_CH; /* Set bit to 1. */
input->flag &= ~GAL_DATA_FLAG_HASBLANK; /* Set bit to 0. */
-- Function:
void
gal_label_clump_significance (gal_data_t *values, gal_data_t
*std, gal_data_t *label, gal_data_t *indexs, struct
gal_tile_two_layer_params *tl, size_t numclumps, size_t
minarea, int variance, int keepsmall, gal_data_t *sig,
gal_data_t *sigind)
This function is usually called after ‘gal_label_watershed’, and is
used as a measure to identify which over-segmented "clumps" are
real and which are noise.
A measurement is done on each clump (using the ‘values’ and ‘std’
datasets, see below). To help in multi-threaded environments, the
operation is only done on pixels which are indexed in ‘indexs’. It
is expected for ‘indexs’ to be sorted by their values in ‘values’.
If not sorted, the measurement may not be reliable. If sorted in a
decreasing order, then clump building will start from their highest
value and vice-versa. See the description of ‘gal_label_watershed’
for more on ‘indexs’.
Each "clump" (identified by a positive integer) is assumed to be
surrounded by at least one river/watershed pixel (with a
non-positive label). This function will parse the pixels
identified in ‘indexs’ and make a measurement on each clump and
over all the river/watershed pixels. The number of clumps
(‘numclumps’) must be given as an input argument and any clump that
is smaller than ‘minarea’ is ignored (because of scatter). If
‘variance’ is non-zero, then the ‘std’ dataset is interpreted as
variance, not standard deviation.
The ‘values’ and ‘std’ datasets must have a ‘float’ (32-bit
floating point) type. Also, ‘label’ and ‘indexs’ must respectively
have ‘int32’ and ‘size_t’ types. ‘values’ and ‘label’ must have
the same size, but ‘std’ can have three possible sizes: 1) a single
element (which will be used for the whole dataset, 2) the same size
as ‘values’ (so a different error can be assigned to every pixel),
3) a single value for each tile, based on the ‘tl’ tessellation
(see *note Tile grid::). In the last case, a tile/value will be
associated to each clump based on its flux-weighted (only positive
values) center.
The main output is an internally allocated, 1-dimensional array
with one value per label. The array information (length, type,
etc.) will be written into the ‘sig’ generic data container.
Therefore ‘sig->array’ must be ‘NULL’ when this function is called.
After this function, the details of the array (number of elements,
type and size, etc) will be written in to the various components of
‘sig’, see the definition of ‘gal_data_t’ in *note Generic data
container::. Therefore ‘sig’ must already be allocated before
calling this function.
Optionally (when ‘sigind!=NULL’, similar to ‘sig’) the clump labels
of each measurement in ‘sig’ will be written in ‘sigind->array’.
If ‘keepsmall’ zero, small clumps (where no measurement is made)
will not be included in the output table.
This function is initially intended for a multi-threaded
environment. In such cases, you will be writing arrays of clump
measures from different regions in parallel into an array of
‘gal_data_t’s. You can simply allocate (and initialize), such an
array with the ‘gal_data_array_calloc’ function in *note Arrays of
datasets::. For example, if the ‘gal_data_t’ array is called
‘array’, you can pass ‘&array[i]’ as ‘sig’.
Along with some other functions in ‘label.h’, this function was
initially written for *note Segment::. The description of the
parameter used to measure a clump's significance is fully given in
Akhlaghi 2019 (https://arxiv.org/abs/1909.11230).
-- Function:
void
gal_label_grow_indexs (gal_data_t *labels, gal_data_t *indexs,
int withrivers, int connectivity)
Grow the (positive) labels of ‘labels’ over the pixels in ‘indexs’
(see description of ‘gal_label_indexs’). The pixels (position in
‘indexs’, values in ‘labels’) that must be "grown" must have a
value of ‘GAL_LABEL_INIT’ in ‘labels’ before calling this function.
For a demonstration see Columns 2 and 3 of Figure 10 in Akhlaghi
and Ichikawa 2015 (http://arxiv.org/abs/1505.01664).
In many aspects, this function is very similar to over-segmentation
(watershed algorithm, ‘gal_label_watershed’). The big difference
is that in over-segmentation local maximums (that are not touching
any already labeled pixel) get a separate label. However, here the
final number of labels will not change. All pixels that are not
directly touching a labeled pixel just get pushed back to the start
of the loop, and the loop iterates until its size does not change
any more. This is because in a generic scenario some of the
indexed pixels might not be reachable through other indexed pixels.
The next major difference with over-segmentation is that when there
is only one label in growth region(s), it is not mandatory for
‘indexs’ to be sorted by values. If there are multiple labeled
regions in growth region(s), then values are important and you can
use ‘qsort’ with ‘gal_qsort_index_single_d’ to sort the indices by
values in a separate array (see *note Qsort functions::).
This function looks for positive-valued neighbors of each pixel in
‘indexs’ and will label a pixel if it touches one. Therefore, it
is very important that only pixels/labels that are intended for
growth have positive values in ‘labels’ before calling this
function. Any non-positive (zero or negative) value will be
ignored as a label by this function. Thus, it is recommended that
while filling in the ‘indexs’ array values, you initialize all the
pixels that are in ‘indexs’ with ‘GAL_LABEL_INIT’, and set
non-labeled pixels that you do not want to grow to ‘0’.
This function will write into both the input datasets. After this
function, some of the non-positive ‘labels’ pixels will have a new
positivelabel and the number of useful elements in ‘indexs’ will
have decreased. The index of those pixels that could not be
labeled will remain inside ‘indexs’. If ‘withrivers’ is non-zero,
then pixels that are immediately touching more than one positive
value will be given a ‘GAL_LABEL_RIVER’ label.
Note that the ‘indexs->array’ is not re-allocated to its new size
at the end(2). But since ‘indexs->dsize[0]’ and ‘indexs->size’
have new values after this function is returned, the extra elements
just will not be used until they are ultimately freed by
‘gal_data_free’.
Connectivity is a value between ‘1’ (fewest number of neighbors)
and the number of dimensions in the input (most number of
neighbors). For example, in a 2D dataset, a connectivity of ‘1’
and ‘2’ corresponds to 4-connected and 8-connected neighbors.
---------- Footnotes ----------
(1) The watershed algorithm was initially introduced by Vincent and
Soille (https://doi.org/10.1109/34.87344). It starts from the minima
and puts the pixels in, one by one, to grow them until the touch (create
a watershed). For more, also see the Wikipedia article:
.
(2) Note that according to the GNU C Library, even a ‘realloc’ to a
smaller size can also cause a re-write of the whole array, which is not
a cheap operation.
12.3.26 Convolution functions (‘convolve.h’)
--------------------------------------------
Convolution is a very common operation during data analysis and is
thoroughly described as part of Gnuastro's *note Convolve:: program
which is fully devoted to this job. Because of the complete
introduction that was presented there, we will directly skip onto the
currently available convolution functions in Gnuastro's library.
As of this version, only spatial domain convolution is available in
Gnuastro's libraries. We have not had the time to liberate the
frequency domain function convolution and deconvolution functions that
are available in the Convolve program(1).
-- Function:
gal_data_t *
gal_convolve_spatial (gal_data_t *tiles, gal_data_t *kernel,
size_t numthreads, int edgecorrection, int convoverch, int
conv_on_blank)
Convolve the given ‘tiles’ dataset (possibly a list of tiles, see
*note List of gal_data_t:: and *note Tessellation library::) with
‘kernel’ on ‘numthreads’ threads. When ‘edgecorrection’ is
non-zero, it will correct for the edge dimming effects as discussed
in *note Edges in the spatial domain::. When ‘conv_on_blank’ is
non-zero, this function will also attempt convolution over the
blank pixels (and therefore give values to the blank pixels that
are near non-blank pixels).
‘tiles’ can be a single/complete dataset, but in that case the
speed will be very slow. Therefore, for larger images, it is
recommended to give a list of tiles covering a dataset. To create
a tessellation that fully covers an input image, you may use
‘gal_tile_full’, or ‘gal_tile_full_two_layers’ to also define
channels over your input dataset. These functions are discussed in
*note Tile grid::. You may then pass the list of tiles to this
function. This is the recommended way to call this function
because spatial domain convolution is slow and breaking the job
into many small tiles and working on simultaneously on several
threads can greatly speed up the processing.
If the tiles are defined within a channel (a larger tile), by
default convolution will be done within the channel, so pixels on
the edge of a channel will not be affected by their neighbors that
are in another channel. See *note Tessellation:: for the necessity
of channels in astronomical data analysis. This behavior may be
disabled when ‘convoverch’ is non-zero. In this case, it will
ignore channel borders (if they exist) and mix all pixels that
cover the kernel within the dataset.
-- Function:
void
gal_convolve_spatial_correct_ch_edge (gal_data_t *tiles,
gal_data_t *kernel, size_t numthreads, int edgecorrection, int
conv_on_blank, gal_data_t *tocorrect)
Correct the edges of channels in an already convolved image when it
was initially convolved with ‘gal_convolve_spatial’ and
‘convoverch==0’. In that case, strong boundaries might exist on
the channel edges. So if you later need to remove those boundaries
at later steps of your processing, you can call this function. It
will only do convolution on the tiles that are near the edge and
were effected by the channel borders. Other pixels in the image
will not be touched. Hence, it is much faster. When
‘conv_on_blank’ is non-zero, this function will also attempt
convolution over the blank pixels (and therefore give values to the
blank pixels that are near non-blank pixels).
---------- Footnotes ----------
(1) Hence any help would be greatly appreciated.
12.3.27 Pooling functions (‘pool.h’)
------------------------------------
Pooling is the process of reducing the complexity of the input image
(its size and variation of pixel values). Its underlying concepts, and
an analysis of its usefulness, is fully described in *note Pooling
operators::. The following functions are available pooling in Gnuastro.
Just note that unlike the Arithmetic operators, the output of these
functions should contain a correct WCS in their output.
-- Function:
gal_data_t *
gal_pool_max (gal_data_t *input, size_t psize, size_t
numthreads)
Return the max-pool of ‘input’, assuming a pool size of ‘psize’
pixels. The number of threads to use can be set with ‘numthreads’.
-- Function:
gal_data_t *
gal_pool_min (gal_data_t *input, size_t psize, size_t
numthreads)
Return the min-pool of ‘input’, assuming a pool size of ‘psize’
pixels. The number of threads to use can be set with ‘numthreads’.
-- Function:
gal_data_t *
gal_pool_sum (gal_data_t *input, size_t psize, size_t
numthreads)
Return the sum-pool of ‘input’, assuming a pool size of ‘psize’
pixels. The number of threads to use can be set with ‘numthreads’.
-- Function:
gal_data_t *
gal_pool_mean (gal_data_t *input, size_t psize, size_t
numthreads)
Return the mean-pool of ‘input’, assuming a pool size of ‘psize’
pixels. The number of threads to use can be set with ‘numthreads’.
-- Function:
gal_data_t *
gal_pool_median (gal_data_t *input, size_t psize, size_t
numthreads)
Return the median-pool of ‘input’, assuming a pool size of ‘psize’
pixels. The number of threads to use can be set with ‘numthreads’.
12.3.28 Interpolation (‘interpolate.h’)
---------------------------------------
During data analysis, it happens that parts of the data cannot be given
a value, but one is necessary for the higher-level analysis. For
example, a very bright star saturated part of your image and you need to
fill in the saturated pixels with some values. Another common usage
case are masked sky-lines in 1D spectra that similarly need to be
assigned a value for higher-level analysis. In other situations, you
might want a value in an arbitrary point: between the elements/pixels
where you have data. The functions described in this section are for
such operations.
The parametric interpolations discussed below are wrappers around the
interpolation functions of the GNU Scientific Library (or GSL, see *note
GNU Scientific Library::). To identify the different GSL interpolation
types, Gnuastro's ‘gnuastro/interpolate.h’ header file contains macros
that are discussed below. The GSL wrappers provided here are not yet
complete because we are too busy. If you need them, please consider
helping us in adding them to Gnuastro's library. Your contributions
would be very welcome and appreciated.
-- Macro: GAL_INTERPOLATE_NEIGHBORS_METRIC_RADIAL
-- Macro: GAL_INTERPOLATE_NEIGHBORS_METRIC_MANHATTAN
-- Macro: GAL_INTERPOLATE_NEIGHBORS_METRIC_INVALID
The metric used to find distance for nearest neighbor
interpolation. A radial metric uses the simple Euclidean function
to find the distance between two pixels. A manhattan metric will
always be an integer and is like steps (but is also much faster to
calculate than radial metric because it does not need a square root
calculation).
-- Macro: GAL_INTERPOLATE_NEIGHBORS_FUNC_MIN
-- Macro: GAL_INTERPOLATE_NEIGHBORS_FUNC_MAX
-- Macro: GAL_INTERPOLATE_NEIGHBORS_FUNC_MEAN
-- Macro: GAL_INTERPOLATE_NEIGHBORS_FUNC_MEDIAN
-- Macro: GAL_INTERPOLATE_NEIGHBORS_FUNC_INVALID
The various types of nearest-neighbor interpolation functions for
‘gal_interpolate_neighbors’. The names are descriptive for the
operation they do, so we will not go into much more detail here.
The median operator will be one of the most used, but operators
like the maximum are good to fill the center of saturated stars.
-- Function:
gal_data_t *
gal_interpolate_neighbors (gal_data_t *input, struct
gal_tile_two_layer_params *tl, uint8_t metric, size_t
numneighbors, size_t numthreads, int onlyblank, int
aslinkedlist, int function)
Interpolate the values in the input dataset using a calculated
statistics from the distribution of their ‘numneighbors’ closest
neighbors. The desired statistics is determined from the ‘func’
argument, which takes any of the ‘GAL_INTERPOLATE_NEIGHBORS_FUNC_’
macros (see above). This function is non-parametric and thus
agnostic to the input's number of dimension or shape of the
distribution.
Distance can be defined on different metrics that are identified
through ‘metric’ (taking values determined by the
‘GAL_INTERPOLATE_NEIGHBORS_METRIC_’ macros described above). If
‘onlyblank’ is non-zero, then only blank elements will be
interpolated and pixels that already have a value will be left
untouched. This function is multi-threaded and will run on
‘numthreads’ threads (see ‘gal_threads_number’ in *note
Multithreaded programming::).
‘tl’ is Gnuastro's tessellation structure used to define tiles over
an image and is fully described in *note Tile grid::. When
‘tl!=NULL’, then it is assumed that the ‘input->array’ contains one
value per tile and interpolation will respect certain tessellation
properties, for example, to not interpolate over channel borders.
If several datasets have the same set of blank values, you do not
need to call this function multiple times. When ‘aslinkedlist’ is
non-zero, then ‘input’ will be seen as a *note List of
gal_data_t::. In this case, the same neighbors will be used for
all the datasets in the list. Of course, the values for each
dataset will be different, so a different value will be written in
each dataset, but the neighbor checking that is the most CPU
intensive part will only be done once.
This is a non-parametric and robust function for interpolation.
The interpolated values are also always within the range of the
non-blank values and strong outliers do not get created. However,
this type of interpolation must be used with care when there are
gradients. This is because it is non-parametric and if there are
not enough neighbors, step-like features can be created.
-- Macro: GAL_INTERPOLATE_1D_INVALID
This is just a place-holder to manage errors.
-- Macro: GAL_INTERPOLATE_1D_LINEAR
[From GSL:] Linear interpolation. This interpolation method does
not require any additional memory.
-- Macro: GAL_INTERPOLATE_1D_POLYNOMIAL
[From GSL:] Polynomial interpolation. This method should only be
used for interpolating small numbers of points because polynomial
interpolation introduces large oscillations, even for well-behaved
datasets. The number of terms in the interpolating polynomial is
equal to the number of points.
-- Macro: GAL_INTERPOLATE_1D_CSPLINE
[From GSL:] Cubic spline with natural boundary conditions. The
resulting curve is piece-wise cubic on each interval, with matching
first and second derivatives at the supplied data-points. The
second derivative is chosen to be zero at the first point and last
point.
-- Macro: GAL_INTERPOLATE_1D_CSPLINE_PERIODIC
[From GSL:] Cubic spline with periodic boundary conditions. The
resulting curve is piece-wise cubic on each interval, with matching
first and second derivatives at the supplied data-points. The
derivatives at the first and last points are also matched. Note
that the last point in the data must have the same y-value as the
first point, otherwise the resulting periodic interpolation will
have a discontinuity at the boundary.
-- Macro: GAL_INTERPOLATE_1D_AKIMA
[From GSL:] Non-rounded Akima spline with natural boundary
conditions. This method uses the non-rounded corner algorithm of
Wodicka.
-- Macro: GAL_INTERPOLATE_1D_AKIMA_PERIODIC
[From GSL:] Non-rounded Akima spline with periodic boundary
conditions. This method uses the non-rounded corner algorithm of
Wodicka.
-- Macro: GAL_INTERPOLATE_1D_STEFFEN
[From GSL:] Steffen's method(1) guarantees the monotonicity of the
interpolating function between the given data points. Therefore,
minima and maxima can only occur exactly at the data points, and
there can never be spurious oscillations between data points. The
interpolated function is piece-wise cubic in each interval. The
resulting curve and its first derivative are guaranteed to be
continuous, but the second derivative may be discontinuous.
-- Function:
gsl_spline *
gal_interpolate_1d_make_gsl_spline (gal_data_t *X, gal_data_t
*Y, int type_1d)
Allocate and initialize a GNU Scientific Library (GSL) 1D
‘gsl_spline’ structure using the non-blank elements of ‘Y’.
‘type_1d’ identifies the interpolation scheme and must be one of
the ‘GAL_INTERPOLATE_1D_*’ macros defined above.
If ‘X==NULL’, the X-axis is assumed to be integers starting from
zero (the index of each element in ‘Y’). Otherwise, the values in
‘X’ will be used to initialize the interpolation structure. Note
that when given, ‘X’ must _not_ contain any blank elements and it
must be sorted (in increasing order).
Each interpolation scheme needs a minimum number of elements to
successfully operate. If the number of non-blank values in ‘Y’ is
less than this number, this function will return a ‘NULL’ pointer.
To be as generic and modular as possible, GSL's tools are
low-level. Therefore before doing the interpolation, many steps
are necessary (like preparing your dataset, then allocating and
initializing ‘gsl_spline’). The metadata available in Gnuastro's
*note Generic data container:: make it easy to hide all those
preparations within this function.
Once ‘gsl_spline’ has been initialized by this function, the
interpolation can be evaluated for any X value within the non-blank
range of the input using ‘gsl_spline_eval’ or ‘gsl_spline_eval_e’.
For example, in the small program below (‘sample-interp.c’), we
read the first two columns of the table in ‘table.txt’ and feed
them to this function to later estimate the values in the second
column for three selected points. You can use *note BuildProgram::
to compile and run this function, see *note Library demo programs::
for more.
Contents of the ‘table.txt’ file:
$ cat table.txt
0 0
1 2
3 6
4 8
6 12
8 16
9 18
Contents of the ‘sample-interp.c’ file:
#include
#include
#include
#include
int
main(void)
{
size_t i;
gal_data_t *X, *Y;
gsl_spline *spline;
gsl_interp_accel *acc;
gal_list_str_t *cols=NULL;
/* Change the values based on your input table. */
double points[]={1.8, 2.5, 7};
/* Read the first two columns from `tab.txt'.
IMPORTANT: the list is first-in-first-out, so the output
column order is the inverse of the input order. */
gal_list_str_add(&cols, "1", 0);
gal_list_str_add(&cols, "2", 0);
Y=gal_table_read("table.txt", NULL, NULL, cols,
GAL_TABLE_SEARCH_NAME, 0, 1, -1, 1, NULL);
X=Y->next;
/* Allocate the GSL interpolation accelerator and make the
`gsl_spline' structure. */
acc=gsl_interp_accel_alloc();
spline=gal_interpolate_1d_make_gsl_spline(X, Y,
GAL_INTERPOLATE_1D_STEFFEN);
/* Calculate the respective value for all the given points,
if `spline' could be allocated. */
if(spline)
for(i=0; i<(sizeof points)/(sizeof *points); ++i)
printf("%f: %f\n", points[i],
gsl_spline_eval(spline, points[i], acc));
/* Clean up and return. */
gal_data_free(X);
gal_data_free(Y);
gsl_spline_free(spline);
gsl_interp_accel_free(acc);
gal_list_str_free(cols, 0);
return EXIT_SUCCESS;
}
Compile and run this program with *note BuildProgram:: to see the
interpolation results for the three points within the program.
$ astbuildprog sample-interp.c --quiet
1.800000: 3.600000
2.500000: 5.000000
7.000000: 14.000000
-- Function:
void
gal_interpolate_1d_blank (gal_data_t *in, int type_1d)
Fill the blank elements of ‘in’ using the rest of the elements and
the given interpolation. The interpolation scheme can be set
through ‘type_1d’, which accepts any of the ‘GAL_INTERPOLATE_1D_*’
macros above. The interpolation is internally done in 64-bit
floating point type (‘double’). However the evaluated/interpolated
values (originally blank) will be written (in ‘in’) with its
original numeric datatype, using C's standard type conversion.
By definition, interpolation is only defined "between" valid
points. Therefore, if any number of elements on the start or end
of the 1D array are blank, those elements will not be interpolated
and will remain blank. To see if any blank (non-interpolated)
elements remain, you can use ‘gal_blank_present’ on ‘in’ after this
function is finished.
---------- Footnotes ----------
(1)
12.3.29 Warp library (‘warp.h’)
-------------------------------
Warping an image to a new pixel grid is commonly necessary as part of
astronomical data reduction, for an introduction, see *note Warp::. For
details of how we resample the old pixel grid to the new pixel grid, see
*note Resampling::. Gnuastro's Warp program uses the following
functions for its default mode (when no linear warps are requested).
Through the following functions, you can directly access those features
in your own custom programs. The linear warping operations of the Warp
program aren't yet brought into the library. If you need them please
get in touch with us at ‘bug-gnuastro@gnu.org’. For usage examples of
this library, please see *note Library demo - Warp to another image:: or
*note Library demo - Warp to new grid::.
You are free to provide any valid WCS keywords to the functions
defined in this library using the ‘gal_warp_wcsalign_t’ data type. This
might be used to align the input image to the standard WCS grid,
potentially changing the pixel scale, removing any valid WCS non-linear
distortion available, and projecting to any valid WCS projection type.
Further details of the warp library functions and parameters are shown
below:
-- Macro: GAL_WARP_OUTPUT_NAME_WARPED
-- Macro: GAL_WARP_OUTPUT_NAME_MAXFRAC
Names of the output datasets (in the ‘name’ component of the output
‘gal_data_t’s). By default the output is only a single dataset,
but when the ‘checkmaxfrac’ component of the input is non-zero, it
will contain two datasets.
-- Type (C struct): gal_warp_wcsalign_t
The main data container for inputs, output and internal variables
to simplify the WCS-aligning functions. Due to the large number of
input variables, this structure makes it easy to call the main
functions. Similar to ‘gal_data_t’, the ‘gal_warp_wcsalign_t’ is a
structure ‘typedef’'d as a new type, see *note Library data
container::. Please note that this structure has elements that are
_allocated_ dynamically and must be freed after usage.
‘gal_warp_wcsalign_free’ only frees the internal variables, so you
are responsible for freeing your own inputs (‘cdelt’, ‘input’,
etc.) and the output. The internal variables are cached here to
cut cpu-intensive computations. To prevent from using
uninitialized variables, we recommend using the helper function
‘gal_warp_wcsalign_template’ to get a clean structure before
setting your own variables. The structure and each of its elements
are defined below:
typedef struct
{
/* Arguments given (and later freed) by the caller. If 'twcs' is
given, then the "WCS To build" elements will be ignored. */
gal_data_t *input;
size_t numthreads;
double coveredfrac;
size_t edgesampling;
gal_data_t *widthinpix;
uint8_t checkmaxfrac;
struct wcsprm *twcs; /* WCS Predefined. */
gal_data_t *ctype; /* WCS To build. */
gal_data_t *cdelt; /* WCS To build. */
gal_data_t *center; /* WCS To build. */
/* Output (must be freed by caller) */
gal_data_t *output;
/* Internal variables (allocated and freed internally) */
size_t v0;
size_t nhor;
size_t ncrn;
size_t gcrn;
int isccw;
gal_data_t *vertices;
} gal_warp_wcsalign_t;
‘gal_data_t *input’
The input dataset. This dataset must contain both the image
array of type ‘GAL_TYPE_FLOAT64’, and ‘input->wcs’ should not
be ‘NULL’ for the WCS-aligning operations to work, see *note
Library demo - Warp to new grid::.
‘size_t numthreads’
Number of threads to use during the WCS aligning operations.
If the given value is ‘0’, the library will calculate the
number of available threads at run-time. The ‘warp’ library
functions are _thread-safe_ so you can freely enjoy the merits
of parallel processing.
‘double coveredfrac’
Acceptable fraction of output pixel that is covered by input
pixels. The value should be between 0 and 1 (inclusive). If
the area of an output pixel is covered by less than this
fraction, its value will be ‘NaN’. For more, see the
description of ‘--coveredfrac’ in *note Invoking astwarp::.
‘size_t edgesampling’
Set the number of extra vertices along each edge of the output
pixel's polygon to account for potential curvature due to
projection or distortion. A value of ‘0’ is usually enough
for this (so the pixel is only defined by a four vertice
polygon. Greater values increase memory usage and program
execution time. For more, please see the description of
‘--edgesampling’ in *note Align pixels with WCS considering
distortions::.
‘gal_data_t *widthinpix’
Output image size (width and height) in number of pixels. If
a ‘NULL’ pointer is passed, the WCS-aligning operations will
estimate the output image size internally such that it
contains the full input. This dataset should have a type of
‘GAL_TYPE_SIZE_T’ and contain exactly two _odd_ values. This
ensures that the center of the central pixel lies at the
requested central coordinate (note that an image with an even
number of pixels doesn't have a "central" pixel!
‘struct wcsprm *twcs’
The target grid WCS which must follow the standard WCSLIB
structure. You can read it from a file using ‘gal_wcs_read’
or create an entirely new one with ‘gal_wcs_create’ and later
free it with ‘gal_wcs_free’, see *note World Coordinate
System::. If this element is given, the ‘ctype’, ‘cdelt’ and
‘center’ elements (which are used to construct a WCS
internally) are ignored.
Please note that the ‘wcsprm’ structure doesn't contain the
image size. To set the final image size, you should use
‘widthinpix’.
‘gal_data_t *ctype’
The output's projection type. The dataset has to have the
type ‘GAL_TYPE_STRING’, containing exactly two strings. Both
strings will be directly passed to WCSLIB and should conform
to the FITS standard's ‘CTYPEi’ keywords, see the description
of ‘--ctype’ in *note Align pixels with WCS considering
distortions::. For example, ‘"RA---TAN"’ and ‘"DEC--TAN"’, or
‘"RA---HPX"’ and ‘"DEC--HPX"’.
‘gal_data_t *cdelt’
Output pixel scale (size of pixel in the WCS units: value to
‘CUNITi’ keywords in FITS, usually degrees). The dataset
should have a type of ‘GAL_TYPE_FLOAT64’ and contain exactly
two values. Hint: to convert arcsec to degrees, just divide
by 3600.
‘gal_data_t *center’
WCS coordinate of the center of the central pixel of the
output. The units depend on the WCS, for example, if the
‘CUNITi’ keywords are ‘deg’, it is in degrees. This dataset
should have a type of ‘GAL_TYPE_FLOAT64’ and contain exactly
two values.
‘uint8_t checkmaxfrac’
When this is non-zero, the output will be a two-element *note
List of gal_data_t::. The second element shows the Moiré
pattern (https://en.wikipedia.org/wiki/Moir%C3%A9_pattern) of
the warp. For more, see *note Moire pattern in stacking and
its correction::.
-- Function:
gal_warp_wcsalign_t
gal_warp_wcsalign_template (void)
A high-level helper function that returns a clean
‘gal_warp_wcsalign_t’ struct with all values initialized This
function returns a copy of a statically allocated structure. So
you don't need to free the returned structure.
The Warp library decides on the program flow based on this struct.
Uninitialized pointers can point to random space in RAM which can
create segmentation faults, or even worse, produce unnoticed
side-effects. It is therefore good practice to manually set unused
pointers to ‘NULL’ and give blank values to numbers Since there are
many variables and pointers in ‘gal_warp_wcsalign_t’, it is easy to
forget _initializing_ them. With that said, we recommend using
this function to minimize human error.
-- Function:
void
gal_warp_wcsalign (gal_warp_wcsalign_t *wa)
A high-level function to align the input dataset's pixels to its
WCS coordinates and write the result in ‘wa->output’. This
function assumes that the input variables have already been set in
the ‘wa’ structure. The input variables are clearly shown in the
definition of ‘gal_warp_wcsalign_t’. It will call the lower level
functions below to do the job and will free the internal variables
afterwards.
The following low-level functions are called from the high-level
‘gal_warp_wcsalign’ function. They are provided here in scenarios where
fine grain control over the thread workflow is necessary, see *note
Multithreaded programming::.
-- Function:
void
gal_warp_wcsalign_init (gal_warp_wcsalign_t *wa)
Low-level function to initialize all the elements inside the ‘wa’
structure assuming that the input variables have been set. The
input variables are clearly shown in the definition of
‘gal_warp_wcsalign_t’. This includes sanity checking the input
arguments, as well as allocating the output image's empty pixels
(that can be filled with ‘gal_warp_wcsalign_onpix’, possibly on
threads).
-- Function:
void
gal_warp_wcsalign_onpix (gal_warp_wcsalign_t *nl, size_t ind)
Low-level function that fills pixel ‘ind’ (counting from 0) in the
already initialized output image.
-- Function:
void *
gal_warp_wcsalign_onthread (void *inparam)
Low-level worker function that can be passed to the high-level
‘gal_threads_spin_off’ or the lower-level ‘pthread_create’ with
some modifications, see *note Multithreaded programming::.
-- Function:
void
gal_warp_wcsalign_free (gal_warp_wcsalign_t *wa)
Low-level function to free the internal variables inside ‘wa’ only.
The caller must free the input pointers themselves, this function
will not free them (they may be necessary in other parts of the
caller's higher-level architecture).
-- Function:
void
gal_warp_pixelarea (gal_warp_wcsalign_t *wa)
Calculate each input pixel's area based on its WCS and save it to a
copy of the input image with only one difference: the pixel values
now show pixel area. For examples on its usage, see *note Pixel
information images::.
12.3.30 Color functions (‘color.h’)
-----------------------------------
The available pre-defined colors in Gnuastro are shown and discussed in
*note Vector graphics colors::. This part of Gnuastro is currently in
charge of mapping the color names to the color IDs and to return the
red-green-blue fractions of each color. On a terminal that supports
24-bit (true color), you can see the full list of color names and a demo
of each color with this command:
$ astconvertt --listcolors
For each color we have a separate macro that starts with ‘GAL_COLOR_’,
and ends with the color name in all-caps.
-- Macro: GAL_COLOR_INVALID
-- Macro: GAL_COLOR_MEDIUMVIOLETRED
-- Macro: GAL_COLOR_DEEPPINK
-- Macro: GAL_COLOR_*
The integer identifiers for each of the named colors in Gnuastro.
Except for the first one (‘GAL_COLOR_INVALID’), we currently have
140 colors from the extended web colors
(https://en.wikipedia.org/wiki/Web_colors#Extended_colors). The
full list of colors and a demo can be visually inspected on the
command-line with the ‘astconvertt --listcolors’ command and is
also shown in *note Vector graphics colors::. The macros have the
same names, just in full-caps.
The functions below can be used to interact with the pre-defined colors:
-- Function:
uint8_t
gal_color_name_to_id (char *name)
Given the name of a color, return the identifier. The name
matching is not case-sensitive.
-- Function:
char *
gal_color_id_to_name (uint8_t color)
Given the ID of a color, return its name.
-- Function:
void
gal_color_in_rgb (uint8_t color, float *f)
Given the identifier of a color, write the color's red-green-blue
fractions in the space that ‘f’ points to. It is up to the caller
to have the space for three 32-bit floating point numbers to be
already allocated before calling this function.
12.3.31 Git wrappers (‘git.h’)
------------------------------
Git is one of the most common tools for version control and it can often
be useful during development, for example, see ‘COMMIT’ keyword in *note
Output FITS files::. At installation time, Gnuastro will also check for
the existence of libgit2, and store the value in the
‘GAL_CONFIG_HAVE_LIBGIT2’, see *note Configuration information:: and
*note Optional dependencies::. ‘gnuastro/git.h’ includes
‘gnuastro/config.h’ internally, so you will not have to include both for
this macro.
-- Function:
char *
gal_git_describe ( )
When libgit2 is present and the program is called within a
directory that is version controlled, this function will return a
string containing the commit description (similar to Gnuastro's
unofficial version number, see *note Version numbering::). If
there are uncommitted changes in the running directory, it will add
a '‘-dirty’' prefix to the description. When there is no tagged
point in the previous commit, this function will return a uniquely
abbreviated commit object as fallback. This function is used for
generating the value of the ‘COMMIT’ keyword in *note Output FITS
files::. The output string is similar to the output of the
following command:
$ git describe --dirty --always
Space for the output string is allocated within this function, so
after using the value you have to ‘free’ the output string. If
libgit2 is not installed or the program calling this function is
not within a version controlled directory, then the output will be
the ‘NULL’ pointer.
12.3.32 Python interface (‘python.h’)
-------------------------------------
Python (https://en.wikipedia.org/wiki/Python_(programming_language)) is
a high-level interpreted programming language that is used by some for
data analysis. Python itself is written in C, which is the same
language that Gnuastro is written in. Hence Gnuastro's library can be
directly used in Python wrappers. The functions in this section provide
some low-level features to simplify the creation of Python modules that
may want to use Gnuastro's advanced and powerful features directly. To
see why Gnuastro was written in C, please see *note Why C::.
*Python interface is not built by default:* to have the features
described in this section, Gnuastro's library needs to be built with the
‘--with-python’ configuration option. For more, on this configuration
option, see *note Gnuastro configure options::. To see if the Gnuastro
library that you are linking with has these features, you can check the
value of ‘GAL_CONFIG_HAVE_PYTHON’ macro, see *note Configuration
information::.
The Gnuastro Python Package is built using CPython. This entails
using Python wrappers around currently existing Gnuastro library
functions to build Python Extension Modules
(https://docs.python.org/3/extending/extending.html#). It also makes
use of the NumPy C-API
(https://numpy.org/doc/stable/reference/c-api/index.html) for dealing
with data arrays. Writing an interface between these and Gnuastro can
be simplified using the functions below. Since many of these functions
depend on the Gnuastro Library itself, it is more convenient to package
them with the Library to facilitate the work of Python package. These
functions will be expanding as Gnuastro's own Python module (pyGnuastro)
grows.
The Python interface of Gnuastro's library is built and installed by
default if a Python 3.0.0 or greater with NumPy is found in ‘$PATH’.
Users may disable this interface with the ‘--without-python’ option to
‘./configure’ when they installed Gnuastro, see *note Gnuastro configure
options::. If you have problems in a Python virtual env, see *note
Optional dependencies::.
Because Python is an optional dependency of Gnuastro, the following
functions may not be available on some systems. To check if the
installed Gnuastro library was compiled with the following functions,
you can use the ‘GAL_CONFIG_HAVE_PYTHON’ macro which is defined in
‘gnuastro/config.h’, see *note Configuration information::.
-- Function:
int
gal_python_type_to_numpy (uint8_t type)
Returns the NumPy datatype corresponding to a certain Gnuastro
‘type’, see *note Library data types::.
-- Function:
uint8_t
gal_python_type_from_numpy (int type)
Returns Gnuastro's numerical datatype that corresponds to the input
NumPy ‘type’. For Gnuastro's recognized data types, see *note
Library data types::.
12.3.33 Unit conversion library (‘units.h’)
-------------------------------------------
Datasets can contain values in various formats or units. The functions
in this section are defined to facilitate the easy conversion between
them and are declared in ‘units.h’. If there are certain conversions
that are useful for your work, please get in touch.
-- Function:
int
gal_units_extract_decimal (char *convert, const char
*delimiter, double *args, size_t n)
Parse the input ‘convert’ string with a certain delimiter (for
example, ‘01:23:45’, where the delimiter is ‘":"’) as multiple
numbers (for example, 1,23,45) and write them as an array in the
space that ‘args’ is pointing to. The expected number of values in
the string is specified by the ‘n’ argument (3 in the example
above).
If the function succeeds, it will return 1, otherwise it will
return 0 and the values may not be fully written into ‘args’. If
the number of values parsed in the string is different from ‘n’,
this function will fail.
-- Function:
double
gal_units_ra_to_degree (char *convert)
Convert the input Right Ascension (RA) string (in the format of
hours, minutes and seconds either as ‘_h_m_s’ or ‘_:_:_’) to
degrees (a single floating point number).
-- Function:
double
gal_units_dec_to_degree (char *convert)
Convert the input Declination (Dec) string (in the format of
degrees, arc-minutes and arc-seconds either as ‘_d_m_s’ or ‘_:_:_’)
to degrees (a single floating point number).
-- Function:
char *
gal_units_degree_to_ra (double decimal, int usecolon)
Convert the input Right Ascension (RA) degree (a single floating
point number) to old/standard notation (in the format of hours,
minutes and seconds of ‘_h_m_s’). If ‘usecolon!=0’, then the
delimiters between the components will be colons: ‘_:_:_’.
-- Function:
char *
gal_units_degree_to_dec (double decimal, int usecolon)
Convert the input Declination (Dec) degree (a single floating point
number) to old/standard notation (in the format of degrees,
arc-minutes and arc-seconds of ‘_d_m_s’). If ‘usecolon!=0’, then
the delimiters between the components will be colons: ‘_:_:_’.
-- Function:
double
gal_units_counts_to_mag (double counts, double zeropoint)
Convert counts to magnitudes through the given zero point. For
more on the equation, see *note Brightness flux magnitude::.
-- Function:
double
gal_units_mag_to_counts (double mag, double zeropoint)
Convert magnitudes to counts through the given zero point. For
more on the equation, see *note Brightness flux magnitude::.
-- Function:
double
gal_units_mag_to_sb (double mag, double area_arcsec2)
Calculate the surface brightness of a given magnitude, over a
certain area in units of arcsec$^2$. For more on the equation, see
*note Brightness flux magnitude::.
-- Function:
double
gal_units_sb_to_mag (double sb, double area_arcsec2)
Calculate the magnitude of a given surface brightness, over a
certain area in units of arcsec$^2$. For more on the equation, see
*note Brightness flux magnitude::.
-- Function:
double
gal_units_counts_to_sb (double counts, double zeropoint_ab,
double area_arcsec2)
Calculate the surface brightness of a given count level, over a
certain area in units of arcsec$^2$, assuming a certain AB zero
point. For more on the equation, see *note Brightness flux
magnitude::.
-- Function:
double
gal_units_sb_to_counts (double sb, double zeropoint_ab, double
area_arcsec2)
Calculate the counts corresponding to a given surface brightness,
over a certain area in units of arcsec$^2$. For more on the
equation, see *note Brightness flux magnitude::.
-- Function:
double
gal_units_counts_to_jy (double counts, double zeropoint_ab)
Convert counts to Janskys through an AB magnitude-based zero point.
For more on the equation, see *note Brightness flux magnitude::.
-- Function:
double
gal_units_au_to_pc (double au)
Convert the input value (assumed to be in Astronomical Units) to
Parsecs. For the conversion equation, see the description of
‘au-to-pc’ operator in *note Arithmetic operators::.
-- Function:
double
gal_units_zeropoint_change (double counts, double
zeropoint_in, double zeropoint_out)
Convert the zero point of ‘counts’ (which is ‘zeropoint_in’) to
‘zeropoint_out’.
-- Function:
double
gal_units_counts_to_nanomaggy (double counts, double
zeropoint_ab)
Convert counts to Nanomaggy (with fixed zero point of 22.5) through
an AB magnitude-based zero point. This is just a wrapper around
‘gal_units_zeropoint_change’ with ‘zeropoint_out=22.5’.
-- Function:
double
gal_units_nanomaggy_to_counts (double counts, double
zeropoint_ab)
Convert Nanomaggy (with fixed zero point of 22.5) to counts through
an AB magnitude-based zero point. This is just a wrapper around
‘gal_units_zeropoint_change’ with ‘zeropoint_in=22.5’.
-- Function:
double
gal_units_pc_to_au (double pc)
Convert the input value (assumed to be in Parsecs) to Astronomical
Units (AUs). For the conversion equation, see the description of
‘au-to-pc’ operator in *note Arithmetic operators::.
-- Function:
double
gal_units_ly_to_pc (double ly)
Convert the input value (assumed to be in Light-years) to Parsecs.
For the conversion equation, see the description of ‘ly-to-pc’
operator in *note Arithmetic operators::.
-- Function:
double
gal_units_pc_to_ly (double pc)
Convert the input value (assumed to be in Parsecs) to Light-years.
For the conversion equation, see the description of ‘ly-to-pc’
operator in *note Arithmetic operators::.
-- Function:
double
gal_units_ly_to_au (double ly)
Convert the input value (assumed to be in Light-years) to
Astronomical Units. For the conversion equation, see the
description of ‘ly-to-pc’ operator in *note Arithmetic operators::.
-- Function:
double
gal_units_au_to_ly (double au)
Convert the input value (assumed to be in Astronomical Units) to
Light-years. For the conversion equation, see the description of
‘ly-to-pc’ operator in *note Arithmetic operators::.
12.3.34 Spectral lines library (‘speclines.h’)
----------------------------------------------
Gnuastro's library has the following macros and functions for dealing
with spectral lines. All these functions are declared in
‘gnuastro/spectra.h’.
-- Macro: GAL_SPECLINES_INVALID
-- Macro: GAL_SPECLINES_Ne_VIII_770
-- Macro: GAL_SPECLINES_Ne_VIII_780
-- Macro: GAL_SPECLINES_Ly_epsilon
-- Macro: GAL_SPECLINES_Ly_delta
-- Macro: GAL_SPECLINES_Ly_gamma
-- Macro: GAL_SPECLINES_C_III_977
-- Macro: GAL_SPECLINES_N_III_990
-- Macro: GAL_SPECLINES_N_III_991_51
-- Macro: GAL_SPECLINES_N_III_991_58
-- Macro: GAL_SPECLINES_Ly_beta
-- Macro: GAL_SPECLINES_O_VI_1032
-- Macro: GAL_SPECLINES_O_VI_1038
-- Macro: GAL_SPECLINES_Ar_I_1067
-- Macro: GAL_SPECLINES_Ly_alpha
-- Macro: GAL_SPECLINES_N_V_1238
-- Macro: GAL_SPECLINES_N_V_1243
-- Macro: GAL_SPECLINES_Si_II_1260
-- Macro: GAL_SPECLINES_Si_II_1265
-- Macro: GAL_SPECLINES_O_I_1302
-- Macro: GAL_SPECLINES_C_II_1335
-- Macro: GAL_SPECLINES_C_II_1336
-- Macro: GAL_SPECLINES_Si_IV_1394
-- Macro: GAL_SPECLINES_O_IV_1397
-- Macro: GAL_SPECLINES_O_IV_1400
-- Macro: GAL_SPECLINES_Si_IV_1403
-- Macro: GAL_SPECLINES_N_IV_1486
-- Macro: GAL_SPECLINES_C_IV_1548
-- Macro: GAL_SPECLINES_C_IV_1551
-- Macro: GAL_SPECLINES_He_II_1640
-- Macro: GAL_SPECLINES_O_III_1661
-- Macro: GAL_SPECLINES_O_III_1666
-- Macro: GAL_SPECLINES_N_III_1747
-- Macro: GAL_SPECLINES_N_III_1749
-- Macro: GAL_SPECLINES_Al_III_1855
-- Macro: GAL_SPECLINES_Al_III_1863
-- Macro: GAL_SPECLINES_Si_III
-- Macro: GAL_SPECLINES_C_III_1909
-- Macro: GAL_SPECLINES_N_II_2143
-- Macro: GAL_SPECLINES_O_III_2321
-- Macro: GAL_SPECLINES_C_II_2324
-- Macro: GAL_SPECLINES_C_II_2325
-- Macro: GAL_SPECLINES_Fe_XI_2649
-- Macro: GAL_SPECLINES_He_II_2733
-- Macro: GAL_SPECLINES_Mg_V_2783
-- Macro: GAL_SPECLINES_Mg_II_2796
-- Macro: GAL_SPECLINES_Mg_II_2803
-- Macro: GAL_SPECLINES_Fe_IV_2829
-- Macro: GAL_SPECLINES_Fe_IV_2836
-- Macro: GAL_SPECLINES_Ar_IV_2854
-- Macro: GAL_SPECLINES_Ar_IV_2868
-- Macro: GAL_SPECLINES_Mg_V_2928
-- Macro: GAL_SPECLINES_He_I_2945
-- Macro: GAL_SPECLINES_O_III_3133
-- Macro: GAL_SPECLINES_He_I_3188
-- Macro: GAL_SPECLINES_He_II_3203
-- Macro: GAL_SPECLINES_O_III_3312
-- Macro: GAL_SPECLINES_Ne_V_3346
-- Macro: GAL_SPECLINES_Ne_V_3426
-- Macro: GAL_SPECLINES_O_III_3444
-- Macro: GAL_SPECLINES_N_I_3466_50
-- Macro: GAL_SPECLINES_N_I_3466_54
-- Macro: GAL_SPECLINES_He_I_3488
-- Macro: GAL_SPECLINES_Fe_VII_3586
-- Macro: GAL_SPECLINES_Fe_VI_3663
-- Macro: GAL_SPECLINES_H_19
-- Macro: GAL_SPECLINES_H_18
-- Macro: GAL_SPECLINES_H_17
-- Macro: GAL_SPECLINES_H_16
-- Macro: GAL_SPECLINES_H_15
-- Macro: GAL_SPECLINES_H_14
-- Macro: GAL_SPECLINES_O_II_3726
-- Macro: GAL_SPECLINES_O_II_3729
-- Macro: GAL_SPECLINES_H_13
-- Macro: GAL_SPECLINES_H_12
-- Macro: GAL_SPECLINES_Fe_VII_3759
-- Macro: GAL_SPECLINES_H_11
-- Macro: GAL_SPECLINES_H_10
-- Macro: GAL_SPECLINES_H_9
-- Macro: GAL_SPECLINES_Fe_V_3839
-- Macro: GAL_SPECLINES_Ne_III_3869
-- Macro: GAL_SPECLINES_He_I_3889
-- Macro: GAL_SPECLINES_H_8
-- Macro: GAL_SPECLINES_Fe_V_3891
-- Macro: GAL_SPECLINES_Fe_V_3911
-- Macro: GAL_SPECLINES_Ne_III_3967
-- Macro: GAL_SPECLINES_H_epsilon
-- Macro: GAL_SPECLINES_He_I_4026
-- Macro: GAL_SPECLINES_S_II_4069
-- Macro: GAL_SPECLINES_Fe_V_4071
-- Macro: GAL_SPECLINES_S_II_4076
-- Macro: GAL_SPECLINES_H_delta
-- Macro: GAL_SPECLINES_He_I_4144
-- Macro: GAL_SPECLINES_Fe_II_4179
-- Macro: GAL_SPECLINES_Fe_V_4181
-- Macro: GAL_SPECLINES_Fe_II_4233
-- Macro: GAL_SPECLINES_Fe_V_4227
-- Macro: GAL_SPECLINES_Fe_II_4287
-- Macro: GAL_SPECLINES_Fe_II_4304
-- Macro: GAL_SPECLINES_O_II_4317
-- Macro: GAL_SPECLINES_H_gamma
-- Macro: GAL_SPECLINES_O_III_4363
-- Macro: GAL_SPECLINES_Ar_XIV
-- Macro: GAL_SPECLINES_O_II_4415
-- Macro: GAL_SPECLINES_Fe_II_4417
-- Macro: GAL_SPECLINES_Fe_II_4452
-- Macro: GAL_SPECLINES_He_I_4471
-- Macro: GAL_SPECLINES_Fe_II_4489
-- Macro: GAL_SPECLINES_Fe_II_4491
-- Macro: GAL_SPECLINES_N_III_4510
-- Macro: GAL_SPECLINES_Fe_II_4523
-- Macro: GAL_SPECLINES_Fe_II_4556
-- Macro: GAL_SPECLINES_Fe_II_4583
-- Macro: GAL_SPECLINES_Fe_II_4584
-- Macro: GAL_SPECLINES_Fe_II_4630
-- Macro: GAL_SPECLINES_N_III_4634
-- Macro: GAL_SPECLINES_N_III_4641
-- Macro: GAL_SPECLINES_N_III_4642
-- Macro: GAL_SPECLINES_C_III_4647
-- Macro: GAL_SPECLINES_C_III_4650
-- Macro: GAL_SPECLINES_C_III_5651
-- Macro: GAL_SPECLINES_Fe_III_4658
-- Macro: GAL_SPECLINES_He_II_4686
-- Macro: GAL_SPECLINES_Ar_IV_4711
-- Macro: GAL_SPECLINES_Ar_IV_4740
-- Macro: GAL_SPECLINES_H_beta
-- Macro: GAL_SPECLINES_Fe_VII_4893
-- Macro: GAL_SPECLINES_Fe_IV_4903
-- Macro: GAL_SPECLINES_Fe_II_4924
-- Macro: GAL_SPECLINES_O_III_4959
-- Macro: GAL_SPECLINES_O_III_5007
-- Macro: GAL_SPECLINES_Fe_II_5018
-- Macro: GAL_SPECLINES_Fe_III_5085
-- Macro: GAL_SPECLINES_Fe_VI_5146
-- Macro: GAL_SPECLINES_Fe_VII_5159
-- Macro: GAL_SPECLINES_Fe_II_5169
-- Macro: GAL_SPECLINES_Fe_VI_5176
-- Macro: GAL_SPECLINES_Fe_II_5198
-- Macro: GAL_SPECLINES_N_I_5200
-- Macro: GAL_SPECLINES_Fe_II_5235
-- Macro: GAL_SPECLINES_Fe_IV_5236
-- Macro: GAL_SPECLINES_Fe_III_5270
-- Macro: GAL_SPECLINES_Fe_II_5276
-- Macro: GAL_SPECLINES_Fe_VII_5276
-- Macro: GAL_SPECLINES_Fe_XIV
-- Macro: GAL_SPECLINES_Ca_V
-- Macro: GAL_SPECLINES_Fe_II_5316_62
-- Macro: GAL_SPECLINES_Fe_II_5316_78
-- Macro: GAL_SPECLINES_Fe_VI_5335
-- Macro: GAL_SPECLINES_Fe_VI_5424
-- Macro: GAL_SPECLINES_Cl_III_5518
-- Macro: GAL_SPECLINES_Cl_III_5538
-- Macro: GAL_SPECLINES_Fe_VI_5638
-- Macro: GAL_SPECLINES_Fe_VI_5677
-- Macro: GAL_SPECLINES_C_III_5698
-- Macro: GAL_SPECLINES_Fe_VII_5721
-- Macro: GAL_SPECLINES_N_II_5755
-- Macro: GAL_SPECLINES_C_IV_5801
-- Macro: GAL_SPECLINES_C_IV_5812
-- Macro: GAL_SPECLINES_He_I_5876
-- Macro: GAL_SPECLINES_O_I_6046
-- Macro: GAL_SPECLINES_Fe_VII_6087
-- Macro: GAL_SPECLINES_O_I_6300
-- Macro: GAL_SPECLINES_S_III_6312
-- Macro: GAL_SPECLINES_Si_II_6347
-- Macro: GAL_SPECLINES_O_I_6364
-- Macro: GAL_SPECLINES_Fe_II_6369
-- Macro: GAL_SPECLINES_Fe_X
-- Macro: GAL_SPECLINES_Fe_II_6516
-- Macro: GAL_SPECLINES_N_II_6548
-- Macro: GAL_SPECLINES_H_alpha
-- Macro: GAL_SPECLINES_N_II_6583
-- Macro: GAL_SPECLINES_S_II_6716
-- Macro: GAL_SPECLINES_S_II_6731
-- Macro: GAL_SPECLINES_O_I_7002
-- Macro: GAL_SPECLINES_Ar_V
-- Macro: GAL_SPECLINES_He_I_7065
-- Macro: GAL_SPECLINES_Ar_III_7136
-- Macro: GAL_SPECLINES_Fe_II_7155
-- Macro: GAL_SPECLINES_Ar_IV_7171
-- Macro: GAL_SPECLINES_Fe_II_7172
-- Macro: GAL_SPECLINES_C_II_7236
-- Macro: GAL_SPECLINES_Ar_IV_7237
-- Macro: GAL_SPECLINES_O_I_7254
-- Macro: GAL_SPECLINES_Ar_IV_7263
-- Macro: GAL_SPECLINES_He_I_7281
-- Macro: GAL_SPECLINES_O_II_7320
-- Macro: GAL_SPECLINES_O_II_7331
-- Macro: GAL_SPECLINES_Ni_II_7378
-- Macro: GAL_SPECLINES_Ni_II_7411
-- Macro: GAL_SPECLINES_Fe_II_7453
-- Macro: GAL_SPECLINES_N_I_7468
-- Macro: GAL_SPECLINES_S_XII
-- Macro: GAL_SPECLINES_Ar_III_7751
-- Macro: GAL_SPECLINES_He_I_7816
-- Macro: GAL_SPECLINES_Ar_I_7868
-- Macro: GAL_SPECLINES_Ni_III
-- Macro: GAL_SPECLINES_Fe_XI_7892
-- Macro: GAL_SPECLINES_He_II_8237
-- Macro: GAL_SPECLINES_Pa_20
-- Macro: GAL_SPECLINES_Pa_19
-- Macro: GAL_SPECLINES_Pa_18
-- Macro: GAL_SPECLINES_O_I_8446
-- Macro: GAL_SPECLINES_Pa_17
-- Macro: GAL_SPECLINES_Ca_II_8498
-- Macro: GAL_SPECLINES_Pa_16
-- Macro: GAL_SPECLINES_Ca_II_8542
-- Macro: GAL_SPECLINES_Pa_15
-- Macro: GAL_SPECLINES_Cl_II
-- Macro: GAL_SPECLINES_Pa_14
-- Macro: GAL_SPECLINES_Fe_II_8617
-- Macro: GAL_SPECLINES_Ca_II_8662
-- Macro: GAL_SPECLINES_Pa_13
-- Macro: GAL_SPECLINES_N_I_8680
-- Macro: GAL_SPECLINES_N_I_8703
-- Macro: GAL_SPECLINES_N_I_8712
-- Macro: GAL_SPECLINES_Pa_12
-- Macro: GAL_SPECLINES_Pa_11
-- Macro: GAL_SPECLINES_Fe_II_8892
-- Macro: GAL_SPECLINES_Pa_10
-- Macro: GAL_SPECLINES_S_III_9069
-- Macro: GAL_SPECLINES_Pa_9
-- Macro: GAL_SPECLINES_S_III_9531
-- Macro: GAL_SPECLINES_Pa_epsilon
-- Macro: GAL_SPECLINES_C_I_9824
-- Macro: GAL_SPECLINES_C_I_9850
-- Macro: GAL_SPECLINES_S_VIII
-- Macro: GAL_SPECLINES_He_I_10028
-- Macro: GAL_SPECLINES_He_I_10031
-- Macro: GAL_SPECLINES_Pa_delta
-- Macro: GAL_SPECLINES_S_II_10287
-- Macro: GAL_SPECLINES_S_II_10320
-- Macro: GAL_SPECLINES_S_II_10336
-- Macro: GAL_SPECLINES_Fe_XIII
-- Macro: GAL_SPECLINES_He_I_10830
-- Macro: GAL_SPECLINES_Pa_gamma
-- Macro: GAL_SPECLINES_NUMBER
Internal values/identifiers for recognized spectral lines as is
clear from their names. They are based on the UV an optical table
of galaxy emission lines of Drew Chojnowski(1).
Note the first and last macros, they can be used when parsing the
lines automatically: both do not correspond to any line, but their
integer values correspond to the two integers just before and after
the first and last line identifier: ‘GAL_SPECLINES_INVALID’ has a
value of zero, and allows you to have a fixed integer which never
corresponds to a line. ‘GAL_SPECLINES_INVALID_MAX’ is the total
number of pre-defined lines, plus one. So you can parse all the
known lines with a ‘for’ loop like this:
for(i=1;i
12.3.35 Cosmology library (‘cosmology.h’)
-----------------------------------------
This library does the main cosmological calculations that are commonly
necessary in extra-galactic astronomical studies. The main variable in
this context is the redshift ($z$). The cosmological input parameters
in the functions below are ‘H0’, ‘o_lambda_0’, ‘o_matter_0’,
‘o_radiation_0’ which respectively represent the current (at redshift 0)
expansion rate (Hubble constant in units of km/sec/Mpc), cosmological
constant ($\Lambda$), matter and radiation densities.
All these functions are declared in ‘gnuastro/cosmology.h’. For a
more extended introduction/discussion of the cosmological parameters,
please see *note CosmicCalculator::.
-- Function:
double
gal_cosmology_age (double z, double H0, double o_lambda_0,
double o_matter_0, double o_radiation_0)
Returns the age of the universe at redshift ‘z’ in units of Giga
years.
-- Function:
double
gal_cosmology_proper_distance (double z, double H0, double
o_lambda_0, double o_matter_0, double o_radiation_0)
Returns the proper distance to an object at redshift ‘z’ in units
of Mega parsecs.
-- Function:
double
gal_cosmology_comoving_volume (double z, double H0, double
o_lambda_0, double o_matter_0, double o_radiation_0)
Returns the comoving volume over 4pi stradian to ‘z’ in units of
Mega parsecs cube.
-- Function:
double
gal_cosmology_critical_density (double z, double H0, double
o_lambda_0, double o_matter_0, double o_radiation_0)
Returns the critical density at redshift ‘z’ in units of $g/cm^3$.
-- Function:
double
gal_cosmology_angular_distance (double z, double H0, double
o_lambda_0, double o_matter_0, double o_radiation_0)
Return the angular diameter distance to an object at redshift ‘z’
in units of Mega parsecs.
-- Function:
double
gal_cosmology_luminosity_distance (double z, double H0, double
o_lambda_0, double o_matter_0, double o_radiation_0)
Return the luminosity diameter distance to an object at redshift
‘z’ in units of Mega parsecs.
-- Function:
double
gal_cosmology_distance_modulus (double z, double H0, double
o_lambda_0, double o_matter_0, double o_radiation_0)
Return the distance modulus at redshift ‘z’ (with no units).
-- Function:
double
gal_cosmology_to_absolute_mag (double z, double H0, double
o_lambda_0, double o_matter_0, double o_radiation_0)
Return the conversion from apparent to absolute magnitude for an
object at redshift ‘z’. This value has to be added to the apparent
magnitude to give the absolute magnitude of an object at redshift
‘z’.
-- Function:
double
gal_cosmology_velocity_from_z (double z)
Return the velocity (in km/s) corresponding to the given redshift
(‘z’).
-- Function:
double
gal_cosmology_z_from_velocity (double v)
Return the redshift corresponding to the given velocity (‘v’ in
km/s).
12.3.36 SAO DS9 library (‘ds9.h’)
---------------------------------
This library operates on the output files of SAO DS9(1). SAO DS9 is one
of the most commonly used FITS image and cube viewers today with an easy
to use graphic user interface (GUI), see *note SAO DS9::. But besides
merely opening FITS data, it can also produce certain kinds of files
that can be useful in common analysis. For example, on DS9's GUI, it is
very easy to define a (possibly complex) polygon as a "region". You can
then save that "region" into a file and using the functions below, feed
the polygon into Gnuastro's programs (or your custom programs).
-- Macro: GAL_DS9_COORD_MODE_IMG
-- Macro: GAL_DS9_COORD_MODE_WCS
-- Macro: GAL_DS9_COORD_MODE_INVALID
Macros to identify the coordinate mode of the DS9 file. Their
names are sufficiently descriptive. The last one (‘INVALID’) is
for sanity checks (for example, to know if the mode is already
selected).
-- Function:
gal_data_t *
gal_ds9_reg_read_polygon (char *filename)
Returns an allocated generic data container (‘gal_data_t’, with an
array of ‘GAL_TYPE_FLOAT64’) containing the vertices of a polygon
within the SAO DS9 region file given by ‘*filename’. Since SAO DS9
region files are 2 dimensional, if there are $N$ vertices in the
SAO DS9 region file, the returned dataset will have $2\times N$
elements (first two elements belonging to first vertice, etc.).
The mode to interpret the vertice coordinates is also read from the
SAO DS9 region file and written into the ‘status’ attribute of the
output ‘gal_data_t’. The coordinate mode can be one of the
‘GAL_DS9_COORD_MODE_*’ macros, mentioned above.
It is assumed that the file begins with ‘# Region file format: DS9’
and it has two more lines (at least): a line containing the mode of
the coordinates (the line should only contain either ‘fk5’ or
‘image’), a line with the polygon vertices following this format:
‘polygon(V1X,V1Y,V2X,V2Y,...)’ where ‘V1X’ and ‘V1Y’ are the
horizontal and vertical coordinates of the first vertice, and so
on.
For example, here is a minimal acceptable SAO DS9 region file:
# Region file format: DS9
fk5
polygon(53.187414,-27.779152,53.159507,-27.759633,...)
---------- Footnotes ----------
(1)
12.4 Library demo programs
==========================
In this final section of *note Library::, we give some example Gnuastro
programs to demonstrate various features in the library. All these
programs have been tested and once Gnuastro is installed you can compile
and run them with Gnuastro's *note BuildProgram:: program that will take
care of linking issues. If you do not have any FITS file to experiment
on, you can use those that are generated by Gnuastro after ‘make check’
in the ‘tests/’ directory, see *note Quick start::.
12.4.1 Library demo - reading a FITS image
------------------------------------------
The following simple program demonstrates how to read a FITS image into
memory and use the ‘void *array’ pointer in of *note Generic data
container::. For easy linking/compilation of this program along with a
first run see *note BuildProgram:: (in short: Compile, link and run
‘myprogram.c' with this command: '‘astbuildprog myprogram.c’). Before
running, also change the ‘filename’ and ‘hdu’ variable values to specify
an existing FITS file and/or extension/HDU.
This is just intended to demonstrate how to use the ‘array’ pointer
of ‘gal_data_t’. Hence it does not do important sanity checks, for
example in real datasets you may also have blank pixels. In such cases,
this program will return a NaN value (see *note Blank pixels::). So for
general statistical information of a dataset, it is much better to use
Gnuastro's *note Statistics:: program which can deal with blank pixels
and many other issues in a generic dataset.
To encourage good coding practices, this script contains a copyright
notice with a place holder for your name and your email (as you
customize it for your own purpose). Always keep a one-line description
and copyright notice like this in all your scripts, such "metadata" is
very important to accompany every source file you write. Of course,
when you write the source file from scratch and just learn how to use a
single function from this manual, only your name/year should appear.
The existing name of the original author of this example program is only
for cases where you copy-paste this whole file.
/* Reading a FITS image into memory.
*
* The following simple program demonstrates how to read a FITS image
* into memory and use the 'void *array' pointer. This is just intended
* to demonstrate how to use the array pointer of 'gal_data_t'.
*
* Copyright (C) 2024 Your Name
* Copyright (C) 2020-2024 Mohammad Akhlaghi
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see .
*/
#include
#include
#include /* includes gnuastro's data.h and type.h */
#include
int
main(void)
{
size_t i;
float *farray;
double sum=0.0f;
gal_data_t *image;
char *filename="img.fits", *hdu="1";
/* Read `img.fits' (HDU: 1) as a float32 array. */
image=gal_fits_img_read_to_type(filename, hdu, GAL_TYPE_FLOAT32,
-1, 1, NULL);
/* Use the allocated space as a single precision floating
* point array (recall that `image->array' has `void *'
* type, so it is not directly usable). */
farray=image->array;
/* Calculate the sum of all the values. */
for(i=0; isize; ++i)
sum += farray[i];
/* Report the sum. */
printf("Sum of values in %s (hdu %s) is: %f\n",
filename, hdu, sum);
/* Clean up and return. */
gal_data_free(image);
return EXIT_SUCCESS;
}
12.4.2 Library demo - inspecting neighbors
------------------------------------------
The following simple program shows how you can inspect the neighbors of
a pixel using the ‘GAL_DIMENSION_NEIGHBOR_OP’ function-like macro that
was introduced in *note Dimensions::. For easy linking/compilation of
this program along with a first run see *note BuildProgram::. Before
running, also change the file name and HDU (first and second arguments
to ‘gal_fits_img_read_to_type’) to specify an existing FITS file and/or
extension/HDU.
To encourage good coding practices, this script contains a copyright
notice with a place holder for your name and your email (as you
customize it for your own purpose). Always keep a one-line description
and copyright notice like this in all your scripts, such "metadata" is
very important to accompany every source file you write. Of course,
when you write the source file from scratch and just learn how to use a
single function from this manual, only your name/year should appear.
The existing name of the original author of this example program is only
for cases where you copy-paste this whole file.
/* Reading a FITS image into memory.
*
* The following simple program shows how you can inspect the neighbors
* of a pixel using the GAL_DIMENSION_NEIGHBOR_OP function-like macro.
*
* Copyright (C) 2024 Your Name
* Copyright (C) 2020-2024 Mohammad Akhlaghi
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see .
*/
#include
#include
#include
#include
int
main(void)
{
double sum;
float *array;
size_t i, num, *dinc;
gal_data_t *input=gal_fits_img_read_to_type("input.fits", "1",
GAL_TYPE_FLOAT32, -1, 1,
NULL);
/* To avoid the `void *' pointer and have `dinc'. */
array=input->array;
dinc=gal_dimension_increment(input->ndim, input->dsize);
/* Go over all the pixels. */
for(i=0;isize;++i)
{
num=0;
sum=0.0f;
GAL_DIMENSION_NEIGHBOR_OP( i, input->ndim, input->dsize,
input->ndim, dinc,
{++num; sum+=array[nind];} );
printf("%zu: num: %zu, sum: %f\n", i, num, sum);
}
/* Clean up and return. */
gal_data_free(input);
free(dinc);
return EXIT_SUCCESS;
}
12.4.3 Library demo - multi-threaded operation
----------------------------------------------
The following simple program shows how to use Gnuastro to simplify
spinning off threads and distributing different jobs between the
threads. The relevant thread-related functions are defined in *note
Gnuastro's thread related functions::. For easy linking/compilation of
this program, along with a first run, see Gnuastro's *note
BuildProgram::. Before running, also change the ‘filename’ and ‘hdu’
variable values to specify an existing FITS file and/or extension/HDU.
This is a very simple program to open a FITS image, distribute its
pixels between different threads and print the value of each pixel and
the thread it was assigned to. The actual operation is very simple (and
would not usually be done with threads in a real-life program). It is
intentionally chosen to put more focus on the important steps in
spinning off threads and how the worker function (which is called by
each thread) can identify the job-IDs it should work on.
For example, instead of an array of pixels, you can define an array
of tiles or any other context-specific structures as separate targets.
The important thing is that each action should have its own unique ID
(counting from zero, as is done in an array in C). You can then follow
the process below and use each thread to work on all the targets that
are assigned to it. Recall that spinning off threads is itself an
expensive process and we do not want to spin-off one thread for each
target (see the description of ‘gal_threads_dist_in_threads’ in *note
Gnuastro's thread related functions::.
There are many (more complicated, real-world) examples of using
‘gal_threads_spin_off’ in Gnuastro's actual source code, you can see
them by searching for the ‘gal_threads_spin_off’ function from the top
source (after unpacking the tarball) directory (for example, with this
command):
$ grep -r gal_threads_spin_off ./
To encourage good coding practices, this script contains a copyright
notice with a place holder for your name and your email (as you
customize it for your own purpose). Always keep a one-line description
and copyright notice like this in all your scripts, such "metadata" is
very important to accompany every source file you write. Of course,
when you write the source file from scratch and just learn how to use a
single function from this manual, only your name/year should appear.
The existing name of the original author of this example program is only
for cases where you copy-paste this whole file.
The code of this demonstration program is shown below. This program
was also built and run when you ran ‘make check’ during the building of
Gnuastro (‘tests/lib/multithread.c’), so it is already tested for your
system and you can safely use it as a guide.
/* Demo of Gnuastro's high-level multi-threaded interface.
*
* This is a very simple program to open a FITS image, distribute its
* pixels between different threads and print the value of each pixel
* and the thread it was assigned to.
*
* Copyright (C) 2024 Your Name
* Copyright (C) 2020-2024 Mohammad Akhlaghi
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see .
*/
#include
#include
#include
#include
/* This structure can keep all information you want to pass onto the
* worker function on each thread. */
struct params
{
gal_data_t *image; /* Dataset to print values of. */
};
/* This is the main worker function which will be called by the
* different threads. `gal_threads_params' is defined in
* `gnuastro/threads.h' and contains the pointer to the parameter we
* want. Note that the input argument and returned value of this
* function always must have `void *' type. */
void *
worker_on_thread(void *in_prm)
{
/* Low-level definitions to be done first. */
struct gal_threads_params *tprm=(struct gal_threads_params *)in_prm;
struct params *p=(struct params *)tprm->params;
/* Subsequent definitions. */
float *array=p->image->array;
size_t i, index, *dsize=p->image->dsize;
/* Go over all the actions (pixels in this case) that were assigned
* to this thread. */
for(i=0; tprm->indexs[i] != GAL_BLANK_SIZE_T; ++i)
{
/* For easy reading. */
index = tprm->indexs[i];
/* Print the information. */
printf("(%zu, %zu) on thread %zu: %g\n", index%dsize[1]+1,
index/dsize[1]+1, tprm->id, array[index]);
}
/* Wait for all the other threads to finish, then return. */
if(tprm->b) pthread_barrier_wait(tprm->b);
return NULL;
}
/* High-level function (called by the operating system). */
int
main(void)
{
struct params p;
char *filename="input.fits", *hdu="1";
size_t numthreads=gal_threads_number();
/* We are using * `-1' for `minmapsize' to ensure that the image is
* read into * memory and `1' for `quietmmap' (which can also be
* zero), see the "Memory management" section in the book. */
int quietmmap=1;
size_t minmapsize=-1;
/* Read the image into memory as a float32 data type. */
p.image=gal_fits_img_read_to_type(filename, hdu, GAL_TYPE_FLOAT32,
minmapsize, quietmmap, NULL);
/* Print some basic information before the actual contents: */
printf("Pixel values of %s (HDU: %s) on %zu threads.\n", filename,
hdu, numthreads);
printf("Used to check the compiled library's capability in opening "
"a FITS file, and also spinning off threads.\n");
/* A small sanity check: this is only intended for 2D arrays (to
* print the coordinates of each pixel). */
if(p.image->ndim!=2)
{
fprintf(stderr, "only 2D images are supported.");
exit(EXIT_FAILURE);
}
/* Spin-off the threads and do the processing on each thread. */
gal_threads_spin_off(worker_on_thread, &p, p.image->size, numthreads,
minmapsize, quietmmap);
/* Clean up and return. */
gal_data_free(p.image);
return EXIT_SUCCESS;
}
12.4.4 Library demo - reading and writing table columns
-------------------------------------------------------
Tables are some of the most common inputs to, and outputs of programs.
This section contains a small program for reading and writing tables
using the constructs described in *note Table input output::. For easy
linking/compilation of this program, along with a first run, see
Gnuastro's *note BuildProgram::. Before running, also set the following
file and column names in the first two lines of ‘main’. The input and
output names may be ‘.txt’ and ‘.fits’ tables, ‘gal_table_read’ and
‘gal_table_write’ will be able to write to both formats. For plain text
tables see *note Gnuastro text table format::. If you do not have any
table in text file format to use as your input, you can use the table
that is generated in *note Sufi simulates a detection:: section.
This example program reads three columns from a table. The first two
columns are selected by their name (‘NAME1’ and ‘NAME2’) and the third
is selected by its number: column 10 (counting from 1). Gnuastro's
column selection is discussed in *note Selecting table columns::. The
first and second columns can be any type, but this program will convert
them to ‘int32_t’ and ‘float’ for its internal usage respectively.
However, the third column must be double for this program. So if it is
not, the program will abort with an error. Having the columns in
memory, it will print them out along with their sum (just a simple
application, you can do what ever you want at this stage). Reading the
table finishes here.
The rest of the program is a demonstration of writing a table. While
parsing the rows, this program will change the first column (to be
counters) and multiply the second by 10 (so the output will be
different). Then it will define the order of the output columns by
setting the ‘next’ element (to create a *note List of gal_data_t::).
Before writing, this function will also set names for the columns (units
and comments can be defined in a similar manner). Writing the columns
to a file is then done through a simple call to ‘gal_table_write’.
The operations that are shown in this example program are not
necessary all the time. For example, in many cases, you know the
numerical data type of the column before writing your program (see *note
Numeric data types::), so type checking and copying to a specific type
will not be necessary.
To encourage good coding practices, this script contains a copyright
notice with a place holder for your name and your email (as you
customize it for your own purpose). Always keep a one-line description
and copyright notice like this in all your scripts, such "metadata" is
very important to accompany every source file you write. Of course,
when you write the source file from scratch and just learn how to use a
single function from this manual, only your name/year should appear.
The existing name of the original author of this example program is only
for cases where you copy-paste this whole file.
/* Reading and writing table columns.
*
* This example program reads three columns from a table. Having the
* columns in memory, it will print them out along with their sum. The
* rest of the program is a demonstration of writing a table.
*
* Copyright (C) 2024 Your Name
* Copyright (C) 2020-2024 Mohammad Akhlaghi
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see .
*/
#include
#include
#include
int
main(void)
{
/* File names and column names (which may also be numbers). */
char *c1_name="NAME1", *c2_name="NAME2", *c3_name="10";
char *inname="input.fits", *hdu="1", *outname="out.fits";
/* Internal parameters. */
float *array2=NULL;
double *array3=NULL;
int32_t *array1=NULL;
size_t i, counter=0;
gal_data_t *c1=NULL;
gal_data_t *c2=NULL;
gal_data_t tmp, *col, *columns;
gal_list_str_t *column_ids=NULL;
/* Define the columns to read. */
gal_list_str_add(&column_ids, c1_name, 0);
gal_list_str_add(&column_ids, c2_name, 0);
gal_list_str_add(&column_ids, c3_name, 0);
/* The columns were added in reverse, so correct it. */
gal_list_str_reverse(&column_ids);
/* Read the desired columns. */
columns = gal_table_read(inname, hdu, NULL, column_ids,
GAL_TABLE_SEARCH_NAME, 0, 1, -1, 1, NULL);
/* Go over the columns, we will assume that you do not know their type
* a-priori, so we will check */
counter=1;
for(col=columns; col!=NULL; col=col->next)
switch(counter++)
{
case 1: /* First column: we want it as int32_t. */
c1=gal_data_copy_to_new_type(col, GAL_TYPE_INT32);
array1 = c1->array;
break;
case 2: /* Second column: we want it as float. */
c2=gal_data_copy_to_new_type(col, GAL_TYPE_FLOAT32);
array2 = c2->array;
break;
case 3: /* Third column: it MUST be double. */
if(col->type!=GAL_TYPE_FLOAT64)
{
fprintf(stderr, "Column %s must be float64 type, it is "
"%s", c3_name, gal_type_name(col->type, 1));
exit(EXIT_FAILURE);
}
array3 = col->array;
break;
default:
exit(EXIT_FAILURE);
}
/* As an example application we will just print them out. In the
* meantime (just for a simple demonstration), change the first
* array value to the counter and multiply the second by 10. */
for(i=0;isize;++i)
{
printf("%zu: %d + %f + %f = %f\n", i+1, array1[i], array2[i],
array3[i], array1[i]+array2[i]+array3[i]);
array1[i] = i+1;
array2[i] *= 10;
}
/* Link the first two columns as a list. */
c1->next = c2;
c2->next = NULL;
/* Set names for the columns and write them out. */
c1->name = "COUNTER";
c2->name = "VALUE";
gal_table_write(c1, NULL, NULL, GAL_TABLE_FORMAT_BFITS, outname,
"MY-COLUMNS", 0, 0);
/* The names were not allocated, so to avoid cleaning-up problems,
* we will set them to NULL. */
c1->name = c2->name = NULL;
/* Clean up and return. */
gal_data_free(c1);
gal_data_free(c2);
gal_list_data_free(columns);
gal_list_str_free(column_ids, 0); /* strings were not allocated. */
return EXIT_SUCCESS;
}
12.4.5 Library demo - Warp to another image
-------------------------------------------
Gnuastro's warp library (that you can access by including
‘gnuastro/warp.h’) allows you to resample an image from a grid to
another entirely using the WCSLIB (while accounting for distortions if
necessary; see *note Warp library::). The Warp library uses a
pixel-mixing or area-based resampling approach which is fully described
in *note Resampling::. The most generic uses cases for this library are
already available in the *note Invoking astwarp:: program. For a
related demo (where the output grid and WCS are constructed from
scratch), see *note Library demo - Warp to new grid::.
In the example below, we are warping the ‘input.fits’ file to the
same pixel grid and WCS as ‘reference.fits’ image (assuming it is in hdu
‘0’). You can download the FITS files in the *note Color channels in
same pixel grid:: section and use them as ‘input.fits’ and
‘reference.fits’ files. Feel free to change these names to your own
test file names. This can be useful when you have a complex grid and
WCS containing various keywords such as non-linear distortion
coefficients, etc. For example datasets, see the description of the
‘--gridfile’ option in *note Align pixels with WCS considering
distortions::.
To compile the demonstration program below, copy and paste the
contents in a plain-text file (let's assume you named it
‘align-to-img.c’) and use *note BuildProgram:: with this command:
'‘astbuildprog align-to-img.c’'. Please note that the demo program does
not perform many sanity checks to avoid making it too complex and to
highlight this particular feature in the library. For a robust method
write programs with all the necessary sanity checks, see Gnuastro's Warp
source code, see *note Program source::.
To encourage good coding practices, this script contains a copyright
notice with a place holder for your name and your email (as you
customize it for your own purpose). Always keep a one-line description
and copyright notice like this in all your scripts, such "metadata" is
very important to accompany every source file you write. Of course,
when you write the source file from scratch and just learn how to use a
single function from this manual, only your name/year should appear.
The existing name of the original author of this example program is only
for cases where you copy-paste this whole file.
/* Warp to another image.
*
* In the example below, we are warping the input.fits file to the same
* pixel grid and WCS as reference.fits image.
*
* Copyright (C) 2024 Your Name
* Copyright (C) 2022-2024 Pedram Ashofteh-Ardakani
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see .
*/
#include
#include
#include /* contains gnuastro's fits.h */
#include /* contains gnuastro's data.h */
#include /* contains gnuastro's type.h */
int
main(void)
{
/* Input file's name and HDU. */
char *filename="input.fits", *hdu="1";
/* Reference file's name and HDU. */
char *gridfile="reference.fits", *gridhdu="0";
/* Output file name. */
char *outname="align-to-img.fits";
/* Low-level variables needed to read the reference file's size. */
int nwcs;
size_t ndim, *dsize;
/* Initialize the 'wa' struct with empty values and NULL pointers. */
gal_warp_wcsalign_t wa=gal_warp_wcsalign_template();
/* Read the input image and its WCS. */
wa.input=gal_array_read_one_ch_to_type(filename, hdu, NULL,
GAL_TYPE_FLOAT64, -1, 0, NULL);
wa.input->wcs=gal_wcs_read(filename, hdu, 0, 0, 0, &wa.input->nwcs,
NULL);
/* Prepare the warp input structure, use all threads available. */
wa.coveredfrac=1; wa.edgesampling=0; wa.numthreads=0;
/* Set the target grid to be the same as wcsref.fits file on hdu 0. */
wa.twcs=gal_wcs_read(gridfile, gridhdu, 0, 0, 0, &nwcs, NULL);
if(wa.twcs==NULL)
{
fprintf(stderr, "%s (hdu %s): no WCS! Can't continue\n",
gridfile, gridhdu);
exit(EXIT_FAILURE);
}
/* Read the output image size (from the reference image). Note that
* 'dsize' will be freed while freeing 'widthinpix'). */
dsize=gal_fits_img_info_dim(gridfile, gridhdu, &ndim, NULL);
/* Convert the 'dsize' to a 'gal_data_t' so the library can use it. */
wa.widthinpix=gal_data_alloc(dsize, GAL_TYPE_SIZE_T, 1, &ndim,
NULL, 1, -1, 0, NULL, NULL, NULL);
/* Do the warp, then convert the output to a 32-bit float (the default
* float64 is too much for observational data and just wastes
* storage!). But if you are warping mock data before adding noise
* (where you do have float64 level precision), remove the type
* conversion line. */
gal_warp_wcsalign(&wa);
wa.output=gal_data_copy_to_new_type_free(wa.output, GAL_TYPE_FLOAT32);
/* WARNING: make sure there is no file with same name as 'out.fits'
* or the result will be appended to its final HDU. */
gal_fits_img_write(wa.output, outname, NULL, 0);
/* Clean up. */
gal_data_free(wa.input);
gal_data_free(wa.output);
gal_data_free(wa.widthinpix);
/* Give control back to the operating system. */
return EXIT_SUCCESS;
}
12.4.6 Library demo - Warp to new grid
--------------------------------------
Gnuastro's warp library (that you can access by including
‘gnuastro/warp.h’) allows you to resample an image from a grid to
another entirely using the WCSLIB (while accounting for distortions if
necessary; see *note Warp library::). The Warp library uses a
pixel-mixing or area-based resampling approach which is fully described
in *note Resampling::. The most generic uses cases for this library are
already available in the *note Invoking astwarp:: program. For a
related demo (where the output grid and WCS are imported from another
file), see *note Library demo - Warp to another image::.
In the example below, we'll assume you have the SDSS image downloaded
in *note Downloading and validating input data::. After downloading the
image as described there, you will have ‘r.fits’ in your current
directory. We will therefore use ‘r.fits’ as the input to the rest
program here. The image is not aligned to the celestial coordinates, so
we will align the pixel and WCS coordinates, but set the center of the
pixel grid to be at (RA,Dec) of (202.4173735,47.3374525). We also give
it a ‘TAN’ projection with a pixel scale of 0.27 arcsecs, a defined
center pixel. However, we'll let the Warp library measure the proper
output image size that will contain the aligned image.
To compile the demonstration program below, copy and paste the
contents in a plain-text file (let's assume you named it
‘align-to-new.c’) and use *note BuildProgram:: with this command:
'‘astbuildprog align-to-new.c’'. Please note that the demo program does
not perform many sanity checks to avoid making it too complex and to
highlight this particular feature in the library. For a robust method
write programs with all the necessary sanity checks, see Gnuastro's Warp
source code, see *note Program source::.
To encourage good coding practices, this script contains a copyright
notice with a place holder for your name and your email (as you
customize it for your own purpose). Always keep a one-line description
and copyright notice like this in all your scripts, such "metadata" is
very important to accompany every source file you write. Of course,
when you write the source file from scratch and just learn how to use a
single function from this manual, only your name/year should appear.
The existing name of the original author of this example program is only
for cases where you copy-paste this whole file.
/* Warp an image to a new grid.
*
* In the example below, We will use 'r.fits' as the input. The image is
* not aligned to the celestial coordinates, so we will align the pixel
* and WCS coordinates. We also give it a TAN projection. However, we’ll
* let the Warp library measure the proper output image size that will
* contain the aligned image.
*
* Copyright (C) 2024 Your Name
* Copyright (C) 2022-2024 Pedram Ashofteh-Ardakani
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see .
*/
#include
#include
#include /* Contains gnuastro's fits.h */
#include /* Contains gnuastro's data.h */
#include /* Contains gnuastro's type.h */
int
main(void)
{
/* Input file's name and HDU. */
char *filename="r.fits", *hdu="0";
/* Output file name. */
char *outname="align-to-new.fits";
/* RA/Dec of the center of the central pixel of output. Please
* change the center based on your input. */
double center[]={202.4173735, 47.3374525};
/* Coordinate and Projection algorithms of output. */
char *ctype[2]={"RA---TAN", "DEC--TAN"};
/* Output pixel scale (in units of degrees/pixel). */
double cdelt[]={0.27/3600, 0.27/3600};
/* For intermediate steps. */
size_t two=2;
/* Initialize the 'wa' struct with empty values and NULL pointers. */
gal_warp_wcsalign_t wa=gal_warp_wcsalign_template();
/* Set the width (and height!) of the output in pixels (as a 1D and
* 2 element 'gal_data_t'). When it is NULL, the library will
* calculate the appropriate width to fully fit the input image
* after alignment. */
wa.widthinpix=NULL;
/* Set the number of threads to use. If the value is '0', the
* library will estimate the maximum available threads at
* run-time on the host operating system. */
wa.numthreads=0;
/* Read the input image and its WCS. */
wa.input=gal_array_read_one_ch_to_type(filename, hdu, NULL,
GAL_TYPE_FLOAT64, -1, 0, NULL);
wa.input->wcs=gal_wcs_read(filename, hdu, 0, 0, 0, &wa.input->nwcs,
NULL);
/* Prepare the warp input structure. */
wa.coveredfrac=1; wa.edgesampling=0;
wa.ctype=gal_data_alloc(ctype, GAL_TYPE_STRING, 1, &two, NULL, 1,
-1, 0, NULL, NULL, NULL);
wa.cdelt=gal_data_alloc(cdelt, GAL_TYPE_FLOAT64, 1, &two, NULL, 1,
-1, 0, NULL, NULL, NULL);
wa.center=gal_data_alloc(center, GAL_TYPE_FLOAT64, 1, &two, NULL, 1,
-1, 0, NULL, NULL, NULL);
/* Do the warp, then convert it to a 32-bit float. */
gal_warp_wcsalign(&wa);
wa.output=gal_data_copy_to_new_type_free(wa.output, GAL_TYPE_FLOAT32);
/* WARNING: make sure there is no file with same name as 'out.fits'
* or the result will be appended to its final HDU. */
gal_fits_img_write(wa.output, outname, NULL, 0);
/* Remove the pointers to arrays that we didn't allocate (and thus,
* should not be freed by 'gal_data_free' below). */
wa.cdelt->array=wa.center->array=wa.ctype->array=NULL;
/* Clean up. */
gal_data_free(wa.cdelt); gal_data_free(wa.ctype);
gal_data_free(wa.input); gal_data_free(wa.output);
gal_data_free(wa.center); gal_data_free(wa.widthinpix);
/* Give control back to the operating system. */
return EXIT_SUCCESS;
}
13 Developing
*************
The basic idea of GNU Astronomy Utilities is for an interested
astronomer to be able to easily understand the code of any of the
programs or libraries, be able to modify the code if s/he feels there is
an improvement and finally, to be able to add new programs or libraries
for their own benefit, and the larger community if they are willing to
share it. In short, we hope that at least from the software point of
view, the "obscurantist faith in the expert's special skill and in his
personal knowledge and authority" can be broken, see *note Science and
its tools::. With this aim in mind, Gnuastro was designed to have a
very basic, simple, and easy to understand architecture for any
interested inquirer.
This chapter starts with very general design choices, in particular
*note Why C:: and *note Program design philosophy::. It will then get a
little more technical about the Gnuastro code and file/directory
structure in *note Coding conventions:: and *note Program source::.
*note The TEMPLATE program:: discusses a minimal (and working) template
to help in creating new programs or easier learning of a program's
internal structure. Some other general issues about documentation,
building and debugging are then discussed. This chapter concludes with
how you can learn about the development and get involved in *note
Gnuastro project webpage::, *note Developing mailing lists:: and *note
Contributing to Gnuastro::.
13.1 Why C programming language?
================================
Currently the programming languages that are commonly used in scientific
applications are C++(1), Java(2); Python(3), and Julia(4) (which is a
newcomer but swiftly gaining ground). One of the main reasons behind
choosing these is their high-level abstractions. However, GNU Astronomy
Utilities is fully written in the C programming language(5). The
reasons can be summarized with simplicity, portability and
efficiency/speed. All four are very important in a scientific software
and we will discuss them below.
Simplicity can best be demonstrated in a comparison of the main books
of C++ and C. The "C programming language"(6) book, written by the
authors of C, is only 286 pages and covers a very good fraction of the
language, it has also remained unchanged from 1988. C is the main
programming language of nearly all operating systems and there is no
plan of any significant update. On the other hand, the most recent "C++
programming language"(7) book, also written by its author, has 1366
pages and its fourth edition came out in 2013! As discussed in *note
Science and its tools::, it is very important for other scientists to be
able to readily read the code of a program at their will with minimum
requirements.
In C++ or Java, inheritance in the object oriented programming
paradigm and their internal functions make the code very easy to write
for a programmer who is deeply invested in those objects and understands
all their relations well. But it simultaneously makes reading the
program for a first time reader (a curious scientist who wants to know
only how a small step was done) extremely hard. Before understanding
the methods, the scientist has to invest a lot of time and energy in
understanding those objects and their relations. But in C, everything
is done with basic language types for example ‘int’s or ‘float’s and
their pointers to define arrays. So when an outside reader is only
interested in one part of the program, that part is all they have to
understand.
Recently it is also becoming common to write scientific software in
Python, or a combination of it with C or C++. Python is a high level
scripting language which does not need compilation. It is very useful
when you want to do something on the go and do not want to be halted by
the troubles of compiling, linking, memory checking, etc. When the
datasets are small and the job is temporary, this ability of Python is
great and is highly encouraged. A very good example might be plotting,
in which Python is undoubtedly one of the best.
But as the data sets increase in size and the processing becomes more
complicated, the speed of Python scripts significantly decrease. So
when the program does not change too often and is widely used in a large
community, mostly on large data sets (like astronomical images), using
Python will waste a lot of valuable research-hours. It is possible to
wrap C or C++ functions with Python to fix the speed issue. But this
creates further complexity, because the interested scientist has to
master two programming languages and their connection (which is not
trivial).
Like C++, Python is object oriented, so as explained above, it needs
a high level of experience with that particular program to reasonably
understand its inner workings. To make things worse, since it is mainly
for on-the-go programming(8), it can undergo significant changes. One
recent example is how Python 2.x and Python 3.x are not compatible.
Lots of research teams that invested heavily in Python 2.x cannot
benefit from Python 3.x or future versions any more. Some converters
are available, but since they are automatic, lots of complications might
arise in the conversion(9). If a research project begins using Python
3.x today, there is no telling how compatible their investments will be
when Python 4.x or 5.x will come out.
Java is also fully object-oriented, but uses a different paradigm:
its compilation generates a hardware-independent _bytecode_, and a _Java
Virtual Machine_ (JVM) is required for the actual execution of this
bytecode on a computer. Java also evolved with time, and tried to
remain backward compatible, but inevitably this evolution required
discontinuities and replacements of a few Java components which were
first declared as becoming _deprecated_, and removed from later
versions.
This stems from the core principles of high-level languages like
Python or Java: that they evolve significantly on the scale of roughly 5
to 10 years. They are therefore useful when you want to solve a
short-term problem and you are ready to pay the high cost of keeping
your software up to date with all the changes in the language. This is
fine for private companies, but usually too expensive for scientific
projects that have limited funding for a fixed period. As a result, the
reproducibility of the result (ability to regenerate the result in the
future, which is a core principal of any scientific result) and
reusability of all the investments that went into the science software
will be lost to future generations! Rebuilding all the dependencies of
a software in an obsolete language is not easy, or even not possible.
Future-proof code (as long as current operating systems will be used) is
therefore written in C.
The portability of C is best demonstrated by the fact that C++, Java
and Python are part of the C-family of programming languages which also
include Julia, Perl, and many other languages. C libraries can be
immediately included in C++, and it is easy to write wrappers for them
in all C-family programming languages. This will allow other scientists
to benefit from C libraries using any C-family language that they
prefer. As a result, Gnuastro's library is already usable in C and C++,
and wrappers will be(10) added for higher-level languages like Python,
Julia and Java.
The final reason was speed. This is another very important aspect of
C which is not independent of simplicity (first reason discussed above).
The abstractions provided by the higher-level languages (which also
makes learning them harder for a newcomer) come at the cost of speed.
Since C is a low-level language(11) (closer to the hardware), it has a
direct access to the CPU(12), is generally considered as being faster in
its execution, and is much less complex for both the human reader _and_
the computer. The benefits of simplicity for a human were discussed
above. Simplicity for the computer translates into more efficient
(faster) programs. This creates a much closer relation between the
scientist/programmer (or their program) and the actual data and
processing. The GNU coding standards(13) also encourage the use of C
over all other languages when generality of usage and "high speed" is
desired.
---------- Footnotes ----------
(1)
(2)
(3)
(4)
(5)
(6) Brian Kernighan, Dennis Ritchie. _The C programming language_.
Prentice Hall, Inc., Second edition, 1988. It is also commonly known as
K&R and is based on the ANSI C and ISO C90 standards.
(7) Bjarne Stroustrup. _The C++ programming language_.
Addison-Wesley Professional; 4 edition, 2013.
(8) Note that Python is good for fast programming, not fast programs.
(9) For example see Jenness 2017 (https://arxiv.org/abs/1712.00461),
which describes how LSST is managing the transition.
(10)
(11) Low-level languages are those that directly operate the hardware
like assembly languages. So C is actually a high-level language, but it
can be considered one of the lowest-level languages among all high-level
languages.
(12) for instance the _long double_ numbers with at least 64-bit
mantissa are not accessible in Python or Java.
(13)
13.2 Program design philosophy
==============================
The core processing functions of each program (and all libraries) are
written mostly with the basic ISO C90 standard. We do make lots of use
of the GNU additions to the C language in the GNU C library(1), but
these functions are mainly used in the user interface functions (reading
your inputs and preparing them prior to or after the analysis). The
actual algorithms, which most scientists would be more interested in,
are much more closer to ISO C90. For this reason, program source files
that deal with user interface issues and those doing the actual
processing are clearly separated, see *note Program source::. If
anything particular to the GNU C library is used in the processing
functions, it is explained in the comments in between the code.
All the Gnuastro programs provide very low level and modular
operations (modeled on GNU Coreutils). Almost all the basic
command-line programs like ‘ls’, ‘cp’ or ‘rm’ on GNU/Linux operating
systems are part of GNU Coreutils. This enables you to use shell
scripting languages (for example, GNU Bash) to operate on a large number
of files or do very complex things through the creative combinations of
these tools that the authors had never dreamed of. We have put a few
simple examples in *note Tutorials::.
For example, all the analysis output can be saved as ASCII tables
which can be fed into your favorite plotting program to inspect
visually. Python's Matplotlib is very useful for fast plotting of the
tables to immediately check your results. If you want to include the
plots in a document, you can use the PGFplots package within LaTeX, no
attempt is made to include such operations in Gnuastro. In short, Bash
can act as a glue to connect the inputs and outputs of all these various
Gnuastro programs (and other programs) in any fashion. Of course,
Gnuastro's programs are just front-ends to the main workhorse (*note
Gnuastro library::), allowing a user to create their own programs (for
example, with *note BuildProgram::). So once the functions within
programs become mature enough, they will be moved within the libraries
for even more general applications.
The advantage of this architecture is that the programs become small
and transparent: the starting and finishing point of every program is
clearly demarcated. For nearly all operations on a modern computer
(fast file input-output) with a modest level of complexity, the
read/write speed is insignificant compared to the actual processing a
program does. Therefore the complexity which arises from sharing memory
in a large application is simply not worth the speed gain. Gnuastro's
design is heavily influenced from Eric Raymond's "The Art of Unix
Programming"(2) which beautifully describes the design philosophy and
practice which lead to the success of Unix-based operating systems(3).
---------- Footnotes ----------
(1) Gnuastro uses many GNU additions to the C library. However,
thanks to the GNU Portability library (Gnulib) which is included in the
Gnuastro tarball, users of non-GNU/Linux operating systems can also
benefit from all these features when using Gnuastro.
(2) Eric S. Raymond, 2004, _The Art of Unix Programming_,
Addison-Wesley Professional Computing Series.
(3) KISS principle: Keep It Simple, Stupid!
13.3 Coding conventions
=======================
In Gnuastro, we try our best to follow the GNU coding standards. Added
to those, Gnuastro defines the following conventions. It is very
important for readability that the whole package follows the same
convention.
• The code must be easy to read by eye. So when the order of several
lines within a function does not matter (for example, when defining
variables at the start of a function). You should put the lines in
the order of increasing length and group the variables with similar
types such that this half-pyramid of declarations becomes most
visible. If the reader is interested, a simple search will show
them the variable they are interested in. However, this visual aid
greatly helps in general inspections of the code and help the
reader get a grip of the function's processing.
• A function that cannot be fully displayed (vertically) in your
monitor is probably too long and may be more useful if it is broken
up into multiple functions. 40 lines is usually a good reference.
When the start and end of a function are clearly visible in one
glance, the function is much more easier to understand. This is
most important for low-level functions (which usually define a lot
of variables). Low-level functions do most of the processing, they
will also be the most interesting part of a program for an
inquiring astronomer. This convention is less important for higher
level functions that do not define too many variables and whose
only purpose is to run the lower-level functions in a specific
order and with checks.
In general you can be very liberal in breaking up the functions
into smaller parts, the GNU Compiler Collection (GCC) will
automatically compile the functions as inline functions when the
optimizations are turned on. So you do not have to worry about
decreasing the speed. By default Gnuastro will compile with the
‘-O3’ optimization flag.
• All Gnuastro hand-written text files (C source code, Texinfo
documentation source, and version control commit messages) should
normally be no more than *75* characters per line. Monitors today
are certainly much wider, but with this limit, reading the
functions becomes much more easier. Also for the developers, it
allows multiple files (or multiple views of one file) to be
displayed beside each other on wide monitors.
Emacs's buffers are excellent for this capability, setting a buffer
width of 80 with '' will allow you to view and work
on several files or different parts of one file using the wide
monitors common today. Emacs buffers can also be used as a shell
prompt and compile the program (with ), and 80
characters is the default width in most terminal emulators. If you
use Emacs, Gnuastro sets the 75 character ‘fill-column’ variable
automatically for you, see cartouche below.
For long comments you can use press in Emacs to separate
them into separate lines automatically. For long literal strings,
you can use the fact that in C, two strings immediately after each
other are concatenated, for example, ‘"The first part, " "and the
second part."’. Note the space character in the end of the first
part. Since they are now separated, you can easily break a long
literal string into several lines and adhere to the maximum 75
character line length policy.
• The headers required by each source file (ending with ‘.c’) should
be defined inside of it. All the headers a complete program needs
should _not_ be stacked in another header to include in all source
files (for example ‘main.h’). Although most 'professional'
programmers choose this single header method, Gnuastro is primarily
written for professional/inquisitive astronomers (who are generally
amateur programmers). The list of header files included provides
valuable general information and helps the reader. ‘main.h’ may
only include the header file(s) that define types that the main
program structure needs, see ‘main.h’ in *note Program source::.
Those particular header files that are included in ‘main.h’ can of
course be ignored (not included) in separate source files.
• The headers should be classified (by an empty line) into separate
groups:
1. ‘#include |