path: root/doc/man/indexamajig.1

.\"
.\" indexamajig man page
.\"
.\" (c) 2009-2011 Thomas White <taw@physics.org>
.\"
.\" Part of CrystFEL - crystallography with a FEL
.\"

.TH INDEXAMAJIG 1
.SH NAME
indexamajig \- bulk indexing and data reduction program
.SH SYNOPSIS
.PP
.B indexamajig
[options]

.SH DESCRIPTION

The "indexamajig" program takes as input a list of diffraction image files,
currently in HDF5 format.  For each image, it attempts to find peaks and then
index the pattern.  If successful, it will measure the intensities of the peaks
at Bragg locations and produce a list of integrated peaks and intensities (and
so on) in an output text file known as a "stream".

For minimal basic use, you need to provide the list of diffraction patterns,
the method which will be used to index, a file describing the geometry of the
detector, a PDB file which contains the unit cell which will be used for the
indexing, and that you'd like the program to output a list of intensities for
each successfully indexed pattern.  Here is what the minimal use might look like
on the command line:

indexamajig -i mypatterns.lst -j 10 -g mygeometry.geom \
.br
            --indexing=mosflm,dirax --peaks=hdf5 --cell-reduction=reduce
.br
            -b myxfel.beam -o test.stream -p mycell.pdb --record=integrated

More typical use includes all the above, but might also include a noise or
common mode filter (--filter-noise or --filter-cm respectively) if detector
noise causes problems for the peak detection.  The HDF5 files might be in some
folder a long way from the current directory, so you might want to specify a
full pathname to be added in front of each filename.  You'll probably want to
run more than one indexing job at a time (-j <n>).

You can include a table of saturation values for in the HDF5 file, if you have
a method for estimating the intensities of saturated peaks.  It goes in
/processing/hitfinder/peakinfo_saturated, and should be an n*3 two dimensional
array, where the first two columns contain fast scan and slow scan coordinates
(in that order) and the third contains the value which should belong in a peak
at the given location.  The value will be spread in a small cross centred on
that location.

See `man crystfel_geometry' for information about how to create a geometry description file.

You can control what information is included in the output stream using
' --record=<flags>'.  Possible flags are:

 integrated        Include a list of reflection intensities, produced by
                   integrating around predicted peak locations.

 peaks             Include peak locations and intensities from the peak
                   search.

 peaksifindexed    As 'peaks', but only if the pattern could be indexed.

 peaksifnotindexed As 'peaks', but only if the pattern could NOT be indexed.

So, if you just want the integrated intensities of indexed peaks, use
"--record=integrated".  If you just want to check that the peak detection is
working, used "--record=peaks".  If you want the integrated peaks for the
indexable patterns, but also want to check the peak detection for the patterns
which could not be indexed, you might use
"--record=integrated,peaksifnotindexed" and then use "check-peak-detection" from
the "scripts" folder to visualise the results of the peak detection.

.SH PEAK DETECTION

You can control the peak detection on the command line.  Firstly, you can choose
the peak detection method using "--peaks=<method>".  Currently, two possible
values for "method" are available.  "hdf5" will take the peak locations from the
HDF5 file.  It expects a two dimensional array at /processing/hitfinder/peakinfo
where size in the first dimension is the number of peaks and the size in the
second dimension is three.  The first two columns contain the x and y
coordinate (see the "Note about data orientation" in geometry.txt for details),
the third contains the intensity.  However, the intensity will be ignored since
the pattern will always be re-integrated using the unit cell provided by the
indexer on the basis of the peaks.

The "zaef" method uses a simple gradient search after Zaefferer (2000).  You can
control the overall threshold and minimum gradient for finding a peak using the
"--threshold" and "--min-gradient" options.  Both of these have units of "ADU"
(i.e. units of intensity according to the contents of the HDF5 file).

A minimum peak separation can also be provided in the geometry description file
(see geometry.txt for details).  This number serves two purposes.  Firstly,
it is the maximum distance allowed between the peak summit and the foot point
(where the gradient exceeds the minimum gradient).  Secondly, it is the minimum
distance allowed between one peak and another, before the later peak will be
rejected "by proximity".

You can suppress peak detection altogether for a panel in the geometry file by
specifying the "no_index" value for the panel as non-zero.


.SH INDEXING METHODS

You can choose between a variety of indexing methods.  You can choose more than
one method, in which case each method will be tried in turn until the later cell
reduction step says that the cell is a "hit".  Choose from:

 dirax    : invoke DirAx
 mosflm   : invoke MOSFLM (DPS)

Depending on what you have installed.  For "dirax" and "mosflm", you need to
have the dirax or ipmosflm binaries in your PATH.

Example:  --indexing=dirax,mosflm

.SH CELL REDUCTION

You can choose from various options for cell reduction with the
"--cell-reduction=" option.  The choices are "none", "reduce" and "compare".
This choice is important because all autoindexing methods produce an "ab
initio" estimate of the unit cell (nine parameters), rather than just finding
the orientation of the target cell (three parameters).  It's clear that this is
not optimal, and will hopefully be fixed in future versions.

With "none", the raw cell from the autoindexer will be used.  The cell probably
won't match the target cell, but it'll still get used.  Use this option to test
whether the patterns are basically "indexable" or not, or if you don't know the
cell parameters.  In the latter case, you'll need to plot some kind of histogram
of the resulting parameters from the output stream to see which are the most
popular.  If you're lucky, this will reveal the true unit cell.

With "reduce", linear combinations of the raw cell will be checked against the
target cell.  If at least one candidate is found for each axis of the target
cell, the angles will be checked to correspondence.  If a match is found, this
cell will be used for further processing.  This option should generate the most
matches, but might produce spurious results in many cases.  The predicted peaks
are always checked to verify that at least 10% of the predicted peaks are close
to peaks located by the peak search.  If not, the next candidate unit cell is
tried until there are no more options.

The "compare" method is like "reduce", but linear combinations are not taken.
That means that the cell must either match or match after a simple permutation
of the axes.  This is useful when the target cell is subject to reticular
twinning, such as if one cell axis length is close to twice another.  With
"reduce", there is a possibility that the axes might be confused in this
situation.  This happens for lysozyme (1VDS), so watch out.

The tolerance for matching with "reduce" and "compare" is hardcoded as 5% in
the reciprocal axis lengths and 1.5 degrees in the (reciprocal) angles.  Cells
from these reduction routines are further constrained to be right-handed.  The
unmatched raw cell might be left-handed: CrystFEL doesn't check this for you.
Always using a right-handed cell means that the Bijvoet pairs can be told
apart.

If the unit cell is centered (i.e. if the space group begins with I, R, C, A or
F), you should be careful when using "compare" for the cell reduction, since
(for example) DirAx will always find a primitive unit cell, and this cell must
be converted to the non-primitive conventional cell from the PDB.


.SH TUNING CPU AFFINITIES FOR NUMA HARDWARE

If you are running indexamajig on a NUMA (non-uniform memory architecture)
machine, a performance gain can sometimes be made by preventing the kernel from
allowing a process or thread to run on a CPU which is distant from the one on
which it started.  Distance, in this context, might mean that the CPU is able to
access all the memory visible to the original CPU, but perhaps only relatively
slowly via a cable link.  In many cases a group of CPUs will have direct access
to a certain region of memory, and so the process may be scheduled on any CPU in
that group without any penalty.  However, scheduling the process to any CPU
outside the group may be slow.  When running under Linux, indexamajig is able to
avoid such sub-optimal process scheduling by setting CPU affinities for its
threads.  The CPU affinities are also inherited by subprocesses (e.g. MOSFLM or
DirAx).

To do this usefully, you need to give indexamajig some information about your
hardware's architecture.  Specify the size of the CPU groups using
"--cpugroup=<n>".  You also need to specify the overall number of CPUs, so that
the program knows when to 'wrap around'.  Using "--cpuoffset=<n>", where "n" is
a group number (not a CPU number), allows you to manually skip a few CPUs, which
may be useful if you do not want to use all the available CPUs but want to avoid
running all your jobs on the same ones.

Note that specifying the above options is NOT the same thing as giving the
number of analyses to run in parallel (the 'number of threads'), which is done
with "-j <n>".  The CPU tuning options provide information to indexamajig about
how to set the CPU affinities for its threads, but it does not specify how many
threads to use.

Example: 72-core Altix UV 100 machine at the author's institution

This machine consists of six blades, each containing two 6-core CPUs and some
local memory.  Any CPU on any blade can access the memory on any other blade,
but the access will be slow compared to accessing memory on the same blade.
When running two instances of indexamajig, a sensible choice of parameters might
be:

1: --cpus=72 --cpugroup=12 --cpuoffset=0 -j 36
2: --cpus=72 --cpugroup=12 --cpuoffset=36 -j 36

This would dedicate half of the CPUs to one instance, and the other half to the
other.


.SH A NOTE ABOUT UNIT CELL SETTINGS

CrystFEL's core symmetry module only knows about one setting for each unit cell.
You must use the same setting for now, but this will be improved in future
versions.  The cell settings are the standard ones from the International
Tables (2006).  For monoclinic cells, the unique axis should be "b".  For
orthorhombic cells in point group mm2, the twofold axis should be along "c",
i.e. no mirror perpendicular to "c".  For tetragonal cells and hexagonal
lattices, the unique axis should be "c".


.SH KNOWN BUGS

Don't run more than one indexamajig jobs simultaneously in the same working
directory - they'll overwrite each other's DirAx or MOSFLM files, causing subtle
problems which can't easily be detected.