path: root/doc/man/indexamajig.1

.\"
.\" indexamajig man page
.\"
.\" Copyright © 2012-2015 Deutsches Elektronen-Synchrotron DESY,
.\"                       a research centre of the Helmholtz Association.
.\"
.\" Part of CrystFEL - crystallography with a FEL
.\"

.TH INDEXAMAJIG 1
.SH NAME
indexamajig \- bulk indexing and data reduction program
.SH SYNOPSIS
.PP
.BR indexamajig
\fB-i\fR \fIfilename\fR \fB-o\fR \fIoutput.stream\fR \fB-g\fR \fIdetector.geom\fR \fB--peaks=\fR\fImethod\fR \fB--indexing=\fR\fImethod\fR
[\fBoptions\fR] \fB...\fR
.PP
\fBindexamajig --help\fR

.SH DESCRIPTION

\fBindexamajig\fR takes a list of diffraction snapshots from crystals in random orientations and attempts to find peaks, index and integrate each one.  The input is a list of diffraction image files in HDF5 format and some auxiliary files and parameters.  The output is a long text file ('stream') containing the results from each image in turn.

For minimal basic use, you need to provide the list of diffraction patterns, the method which will be used to index, a file describing the geometry of the detector, and a file which contains the unit cell which will be used for the indexing.  Here is what the minimal use might look like on the command line:

.IP \fBindexamajig\fR
.PD
-i mypatterns.lst -j 10 -g mygeometry.geom --indexing=mosflm,dirax --peaks=hdf5 -o test.stream -p mycell.pdb

.PP
More typical use includes all the above, but might also include extra parameters to modify the behaviour. For example, you'll probably want to
run more than one indexing job at a time (-j <n>).

See \fBman crystfel_geometry\fR for information about how to create a file describing detector geometry and beam characteristics.

.SH DIFFRACTION PATTERN LIST

Indexamajig requires an input file with a list of diffraction patterns ("events") to process. In its simplest form, this is just a text files containing a list of HDF5 filenames. The HDF5 files might be in some folder a long way from the current directory, so you might want to specify a full pathname to be added in front of each filename. The geometry file includes a description of the data layout within the HDF5 files. Indexamajig uses this description to make an educated guess at the number of diffraction patterns stored in each file, and tries to process them all.  You can however single out an event contained in the file for processing, by putting a string describing the event after the file name (see below).


.SH PEAK DETECTION

You can control the peak detection on the command line.  Firstly, you can choose the peak detection method using \fB--peaks=\fR\fImethod\fR.  Currently, two values for "method" are available.  \fB--peaks=hdf5\fR will take the peak locations from the HDF5 file.  It expects a two dimensional array, by default at /processing/hitfinder/peakinfo, whose size in the first dimension equals the number of peaks and whose size in the second dimension is three.  The first two columns contain the fast scan and slow scan coordinates, the third contains the intensity.  However, the intensity will be ignored since the pattern will always be re-integrated using the unit cell provided by the indexer on the basis of the peaks.  You can tell indexamajig where to find this table inside each HDF5 file using \fB--hdf5-peaks=\fR\fIpath\fR.

If you use \fB--peaks=zaef\fR, indexamajig will use a simple gradient search after Zaefferer (2000).  You can control the overall threshold and minimum squared gradient for finding a peak using \fB--threshold\fR and \fB--min-gradient\fR.  The threshold has arbitrary units matching the pixel values in the data, and the minimum gradient has the equivalent squared units.  Peaks will be rejected if the 'foot point' is further away from the 'summit' of the peak by more than the inner integration radius (see below).  They will also be rejected if the peak is closer than twice the inner integration radius from another peak.

You can suppress peak detection altogether for a panel in the geometry file by specifying the "no_index" value for the panel as non-zero.


.SH INDEXING METHODS

You can choose between a variety of indexing methods.  You can choose more than one method, in which case each method will be tried in turn until one of them reports that the pattern has been successfully indexed.  Choose from:

.IP \fBdirax\fR
.PD
Invoke DirAx, check linear combinations of the resulting cell axes for agreement with your cell, and then check that the cell accounts for at least half of the peaks from the peak search.
.sp
To use this option, 'dirax' must be in your shell's search path.  If you see the DirAx version and copyright information when you run \fBdirax\fR on the command line, things are set up correctly.

.IP \fBmosflm\fR
.PD
As \fBdirax\fR, but invoke MOSFLM instead.  If you provide a unit cell (with \fB-p\fR), the lattice type and centering information will be passed to MOSFLM, which will then return solutions which match.  Note that the lattice parameter information will \fBnot\fR be given to MOSFLM, because it has no way to make use of it.
.sp
To use this option, 'ipmosflm' must be in your shell's search path.  If you see the MOSFLM version and copyright information when you run \fBipmosflm\fR on the command line, things are set up correctly.

.IP \fBreax\fR
.PD
Run the DPS algorithm, looking only for lattice repeats which are close to the axes of the unit cell parameters you gave.  In theory, this method is similar to \fBmosflm\fR but should work better because of making better use of the prior cell information you gave.  In practice, it's experimental.

.IP \fBgrainspotter\fR
.PD
Invoke GrainSpotter, which will use your cell parameters to find multiple crystals in each pattern.
.sp
To use this option, 'GrainSpotter.0.93' must be in your shell's search path.  If you see the GrainSpotter version information when you run \fBGrainSpotter.0.93\fR on the command line, things are set up correctly.

.IP \fBxds\fR
.PD
Invoke XDS, and use its REFIDX procedure to attempt to index the pattern.

.PP
You can add one or more of the following to the above indexing methods:

.IP \fB-raw\fR
.PD
Do not check the resulting unit cell.  This option is useful when you need to determine the unit cell ab initio.  Use with 'dirax' and 'mosflm' - the other indexing methods need the unit cell as input in any case, and cannot determine the unit cell ab initio.  See \fB-comb\fR and \fB-axes\fR.

.IP \fB-axes\fR
.PD
Check permutations of the axes for correspondence with your cell, but do not check linear combinations.  This is useful to avoid a potential problem when one of the unit cell axis lengths is close to a multiple of one of the others.  Can be used with \fBdirax\fR and \fBmosflm\fR.  See \fB-raw\fR and \fB-comb\fR.

.IP \fB-comb\fR
.PD
Check linear combinations of the unit cell basis vectors to see if a cell can be produced which looks like your unit cell.  This is the default behaviour for \fBdirax\fR and \fBmosflm\fR.  See \fB-raw\fR and \fB-axes\fR.

.IP \fB-bad\fR
.PD
Do not check that the cell accounts for any of the peaks as described in \fBdirax\fR above.  Might be useful to debug initial indexing problems, or if there are many multi-crystal patterns and the indexing method has no concept of multiple crystals per pattern (which, at the moment, means all of them except \fBgrainspotter\fR).  Can be used with any indexing method, but is generally a bad idea.

.IP \fB-nolatt\fR
.PD
Do not use the lattice type information to help guide the indexing.  Use with \fBmosflm\fR, which is the only indexing method which can optionally take advantage of this information.  This is the default behaviour for \fBdirax\fR.  This option makes no sense for \fBreax\fR, which is intrinsically based on using known lattice information.

.IP \fB-latt\fR
.PD
This is the opposite of \fB-nolatt\fR, and is the default behaviour for \fBmosflm\fR, \fBxds\fR and \fBgrainspotter\fR.  It is the only behaviour for \fBreax\fR.

.IP \fB-cell\fR
.PD
Provide your unit cell parameters to the indexing algorithm.  This is the default for \fBxds\fR and \fBgrainspotter\fR, and the only behaviour for \fBreax\fR.  This option makes no sense for \fBdirax\fR and \fBmosflm\fR, neither of which can make use of this information.

.IP \fB-nocell\fR
.PD
Do not provide your unit cell parameters to the indexing algorithm.  This is the only behaviour for \fBmosflm\fR and \fBdirax\fR, both of which cannot make use of the information.  Can be used with \fBgrainspotter\fR and \fBxds\fR, and makes no sense for \fBreax\fR, which is intrinsically based on using known cell parameters.

.PP
The default indexing method is 'none', which means no indexing will be done.  This is useful if you just want to check that the peak detection is working properly.

.PP
Your indexing methods will be checked for validity, incompatible flags removed, and warnings given about duplicates.  For example, \fBmosflm\fR and \fBmosflm-comb-latt\fR represent the same indexing method, because \fB-comb\fR and \fB-latt\fR are the default behaviour for \fBmosflm\fR.  The 'long version' of each of your indexing methods will be listed in the output, and the stream will contain a record of which indexing method successfully indexed each pattern.

.PP
It's risky to use \fBmosflm-nolatt\fR in conjunction with either \fB-comb\fR or \fB-axes\fR when you have a rhombohedral cell.  This would be an odd thing to do anyway: why withhold the lattice information from MOSFLM if you know what it is, and want to use it to check the result?  It's risky because MOSFLM will by default return the "H centered" lattice for your rhombohedral cell, and it's not completely certain that MOSFLM consistently uses one or other of the two possible conventions for the relationship between the "H" and "R" cells.  It is, however, very likely that it does.

Examples of indexing methods: 'dirax,mosflm,reax', 'dirax-raw,mosflm-raw', 'dirax-raw-bad'.


.SH PEAK INTEGRATION
If the pattern could be successfully indexed, peaks will be predicted in the pattern and their intensities measured.  You have a choice of integration methods, and you specify the method using \fB--integration\fR.  Choose from:

.IP \fBrings\fR
.PD
Use three concentric rings to determine the peak, buffer and background estimation regions.  The radius of the smallest circle sets the peak region.  The radius of the middle and outer circles describe an annulus from which the background will be estimated.  You can set the radii of the rings using \fB--int-radius\fR (see below).  The default behaviour with \fBrings\fR is \fBnot\fR to center the peak boxes first.  Use \fBrings-cen\fR if you want to use centering.

.IP \fBprof2d\fR
.PD
Integrate the peaks using 2D profile fitting with a planar background, close to the method described by Rossmann (1979) J. Appl. Cryst. 12 p225.  The default behaviour with \fBprof2d\fR is to center the peak first - use \fBprof2d-nocen\fR to skip this step.

.PP
You can add one or more of the following to the above integration methods:

.IP \fB-cen\fR
.PD
Center the peak boxes iteratively on the actual peak locations.  The opposite is \fB-nocen\fR, which is the default.

.IP \fB-sat\fR
.PD
Normally, reflections which contain one or more pixels above max_adu (defined in the detector geometry file) will not be integrated and written to the stream.  Using this option skips this check, and allows saturated reflections to be passed to the later merging stages.  This is not usually a good idea, but might be your only choice if there are many saturated reflections.  The opposite is \fB-nosat\fR, which is the default for all integration methods.

.IP \fB-rescut\fR
.PD
Normally, reflections are integrated all the way to the edge of the detector, even if the crystal diffracts to a lower resolution.  With this option, integration will be performed up to the apparent diffraction limit of the crystal.  You can use \fB--push-res\fR (see below) to integrate to a slightly higher or lower resolution.  The resolution limit is determined by comparing the peaks found by the peak search to the indexing results, so good peak detection is essential when using this option (as it is always).  The opposite is \fB-norescut\fR, which is the default.

.SH OPTIMISING THE INTEGRATION RADII
To determine appropriate values for the integration radii, index some patterns with the default values and view the results using \fBcheck-near-bragg\fR (in the scripts folder).  Set the binning in \fBhdfsee\fR to 1, and adjust the ring radius until none of the rings overlap for any of the patterns.  This ring radius is the outer radius to use. Then reduce the radius until the circles match the sizes of the peaks as closely as possible.  This value is the inner radius.  The middle radius should be between the two, ideally between two and three pixels smaller than the outer radius.
.PP
If it's difficult to do this without setting the middle radius to the
same value as the inner radius, then the peaks are too close together to be
accurately integrated.  Perhaps you got greedy with the resolution and put the
detector too close to the interaction region?

.SH OPTIONS
.PD 0
.IP "\fB-i\fR \fIfilename\fR"
.IP \fB--input=\fR\fIfilename\fR
.PD
Read the list of images to process from \fIfilename\fR.  The default is \fB--input=-\fR, which means to read from stdin.

.PD 0
.IP "\fB-o\fR \fIfilename\fR"
.IP \fB--output=\fR\fIfilename\fR
.PD
Write the output data stream to \fIfilename\fR.  The default is \fB--output=-\fR, which means to write to stdout.

.PD 0
.IP \fB--peaks=\fR\fImethod\fR
.PD
Find peaks in the images using \fImethod\fR.  See the second titled \fBPEAK DETECTION\fB (above) for more information.

.PD 0
.IP \fB--indexing=\fR\fImethod\fR
.PD
Index the patterns using \fImethod\fR.  See the section titled \fBINDEXING METHODS\fR (above) for more information.  The default is \fB--indexing=none\fR.

.PD 0
.IP \fB--integration=\fR\fImethod\fR
.PD
Integrate the reflections using \fImethod\fR.  See the section titled \fBPEAK INTEGRATION\fR (above) for more information.  The default is \fB--integration=rings-nocen\fR.


.PD 0
.IP "\fB-g\fR \fIfilename\fR"
.IP \fB--geometry=\fR\fIfilename\fR
.PD
Read the detector geometry description from \fIfilename\fR.  See \fBman crystfel_geometry\fR for more information.

.PD 0
.IP "\fB-p\fR \fIunitcell.cell\fR"
.IP "\fB-p\fR \fIunitcell.pdb\fR"
.IP \fB--pdb=\fR\fIunitcell.pdb\fR
.PD
Specify the name of the file containing unit cell information, in PDB or CrystFEL format.

.PD 0
.IP \fB--peak-radius=\fR\fIinner,middle,outer\fR
.PD
Set the inner, middle and outer radii for three-ring integration during the peak search.  See the section about \fBPEAK INTEGRATION\fR, above, for details of how to determine
these.  The default is to use the same values as for \fB--int-radius\fR.

.PD 0
.IP \fB--int-radius=\fR\fIinner,middle,outer\fR
.PD
Set the inner, middle and outer radii for three-ring integration.  See the
section about \fBPEAK INTEGRATION\fR, above, for details of how to determine
these.  The defaults are probably not appropriate for your situation.
.PD
The default is \fB--int-radius=4,5,7\fR.

.PD 0
.IP \fB--basename\fR
.PD
Remove the directory parts of the filenames taken from the input file.  If \fB--prefix\fR or \fB-x\fR is also given, the directory parts of the filename will be removed \fIbefore\fR adding the prefix.

.PD 0
.IP "\fB-x\fR \fIprefix\fR"
.IP \fB--prefix=\fR\fIprefix\fR
.PD
Prefix the filenames from the input file with \fIprefix\fR.  If \fB--basename\fR is also given, the filenames will be prefixed \fIafter\fR removing the directory parts of the filenames.

.PD 0
.IP \fB--hdf5-peaks=\fR\fIpath\fR
.PD
When using \fB--peaks=hdf5\fR, read the peak locations from a table in the HDF5 file located at \fIpath\fR.

.PD 0
.IP \fB--tolerance=\fR\fItol\fR
.PD
Set the tolerances for unit cell comparison.  \fItol\fR takes the form \fIa\fR,\fIb\fR,\fIc\fR,\fIang\fR.  \fIa\fR, \fIb\fR and \fIc\fR are the tolerances, in percent, for the respective \fIreciprocal\fR space axes, and \fIang\fR is the tolerance in degrees for the reciprocal space angles.  If the unit cell is centered, the tolerances are applied to the corresponding primitive unit cell.
.PD
The default is \fB--tolerance=5,5,5,1.5\fR.

.PD 0
.IP \fB--median-filter=\fR\fIn\fR
.PD
Apply a median filter with box "radius" \fIn\fR to the image.  The median of the values from a \fI(n+1)\fRx\fI(n+1)\fR square centered on the pixel will be subtracted from each pixel.  This might help with peak detection if the background is high and/or noisy.  The \fIunfiltered\fR image will be used for the final integration of the peaks.  If you also use \fB--noise-filter\fR, the median filter will be applied first.


.PD 0
.IP \fB--filter-noise\fR
.PD
Apply a noise filter to the image with checks 3x3 squares of pixels and sets all of them to zero if any of the nine pixels have a negative value.  This filter may help with peak detection under certain circumstances.  The \fIunfiltered\fR image will be used for the final integration of the peaks, because the filter is destroys a lot of information from the pattern.  If you also use \fB--median-filter\fR, the median filter will be applied first.

.PD 0
.IP \fB--no-sat-corr\fR
.PD
This option is here for historical purposes only, to disable a correction which is done if certain extra information is included in the HDF5 file.

.PD 0
.IP \fB--threshold=\fR\fIthres\fR
.PD
Set the overall threshold for peak detection using \fB--peaks=zaef\fR to \fIthres\fR, which has the same units as the detector data.  The default is \fB--threshold=800\fR.

.PD 0
.IP \fB--min-gradient=\fR\fIgrad\fR
.PD
Set the gradient threshold for peak detection using \fB--peaks=zaef\fR to \fIgrad\fR, which units of "detector units per pixel".  The default is \fB--min-gradient=100000\fR.

.PD 0
.IP \fB--min-snr=\fR\fIsnr\fR
.PD
Set the minimum I/sigma(I) for peak detection when using \fB--peaks=zaef\fR.  The default is \fB--min-snr=5\fR.

.PD 0
.IP \fB--copy-hdf5-field=\fR\fIpath\fR
.PD
Copy the information from \fIpath\fR in the HDF5 file into the output stream.  The information must be a single scalar value.  This option is sometimes useful to allow data to be separated after indexing according to some condition such the presence of an optical pump pulse.  You can give this option as many times as you need to copy multiple bits of information.

.PD 0
.IP "\fB-j\fR \fIn\fR"
.PD
Run \fIn\fR analyses in parallel.  Default: 1.

.PD 0
.IP \fB--no-check-prefix\fR
.PD
Don't attempt to correct the prefix (see \fB--prefix\fR) if it doesn't look correct.

.PD 0
.IP \fB--no-use-saturated\fR
.PD
Normally, peaks which contain one or more pixels above max_adu (defined in the detector geometry file) will be used for indexing (but not used in the final integration - see the section on peak integration above).  Using this option causes saturated peaks to be ignored completely.  The opposite is \fB--use-saturated\fR, which is the default.

.PD 0
.IP \fB--no-revalidate\fR
.PD
When using \fB--peaks=hdf5\fR, the peaks will be put through some of the same checks as if you were using \fB--peaks=zaef\fR.  These checks reject peaks which are too close to panel edges, are saturated (unless you use \fB--use-saturated\fR), have other nearby peaks (closer than twice the inner integration radius, see \fB--int-radius\fR), or have any part in a bad region.  Using this option skips this validation step, and uses the peaks directly.

.PD 0
.IP \fB--check-hdf5-snr\fR
.PD
With this option with \fB--peaks=hdf5\fR, the peaks will additionally be checked to see that they satisfy the minimum SNR specified with \fB--min-snr\fR.

.PD 0
.IP \fB--no-peaks-in-stream\fR
.PD
Do not record peak search results in the stream.  You won't be able to check that the peak detection was any good, but the stream will be around 30% smaller.

.PD 0
.IP \fB--no-refls-in-stream\fR
.PD
Do not record integrated reflections in the stream.  The resulting output won't be usable for merging, but will be a lot smaller.  This option might be useful if you're only interested in things like unit cell parameters and orientations.

.PD 0
.IP \fB--int-diag=\fIcondition\fR
.PD
Show detailed information about reflection integration when \fIcondition\fR is met.  The \fIcondition\fR can be \fBall\fR, \fBnone\fR, a set of Miller indices separated by commas, \fBrandom\fR, \fBimplausible\fR or \fBnegative\fR.  \fBrandom\fR means to show information about a random 1% of the peaks.  \fBnegative\fR means to show peaks with intensities which are negative by more than 3 sigma.  \fBimplausible\fR means to show peaks with intensities which are negative by more than 5 sigma.  \fBstrong\fR means to show peaks with intensities which are positive by more than 3 sigma  The default is \fB--int-diag=none\fR.

.PD 0
.IP \fB--push-res=\fIn\fR
.PD
When \fBrescut\fR is in the integration method, integrate \fIn\fR nm^-1 higher than the apparent resolution limit of each individual crystal.  If \fBrescut\fR is not used, this option has no effect.  \fIn\fR can be negative to integrate \fIlower\fR than the apparent resolution limit.  The default is \fB--rescut=0\fR.  Note that you can also apply this cutoff at the merging stage using \fBprocess_hkl --push-res\fR.

.PD 0
.IP \fB--highres=\fIn\fR
.PD
Mark all pixels on the detector higher than \fIn\fR Angstroms as bad.  This might be useful when you have noisy patterns and don't expect any signal above a certain resolution.


.PD 0
.IP \fB--fix-profile-radius=\fIn\fR
.IP \fB--fix-bandwidth=\fIn\fR
.IP \fB--fix-divergence=\fIn\fR
.PD
Fix the beam and crystal paramters to the given values.  The profile radius is given in m^-1, the bandwidth as a decimal fraction and the divergence in radians (full angle).  The default is to set the divergence to zero, the bandwidth to a very small value, and then to automatically determine the profile radius.
.IP
You do not have to use all three of these options together.  For example, if the automatic profile radius determination is not working well for your data set, you could fix that alone and continue using the default values for the other parameters (which might be automatically determined in future versions of CrystFEL, but are not currently).

.SH IDENTIFYING SINGLE PATTERNS IN THE INPUT FILE

By default indexamajig processes all diffraction patterns ("events") in each of the data files listed in the input list. It is however, possible, to only process single events in a multi-event file, by adding in the list an event description string after the data filename. The event description always includes a first section with alphanumeric strings separated by forward slashes ("/") and a second section with integer numbers also separated by forward slashes. The two sections are in turn separated by a double forward slash ('//'). Any of the two sections can be empty, but the double forward slash separator must always be present.  Indexamajig matches the strings and the numbers in the event description with the event placeholders ('%') present respectively in the 'data' and 'dim' properties defined in the geometry file, and tries to retrieve the full HDF path to the event data and the the its location in a multi-dimensional data space. Consider the following examples:

Example 1:
.br

Assuming that the 'data' and 'dim' properties have been defined like this in the geometry file:

.br
data = /data/%/rawdata
.br
dim0 = ss
.br
dim1 = fs

The following line:
.br

filename.h5  event1//
.br

Identifies an event in the 2-dimensional data block lying at the /data/event1/rawdata HDF path in the filename.h5 file

Example 2:
.br

Assuming that the 'data' and 'dim' properties have been defined like this in the geometry file:

.br
data = /data/rawdata
.br
dim0 = %
.br
dim1 = ss
.br
dim2 = fs

The following line:
.br

filename.h5  //3
.br

Identifies an event in the 3-dimensional data block lying at the /data/rawdata HDF path in the filename.h5 file, specifically the 2-dimensional data slice defined by the value 3 of the first axis of the data space.

Indexamajig tries to match the alphanumerical strings to the placeholders in the 'dim' property defined in the geometry file. The first string is matched to the first placeholder, the second to
the second placeholder, and so on. A similar strategy is followed to match integer numbers to the placeholders in the 'dim' property defined in the geometry file.
For a full explanation of how the internal layout of the data file can be  described in the geometry file, please see \fBman crystfel_geometry\fR.

.SH BUGS
ReAx indexing is experimental.  It works very nicely for some people, and crashes for others.  In a future version, it will be improved and fully supported.

.SH AUTHOR
This page was written by Thomas White.

.SH REPORTING BUGS
Report bugs to <taw@physics.org>, or visit <http://www.desy.de/~twhite/crystfel>.

.SH COPYRIGHT AND DISCLAIMER
Copyright © 2012-2015 Deutsches Elektronen-Synchrotron DESY, a research centre of the Helmholtz Association.
.P
indexamajig, and this manual, are part of CrystFEL.
.P
CrystFEL is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
.P
CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
.P
You should have received a copy of the GNU General Public License along with CrystFEL.  If not, see <http://www.gnu.org/licenses/>.

.SH SEE ALSO
.BR crystfel (7),
.BR crystfel_geometry (5),
.BR cell_explorer (1),
.BR process_hkl (1),
.BR partialator (1)