Documentation for new geometry file format and geoptimiser

4 files changed, 279 insertions, 16 deletions

diff --git a/doc/man/crystfel.7 b/doc/man/crystfel.7
index 9910ee18..a16d2b22 100644
--- a/doc/man/crystfel.7
+++ b/doc/man/crystfel.7
@@ -59,6 +59,9 @@ A simple viewer for images stored in HDF5 format.
 .IP \fBrender_hkl\fR
 A tool for rendering slices of reciprocal space in two dimensions.
 
+.IP \fBgeoptimiser\fR
+A program to refine and optimize detector geometry.
+
 .PP
 There is also a folder full of scripts for achieving many related tasks.
 
@@ -116,7 +119,7 @@ to this list if you experience difficulty with "-y" at any time.
 \fB23\fR, \fBm-3\fR, \fB432\fR, \fB-43m\fR, \fBm-3m\fR.
 
 .SH AUTHOR
-This page was written by Thomas White.
+This page was written by Thomas White and Valerio Mariani.
 
 .SH REPORTING BUGS
 Report bugs to <taw@physics.org>, or visit <http://www.desy.de/~twhite/crystfel>.
diff --git a/doc/man/crystfel_geometry.5 b/doc/man/crystfel_geometry.5
index 7886fd25..c7b0ed0e 100644
--- a/doc/man/crystfel_geometry.5
+++ b/doc/man/crystfel_geometry.5
@@ -19,8 +19,8 @@ program.  Programs which care about the geometry (particularly
 A flexible (and pedantic) representation of the detector has been developed to
 avoid all possible sources of ambiguity.  CrystFEL's representation of a
 detector is broken down into one or more "panels", each of which has its own
-camera length, geometry, resolution and so on.  Each panel fits into the overall
-image taken from the HDF5 file, defined by minimum and maximum coordinates in
+camera length, geometry, resolution and so on.  Each panel fits into an overall
+data block taken from the HDF5 file, defined by minimum and maximum coordinates in
 the "fast scan" and "slow scan" directions.  "Fast scan" refers to the direction
 whose coordinate changes most quickly as the bytes in the HDF5 file are moved
 through.  The coordinates are specified inclusively, meaning that a minimum of 0
@@ -31,7 +31,7 @@ In the current version, panels are assumed to be perpendicular to the incident
 beam and to have their edges parallel.  Within these limitations, any geometry
 can be constructed.
 
-The job of the geometry file is to establish a relationship between the array
+The job of the geometry file is to establish a relationship an array
 of pixel values in the HDF5 file, defined in terms only of the "fast scan" and
 "slow scan" directions, and the laboratory coordinate system defined as follows:
 
@@ -49,6 +49,13 @@ Naively speaking, this means that CrystFEL looks at the images from the "into th
 beam" perspective, but please avoid thinking of things in this way.  It's much
 better to consider the precise way in which the coordinates are mapped.
 
+Some file formats store data for multiple patterns ("events") within a single file.
+Information about the layout of the file data can be included in the geometry file.
+This allows CrystFEL to unambigously identify data blocks which contain
+data for a specific event, and to make an educated guess at the number of events
+that each file contains.
+
+
 The geometry file should contain lines of the following form:
 
 .IP
@@ -75,7 +82,39 @@ The properties which can be set are:
 .PD 0
 .IP \fBdata\fR
 .PD
-The location in the HDF5 file of the data block that contains the panel's data. If this property is not set, the first 2-dimensional data block in the file hierarchy will be used as data source.
+The location in the HDF5 file of the data block that contains the panel's data. If this property is not set, the first 2-dimensional data block in the file hierarchy will be used as data source. If the HDF5 file contains multiple events, and each event is stored in a different data block, the variable part of the path can be represented using the % character placeholder.
+
+Example:
+.IP
+data = /data/%/rawdata
+
+The CrystFEL programs will look for the first event at /data/event1_name/rawdata, for the second at /data/event2_name/rawdata, etc.
+
+.PD 0
+.IP \fBdim\fIn\fR\fR
+.PD
+Information about the layout of the data block identified by the 'data' property. \fIn\fR is an integer number identifying an axis in a multidimensional HDF5 data block. The property value defines the kind of information encoded by the axis. Possible values are:
+.br
+% : event placeholder,the axis encodes events
+.br
+ss: the axis encodes the slow scan index
+.br
+fs: the axis encodes the fast scan index
+.br
+CrystFEL assumes that the data block defined by the 'data' property has a dimensionality equal to the axis with the highest value of \fIn\fR defined by the 'dim' property, and requires the user to provide information about each of the
+axes in the data block. When no 'dim' property is defined in the geometry file, CrystFEL assumes the data block to be 2-dimensional, with the two axes encoding slow scan and fast scan information respectively.
+
+Example:
+.br
+.br
+dim0 = %
+.br
+dim1 = ss
+.br
+dim2 = fs
+.br
+.br
+The data block is 3-dimensional. The first axis encodes events, the second the slow scan panel coordinate, and the third the fast scan panel coordinate
 
 .PD 0
 .IP \fBmin_fs\fR
@@ -104,7 +143,9 @@ The resolution (in pixels per metre) for this panel.  This is one over the pixel
 .PD 0
 .IP \fBclen\fR
 .PD
-The camera length (in metres) for this panel. You can also specify the HDF path to a scalar floating point value containing the camera length in millimetres.  For example: "panel0/clen = /LCLS/detectorPosition".  See \fBcoffset\fR as well.
+The camera length (in metres) for this panel. You can also specify the HDF path to a floating point data block containing the camera length in millimetres.  For example: "panel0/clen = /LCLS/detectorPosition".  If the HDF file contains more than one event, and the data block is scalar, the camera length value
+it contains will be used for all events. If, however, the data block is multidimensional and the second dimension is bigger than one, the CrystFEL programs will try to match the content of the data block with the events in the file, assigning the first value in the data block to the first event in the file,
+the second value in the data block to the second event in the file, etc. See \fBcoffset\fR as well.
 
 .PD 0
 .IP \fBcoffset\fR
@@ -173,9 +214,42 @@ badregionB/min_ss = 256
 badregionB/max_ss = 512
 
 
+.SH RIGID GROUPS AND RIGID GROUP COLLECTIONS
+
+Sometimes, some programs in CrystFEL need to treat a group of panels as a single rigid body. This often happens when the panels are physically connected (for example, a pair of adjacent ASICs in the CSPAD detector).
+Rigid groups can be defined in the geometry file by listing the panels belonging to each group and assigning the group a name.  At a higher level, collections of rigid groups can be defined when rigid groups belong to the same
+conceptual grouping scheme (for example, a collection of the four rigid groups defining quadrants in the CSPAD detector). Definitions of rigid groups and rigid groups collection can appear at any place in the geometry file and can be declared using the following global properties (these are not panel properties and don't follow the usual panel/property syntax):
+
+.PD 0
+.IP "\fBrigid_group_\fIname\fR = panel1 <, panel2> <, panel3> ...\fR"
+.PD
+This property defines the rigid group \fIname\fR, and lists the panels that belong to it
+
+.PD 0
+.IP "\fBrigid_group_collection_\fIname\fR = rigidgroup1 <, rigidgroup2> <, rigidgroup3> ...\fR"
+.PD
+This property defines the rigid group collection \fIname\fR, and lists the rigid groups that it contains
+
 .PP
 See the "examples" folder for some examples (look at the ones ending in .geom).
 
+.SH BEAM CHARACTERISTICS
+
+The geometry file can include information about beam characteristics, using general properties, that can appear anywhere in the geometry file and do not follow the usual panel/property syntax. The following beam properties are supported:
+
+.PD 0
+.IP \fBphoton_energy\fR
+.PD
+The beam photon energy in eV. You can also specify the HDF path to a floating point data block value containing the photon energy in eV.  For example: "photon_energy = /LCLS/photon_energy_eV".  If the HDF file contains more than one event, and the data block is scalar, the photon energy value
+it contains will be used for all events. If, however, the data block is multidimensional and the second dimension is bigger than one, the CrystFEL programs will try to match the content of the data block with the events in the file, assigning the first value in the data block to the first event in the file,
+the second value in the data block to the second event in the file, etc. See also \fBphoton_energy_scale\fR.
+
+.PD 0
+.IP \fBphoton_energy_scale\fR
+.PD
+Sometimes the photon energy value recorded in an HDF5 file differs from the true photon energy value by a multiplication factor. This property defines a correction factor that is applied by the CrystFEL programs. The photon energy value read from a file is multiplied by the value of this property if the property is defined in the geometry file.
+
+
 .SH AUTHOR
 This page was written by Thomas White and Valerio Mariani.
 
diff --git a/doc/man/geoptimiser.1 b/doc/man/geoptimiser.1
new file mode 100644
index 00000000..171ebb24
--- /dev/null
+++ b/doc/man/geoptimiser.1
@@ -0,0 +1,138 @@
+.\"
+.\" geoptimiser man page
+.\"
+.\" Copyright © 2012-2014 Deutsches Elektronen-Synchrotron DESY,
+.\"                       a research centre of the Helmholtz Association.
+.\"
+.\" Part of CrystFEL - crystallography with a FEL
+.\"
+
+.TH GEOPTIMISER 1
+.SH NAME
+geoptimiser \- detector geometry refinement
+.SH SYNOPSIS
+.PP
+.BR geoptimiser
+\fB-i\fR \fIinput.stream\fR \fB-g\fR \fIinput.geom\fR \fB-o\fR \fIoutput.geom\fR \fB-c=\fR\fIconnected_rigidgroup_coll\fR \fB-q=\fR\quadrant_rigigroup_coll\fR
+[\fBoptions\fR] \fB...\fR
+.PP
+\fBgeoptimiser --help\fR
+
+.SH DESCRIPTION
+
+\fBgeoptimiser\fR refines and optimizes the detector geometry by comparing the location of observed Bragg peaks in a set of indexed patterns with the spot locations predicted from
+the crystal indexing procedure. It can refine position, rotation and distance of each module ('panel') relative to the interaction region. It requires a stream file with indexed patterns,
+a geometry file with the detector geometry to refine, and some parameters to specify which modules are physically connected to each other and which are attached to the same physical support
+(for example, all panels in a quadrant of the CSPAD detector). The output is a geometry file with the optimized detector geometry and a set of diagnostic error maps in HDF5 format.
+Several options are available to tweak the number of peaks included in the optimization procedure based on a range of criteria.
+For a complete description of the optimization algorithm, see the following paper:
+
+.IP
+O. Yefanov, V. Mariani, C. Gati, T. A. White, H. N. Chapman, and A. Barty. "Accurate determination of segmented X-ray detector geometry". In preparation
+
+.PP
+For minimal basic use, you need to provide a stream file with diffraction patterns, a geometry file to optimize, a filename for the output optimized geometry, and
+the name of two rigid group collections defined in the geometry file: one describing which modules in the detector are physically connected (and hence whose geometry should
+be optimized as if they were a single panel), and one to describe which modules are attached to the same underlying support (whose position and orientation are likely to be correlated).
+Here is what the minimal use might look like on the command line:
+
+.IP \fBgeoptimiser\fR
+.PD
+-i input.stream -g input.geom -o output.geom -c connected_rg_coll_name -q same_support_rg_coll_name
+
+.PP
+See \fBman crystfel_geometry\fR for information on how to create a file describing the detector geometry, and guidelines to define the required rigid groups and rigid groups collections.
+
+.SH OPTIONS
+.PD 0
+.IP "\fB-i\fR \fIfilename\fR"
+.IP \fB--input=\fR\fIfilename\fR
+.PD
+Read the indexed patterns from the \fIfilename\fR stream file.
+
+.PD 0
+.IP "\fB-g\fR \fIfilename\fR"
+.IP \fB--geometry=\fR\fIfilename\fR
+.PD
+Read the detector geometry to optimize from the \fIfilename\fR file.
+
+.PD 0
+.IP "\fB-o\fR \fIfilename\fR"
+.IP \fB--output=\fR\fIfilename\fR
+.PD
+Write the optimized detector geometry to \fIfilename\fR.
+
+.PD 0
+.IP "\fB-c\fR \fIname\fR"
+.IP \fB--connected=\fR\fIname\fR
+.PD
+Sets the rigid group collection for connected panels to \fIname\fR.  This rigid group collection describes how the panels are physically connected in the detector.
+A set of rigid groups must be defined in the geometry file, with each group containing only panels that are physically connected to each other (for example the pairs of physically-connected ASICs in the
+CSPAD detector). The rigid group collection chosen using this option must also be defined in the geometry file, and must collect all these groups.
+
+.PD 0
+.IP "\fB-q\fR \fIname\fR"
+.IP \fB--quadrants=\fR\fIname\fR
+.PD
+Sets the rigid group collection for quadrants to \fIname\fR.  This rigid group collection describes how panels are connected to the underlying support of the detector.
+A set of rigid groups must be defined in the geometry file, with each group containing only panels that are attached to the same underlying support (for example, all panels belonging to the same quadrant of the CSPAD detector).
+The rigid group collection chosen using this option must also be defined in the geometry file and must collect all these groups.
+
+.PD 0
+.IP "\fB-x\fR \fIn\fR"
+.IP \fB--min-num-peaks-per-pixel=\fR\fIn\fR
+.PD
+Sets to \fIn\fR the minimum number of peaks that should fall within a pixel, across all indexed patterns, to contribute to the geometry optimization. The default value is 3.
+
+.PD 0
+.IP "\fB-p\fR \fIn\fR"
+.IP \fB--min-num-peaks-per-panel=\fR\fIn\fR
+.PD
+Sets to \fIn\fR the minimum number of peaks that should appear in a panel for the panel's geometry to be optimized independently. The default value is 100.
+
+.PD 0
+.IP "\fB-l\fR"
+.IP \fB--most-freq-clen\fR
+.PD
+Some stream files can contain patterns collected using different camera lengths. By default, detector distance is optimized using only patterns collected with the most frequent camera length in the stream file.
+However, all patterns are used to compute panel shifts and rotations. With this option, shifts are rotation are computed with the same subset of patterns used for the detector distance optimization.
+
+.PD 0
+.IP "\fB-s\fR"
+.IP \fB--individual-dist-offset\fR
+.PD
+By default, geoptimiser optimizes detector distance assuming that all panels have the same distance from the sample (i.e., a single distance is optimized for the whole detector). With this option, each panel's
+distance is instead optimized independently.
+
+.PD 0
+.IP "\fB-m\fR \fIdist\fR"
+.IP \fB--max-peak-dist=\fR\fIdist\fR
+.PD
+Geoptimiser refines detector geometry by comparing the predicted position of Bragg peaks with the location of detected peak in indexed patterns. This option sets the maximum distance in pixels between the predicted and the observed peaks for the pair
+to be included in the optimization process. The default maximum distance is 8 pixels.
+
+.PD 0
+.IP \fB--no-stretch\fR
+.PD
+By default, geoptimiser refines the distance between the detector and the sample. This option turns off this optimization.
+
+.SH AUTHOR
+This page was written by Valerio Mariani and Oleksandr Yefanov.
+
+.SH REPORTING BUGS
+Report bugs to <taw@physics.org>, or visit <http://www.desy.de/~twhite/crystfel>.
+
+.SH COPYRIGHT AND DISCLAIMER
+Copyright © 2012-2014 Deutsches Elektronen-Synchrotron DESY, a research centre of the Helmholtz Association.
+.P
+geoptimiser, and this manual, are part of CrystFEL.
+.P
+CrystFEL is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+.P
+CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+.P
+You should have received a copy of the GNU General Public License along with CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
+
+.SH SEE ALSO
+.BR crystfel (7),
+.BR crystfel_geometry (5)
diff --git a/doc/man/indexamajig.1 b/doc/man/indexamajig.1
index 71f836c6..9d82dde5 100644
--- a/doc/man/indexamajig.1
+++ b/doc/man/indexamajig.1
@@ -29,12 +29,15 @@ For minimal basic use, you need to provide the list of diffraction patterns, the
 -i mypatterns.lst -j 10 -g mygeometry.geom --indexing=mosflm,dirax --peaks=hdf5 -b myxfel.beam -o test.stream -p mycell.pdb
 
 .PP
-More typical use includes all the above, but might also include extra parameters to modify the behaviour.  The HDF5 files might be in some
-folder a long way from the current directory, so you might want to specify a
-full pathname to be added in front of each filename.  You'll probably want to
+More typical use includes all the above, but might also include extra parameters to modify the behaviour. For example, you'll probably want to
 run more than one indexing job at a time (-j <n>).
 
-See \fBman crystfel_geometry\fR for information about how to create a geometry description file and a beam parameters file.
+See \fBman crystfel_geometry\fR for information about how to create a file describing detector geometry and beam characteristics.
+
+.SH DIFFRACTION PATTERN LIST
+
+Indexamajig requires an input file with a list of diffraction patterns ("events") to process. In its simplest form, this is just a text files containing a list of HDF5 filenames. The HDF5 files might be in some folder a long way from the current directory, so you might want to specify a full pathname to be added in front of each filename. The geometry file includes a description of the data layout within the HDF5 files. Indexamajig uses this description to make an educated guess at the number of diffraction patterns stored in each file, and tries to process them all.  You can however single out an event contained in the file for processing, by putting a string describing the event after the file name (see below).
+
 
 .SH PEAK DETECTION
 
@@ -205,12 +208,6 @@ Read the beam description from \fIfilename\fR.  See \fBman crystfel_geometry\fR
 Specify the name of the file containing unit cell information, in PDB or CrystFEL format.
 
 .PD 0
-.IP "\fB-e\fR \fIpath\fR"
-.IP \fB--image=\fR\fIpath\fR
-.PD
-Get the image data to display from \fIpath\fR inside the HDF5 file.  For example: \fI/data/rawdata\fR.  If this is not specified, the default behaviour is to use the first two-dimensional dataset with both dimensions greater than 64.
-
-.PD 0
 .IP \fB--peak-radius=\fR\fIinner,middle,outer\fR
 .PD
 Set the inner, middle and outer radii for three-ring integration during the peak search.  See the section about \fBPEAK INTEGRATION\fR, above, for details of how to determine
@@ -334,6 +331,57 @@ When \fBrescut\fR is in the integration method, integrate \fIn\fR nm^-1 higher t
 .PD
 Mark all pixels on the detector higher than \fIn\fR Angstroms as bad.  This might be useful when you have noisy patterns and don't expect any signal above a certain resolution.
 
+
+.SH IDENTIFYING SINGLE PATTERNS IN THE INPUT FILE
+
+By default indexamajig processes all diffraction patterns ("events") in each of the data files listed in the input list. It is however, possible, to only process single events in a multi-event file, by adding in the list an event description string after the data filename. The event description always includes a first section with alphanumeric strings separated by forward slashes ("/") and a second section with integer numbers also separated by forward slashes. The two sections are in turn separated by a double forward slash ('//'). Any of the two sections can be empty, but the double forward slash separator must always be present.  Indexmamajig matches the strings and the numbers in the event description with the event placeholders ('%') present respectively in the 'data' and 'dim' properties defined in the geometry file, and tries to retrieve the full HDF path to the event data and the the its location in a multi-dimensional data space. Consider the following examples:
+
+Example 1:
+.br
+
+Assuming that the 'data' and 'dim' properties have been defined like this in the geometry file:
+
+.br
+data = /data/%/rawdata
+.br
+dim0 = ss
+.br
+dim1 = fs
+
+The following line:
+.br
+
+filename.h5  event1//
+.br
+
+Identifies an event in the 2-dimensional data block lying at the /data/event1/rawdata HDF path in the filename.h5 file
+
+Example 2:
+.br
+
+Assuming that the 'data' and 'dim' properties have been defined like this in the geometry file:
+
+.br
+data = /data/rawdata
+.br
+dim0 = %
+.br
+dim1 = ss
+.br
+dim2 = fs
+
+The following line:
+.br
+
+filename.h5  //3
+.br
+
+Identifies an event in the 3-dimensional data block lying at the /data/rawdata HDF path in the filename.h5 file, specifically the 2-dimensional data slice defined by the value 3 of the first axis of the data space.
+
+Indexamajig tries to match the alphanumerical strings to the placeholders in the 'dim' property defined in the geometry file. The first string is matched to the first placeholder, the second to
+the second placeholder, and so on. A similar strategy is followed to match integer numbers to the placeholders in the 'dim' property defined in the geometry file.
+For a full explanation of how the internal layout of the data file can be  described in the geometry file, please see \fBman crystfel_geometry\fR.
+
 .SH BUGS
 ReAx indexing is experimental.  It works very nicely for some people, and crashes for others.  In a future version, it will be improved and fully supported.