9 files changed, 60 insertions, 41 deletions
diff --git a/doc/man/ambigator.1 b/doc/man/ambigator.1
index d403f2aa..fad89641 100644
--- a/doc/man/ambigator.1
+++ b/doc/man/ambigator.1
@@ -25,7 +25,7 @@ The algorithm proceeds by calculating the individual correlation coefficients be
 
 Only one indexing ambiguity can be resolved at a time.  In other words, each crystal is considered to be indexable in one of only two ways.  If the true indexing ambiguity has more possibilities than this, the resolution must be performed by running \fBambigator\fR multiple times with a different ambiguity operator each time.
 
-If the ambiguity operator is known (or, equivalently, the actual and apparent symmetries are both known), then the algorithm can be enhanced by including in \fIf\fR the correlation coefficients of all the crystals with the opposite indexing assignment to the current one, but after reindexing the other crystal first.  Likewise, \fg\fR includes the correlation coefficients of the crystals with the same indexing assignment after reindexing.  This enhances the algorithm to an extent roughly equivalent to doubling the number of crystals.
+If the ambiguity operator is known (or, equivalently, the actual and apparent symmetries are both known), then the algorithm can be enhanced by including in \fIf\fR the correlation coefficients of all the crystals with the opposite indexing assignment to the current one, but after reindexing the other crystal first.  Likewise, \fIg\fR includes the correlation coefficients of the crystals with the same indexing assignment after reindexing.  This enhances the algorithm to an extent roughly equivalent to doubling the number of crystals.
 
 The default behaviour is to compare each crystal to every other crystal.  This leads to a computational complexity proportional to the square of the number of crystals.  If the number of crystals is large, the number of comparisons can be limited without compromising the algorithm much.  In this case, the crystals to correlate against will be selected randomly.
 
diff --git a/doc/man/check_hkl.1 b/doc/man/check_hkl.1
index ad66dfb2..f02b581b 100644
--- a/doc/man/check_hkl.1
+++ b/doc/man/check_hkl.1
@@ -17,7 +17,7 @@ check_hkl \- calculate figures of merit for reflection data
 \fBcheck_hkl --help\fR
 
 .SH DESCRIPTION
-check_hkl calculates figures of merit for reflection data, such as completeness and average signal strengths, in resolution shells.  check_hkl accepts a single reflection list in CrystFEL's format, and you must also provide a unit cell (in a PDB file).
+check_hkl calculates figures of merit for reflection data, such as completeness and average signal strengths, in resolution shells.  check_hkl accepts a single reflection list in CrystFEL's format, and you must also provide a unit cell (in a PDB file or CrystFEL unit cell format).
 
 .SH OPTIONS
 .PD 0
diff --git a/doc/man/compare_hkl.1 b/doc/man/compare_hkl.1
index b628f0b6..8e464d12 100644
--- a/doc/man/compare_hkl.1
+++ b/doc/man/compare_hkl.1
@@ -12,12 +12,12 @@
 compare_hkl \- compare reflection data
 .SH SYNOPSIS
 .PP
-\fBcompare_hkl\fR \fR [\fIoptions\fR] \fB...\fR \fIfile1.hkl\fR \fIfile2.hkl\fR
+\fBcompare_hkl\fR [\fIoptions\fR] \fB... \fIfile1.hkl \fIfile2.hkl\fR -p \fIcell.pdb\fR
 .PP
 \fBcompare_hkl --help\fR
 
 .SH DESCRIPTION
-compare_hkl compares two sets of reflection data and calculates figures of merit such as R-factors.  Reflections will be considered equivalent according to your choice of point group.
+compare_hkl compares two sets of reflection data and calculates figures of merit such as R-factors.  Reflections will be considered equivalent according to your choice of point group. You need to provide a unit cell, as a PDB file or a CrystFEL unit cell file.
 
 .SH OPTIONS
 .PD 0
diff --git a/doc/man/crystfel.7 b/doc/man/crystfel.7
index fd3f337c..bc545069 100644
--- a/doc/man/crystfel.7
+++ b/doc/man/crystfel.7
@@ -25,7 +25,7 @@ The crystal orientations in each pattern are random and uncorrelated, which lead
 .RE
 
 CrystFEL includes programs for simulating and processing patterns subject to the
-above characteristics.  Twelve programs form the core of CrystFEL.  They are:
+above characteristics.  Fifteen programs form the core of CrystFEL.  They are:
 
 .IP \fBindexamajig\fR
 Batch indexing, integration and data reduction program, which produces a "stream" containing the indexing and integration results for each diffraction pattern.
@@ -63,6 +63,12 @@ A tool for rendering slices of reciprocal space in two dimensions.
 .IP \fBgeoptimiser\fR
 A program to refine and optimize detector geometry.
 
+.IP \fBlist_events\fR
+A tool for creating lists of events from multi-event data files.
+
+.IP \fBwhirligig\fR
+A tool for locating runs of crystals with similar orientations, e.g. from 'mini rotation series' arising from the use of a slow extrusion sample injector.
+
 .PP
 There is also a folder full of scripts for achieving many related tasks.
 
@@ -149,4 +155,7 @@ You should have received a copy of the GNU General Public License along with Cry
 .BR render_hkl (1),
 .BR hdfsee (1),
 .BR get_hkl (1),
+.BR geoptimiser (1),
+.BR whirligig (1),
+.BR list_events (1),
 .BR crystfel_geometry (5).
diff --git a/doc/man/crystfel_geometry.5 b/doc/man/crystfel_geometry.5
index fe4abc7f..974d602b 100644
--- a/doc/man/crystfel_geometry.5
+++ b/doc/man/crystfel_geometry.5
@@ -55,14 +55,13 @@ Information about the layout of the file data can be included in the geometry fi
 This allows CrystFEL to unambigously identify data blocks which contain
 data for a specific event, and to determine the number of events that each file contains.
 
-
 The geometry file should contain lines of the following form:
 
 .IP
-\fIpanel\fR\fB/\fIproperty\fB = \fIvalue\fR
+\fIpanel\fR/clen = 0.560
 
 .PP
-\fIpanel\fR can be any name of your choosing.  You can make up names for your panels however you please, as long as the first three letters are not "\fBbad\fR" (in lower case), because this is used to identify bad regions.
+\fIpanel\fR can be any name of your choosing.  You can make up names for your panels however you please, as long as the first three letters are not "\fBbad\fR" or "rigid_group" (in lower case), because these are used for special purposes (see below).
 
 .PP
 You can also specify values without a panel name, for example:
@@ -133,7 +132,7 @@ The range of pixels in the data block specified by the 'data' property that corr
 .PD 0
 .IP \fBadu_per_eV\fR
 .PD
-The number of detector intensity units (ADU) which will arise from one electron-Volt of photon energy.  This is used to estimate Poisson errors.
+The number of detector intensity units (ADU) which will arise from one electron-Volt of photon energy.  This is used to estimate Poisson errors.  Note that setting different values for this parameter for different panels does \fBnot\fR result in the intensities being scaled accordingly.
 
 .PD 0
 .IP \fBbadrow_direction\fR
@@ -187,7 +186,7 @@ Set this to 1 or "true" to ignore this panel completely.
 .IP \fBmask_bad\fR
 .PD
 Bitmasks for bad pixel masks. The pixel is considered good if all of the bits which are set in \fBmask_good\fR are set, \fIand\fR if none of the bits which are set in \fBmask_bad\fR are set. Example:
-.br
+.IP
 mask = /processing/hitfinder/masks
 .br
 mask_good = 0x27
@@ -220,6 +219,8 @@ badregionB/max_fs = 160
 badregionB/min_ss = 256
 .br
 badregionB/max_ss = 512
+.br
+badregionB/panel = q0a1
 
 
 .SH RIGID GROUPS AND RIGID GROUP COLLECTIONS
diff --git a/doc/man/geoptimiser.1 b/doc/man/geoptimiser.1
index 19ce4534..cfd32e2d 100644
--- a/doc/man/geoptimiser.1
+++ b/doc/man/geoptimiser.1
@@ -14,7 +14,7 @@ geoptimiser \- detector geometry refinement
 .PP
 .BR geoptimiser
 \fB-i \fIinput.stream \fB-g \fIinput.geom \fB-o \fIoutput.geom \fB-c \fIconnected_rigidgroup_coll \fB-q \fI\quadrant_rigidgroup_coll\fR
-[\fBoptions\fR] \fB...\fR
+[\fBoptions\fR]
 .PP
 \fBgeoptimiser --help\fR
 
@@ -38,7 +38,7 @@ See \fBman crystfel_geometry\fR for information on how to create a file describi
 .IP "\fB-i\fR \fIfilename\fR"
 .IP \fB--input=\fR\fIfilename\fR
 .PD
-Read the indexed patterns from the \fIfilename\fR stream file.
+Give the filename of the stream from which to read the indexed patterns.
 
 .PD 0
 .IP "\fB-g\fR \fIfilename\fR"
@@ -115,7 +115,7 @@ This page was written by Valerio Mariani, Oleksandr Yefanov and Thomas White.
 Report bugs to <taw@physics.org>, or visit <http://www.desy.de/~twhite/crystfel>.
 
 .SH COPYRIGHT AND DISCLAIMER
-Copyright © 2012-2015 Deutsches Elektronen-Synchrotron DESY, a research centre of the Helmholtz Association.
+Copyright © 2014-2015 Deutsches Elektronen-Synchrotron DESY, a research centre of the Helmholtz Association.
 .P
 geoptimiser, and this manual, are part of CrystFEL.
 .P
diff --git a/doc/man/indexamajig.1 b/doc/man/indexamajig.1
index dc40a573..9b587a88 100644
--- a/doc/man/indexamajig.1
+++ b/doc/man/indexamajig.1
@@ -26,22 +26,24 @@ For minimal basic use, you need to provide the list of diffraction patterns, the
 
 .IP \fBindexamajig\fR
 .PD
--i mypatterns.lst -j 10 -g mygeometry.geom --indexing=mosflm,dirax --peaks=hdf5 -o test.stream -p mycell.pdb
+-i mypatterns.lst -g mygeometry.geom --indexing=mosflm,dirax --peaks=hdf5 -o test.stream -p mycell.pdb
 
 .PP
 More typical use includes all the above, but might also include extra parameters to modify the behaviour. For example, you'll probably want to
 run more than one indexing job at a time (-j <n>).
 
-See \fBman crystfel_geometry\fR for information about how to create a file describing detector geometry and beam characteristics.
+See \fBman crystfel_geometry\fR for information about how to create a file describing the detector geometry and beam characteristics.
 
 .SH DIFFRACTION PATTERN LIST
 
-Indexamajig requires an input file with a list of diffraction patterns ("events") to process. In its simplest form, this is just a text files containing a list of HDF5 filenames. The HDF5 files might be in some folder a long way from the current directory, so you might want to specify a full pathname to be added in front of each filename. The geometry file includes a description of the data layout within the HDF5 files. Indexamajig uses this description to make an educated guess at the number of diffraction patterns stored in each file, and tries to process them all.  You can however single out an event contained in the file for processing, by putting a string describing the event after the file name (see below).
+Indexamajig requires an input file with a list of diffraction patterns ("events") to process. In its simplest form, this is just a text files containing a list of HDF5 filenames. The HDF5 files might be in some folder a long way from the current directory, so you might want to specify a full pathname to be added in front of each filename. The geometry file includes a description of the data layout within the HDF5 files. Indexamajig uses this description to determine the number of diffraction patterns stored in each file, and tries to process them all.  You can also specify explicity which event(s) you would like to process by putting a string describing the event after the file name(s) in this list.
 
 
 .SH PEAK DETECTION
 
-You can control the peak detection on the command line.  Firstly, you can choose the peak detection method using \fB--peaks=\fR\fImethod\fR.  Currently, two values for "method" are available.  \fB--peaks=hdf5\fR will take the peak locations from the HDF5 file.  It expects a two dimensional array, by default at /processing/hitfinder/peakinfo, whose size in the first dimension equals the number of peaks and whose size in the second dimension is three.  The first two columns contain the fast scan and slow scan coordinates, the third contains the intensity.  However, the intensity will be ignored since the pattern will always be re-integrated using the unit cell provided by the indexer on the basis of the peaks.  You can tell indexamajig where to find this table inside each HDF5 file using \fB--hdf5-peaks=\fR\fIpath\fR.
+You can control the peak detection on the command line.  Firstly, you can choose the peak detection method using \fB--peaks=\fR\fImethod\fR.  There are three possibilities for "method" here.  \fB--peaks=hdf5\fR will take the peak locations from the HDF5 file.  It expects a two dimensional array, by default at /processing/hitfinder/peakinfo, whose size in the first dimension equals the number of peaks and whose size in the second dimension is three.  The first two columns contain the fast scan and slow scan coordinates, the third contains the intensity.  However, the intensity will be ignored since the pattern will always be re-integrated using the unit cell provided by the indexer on the basis of the peaks.  You can tell indexamajig where to find this table inside each HDF5 file using \fB--hdf5-peaks=\fR\fIpath\fR.
+
+\fB--peaks=cxi\fR works similarly to this, but expects four separate HDF5 datasets beneath \fIpath\fR, \fBnPeaks\fR, \fBpeakXPosRaw\fR, \fBpeakYPosRaw\fR and \fBpeakTotalIntensity\fR.  See the specification for the CXI file format at http://www.cxidb.org/ for more details.
 
 If you use \fB--peaks=zaef\fR, indexamajig will use a simple gradient search after Zaefferer (2000).  You can control the overall threshold and minimum squared gradient for finding a peak using \fB--threshold\fR and \fB--min-gradient\fR.  The threshold has arbitrary units matching the pixel values in the data, and the minimum gradient has the equivalent squared units.  Peaks will be rejected if the 'foot point' is further away from the 'summit' of the peak by more than the inner integration radius (see below).  They will also be rejected if the peak is closer than twice the inner integration radius from another peak.
 
@@ -230,7 +232,7 @@ Prefix the filenames from the input file with \fIprefix\fR.  If \fB--basename\fR
 .PD 0
 .IP \fB--hdf5-peaks=\fR\fIpath\fR
 .PD
-When using \fB--peaks=hdf5\fR, read the peak locations from a table in the HDF5 file located at \fIpath\fR.
+When using \fB--peaks=hdf5\fR or \fB--peaks=cxi\fR, read the peak positions from location \fIpath\fR.  See \fBPEAK DETECTION\fR above.
 
 .PD 0
 .IP \fB--tolerance=\fR\fItol\fR
@@ -293,7 +295,7 @@ Normally, peaks which contain one or more pixels above max_adu (defined in the d
 .PD 0
 .IP \fB--no-revalidate\fR
 .PD
-When using \fB--peaks=hdf5\fR, the peaks will be put through some of the same checks as if you were using \fB--peaks=zaef\fR.  These checks reject peaks which are too close to panel edges, are saturated (unless you use \fB--use-saturated\fR), have other nearby peaks (closer than twice the inner integration radius, see \fB--int-radius\fR), or have any part in a bad region.  Using this option skips this validation step, and uses the peaks directly.
+When using \fB--peaks=hdf5\fR or \fB--peaks=cxi\fR, the peaks will be put through some of the same checks as if you were using \fB--peaks=zaef\fR.  These checks reject peaks which are too close to panel edges, are saturated (unless you use \fB--use-saturated\fR), have other nearby peaks (closer than twice the inner integration radius, see \fB--int-radius\fR), or have any part in a bad region.  Using this option skips this validation step, and uses the peaks directly.
 
 .PD 0
 .IP \fB--check-hdf5-snr\fR
@@ -339,10 +341,7 @@ You do not have to use all three of these options together.  For example, if the
 
 By default indexamajig processes all diffraction patterns ("events") in each of the data files listed in the input list. It is however, possible, to only process single events in a multi-event file, by adding in the list an event description string after the data filename. The event description always includes a first section with alphanumeric strings separated by forward slashes ("/") and a second section with integer numbers also separated by forward slashes. The two sections are in turn separated by a double forward slash ('//'). Any of the two sections can be empty, but the double forward slash separator must always be present.  Indexamajig matches the strings and the numbers in the event description with the event placeholders ('%') present respectively in the 'data' and 'dim' properties defined in the geometry file, and tries to retrieve the full HDF path to the event data and the the its location in a multi-dimensional data space. Consider the following examples:
 
-Example 1:
-.br
-
-Assuming that the 'data' and 'dim' properties have been defined like this in the geometry file:
+\fBExample 1:\fR The 'data' and 'dim' properties have been defined like this in the geometry file:
 
 .br
 data = /data/%/rawdata
@@ -351,18 +350,15 @@ dim0 = ss
 .br
 dim1 = fs
 
-The following line:
+The event list contains the following line:
 .br
 
 filename.h5  event1//
 .br
 
-Identifies an event in the 2-dimensional data block lying at the /data/event1/rawdata HDF path in the filename.h5 file
+This identifies an event in the 2-dimensional data block located at /data/event1/rawdata in the HDF5 file called filename.h5.
 
-Example 2:
-.br
-
-Assuming that the 'data' and 'dim' properties have been defined like this in the geometry file:
+\fBExample 2:\fR The 'data' and 'dim' properties have been defined like this in the geometry file:
 
 .br
 data = /data/rawdata
@@ -373,18 +369,21 @@ dim1 = ss
 .br
 dim2 = fs
 
-The following line:
+The event list contains the following line:
 .br
 
 filename.h5  //3
 .br
 
-Identifies an event in the 3-dimensional data block lying at the /data/rawdata HDF path in the filename.h5 file, specifically the 2-dimensional data slice defined by the value 3 of the first axis of the data space.
+This identifies an event in the 3-dimensional data block located at /data/rawdata in the HDF5 file called filename.h5, specifically the 2-dimensional data slice defined by the value 3 of the first axis of the data space.
 
 Indexamajig tries to match the alphanumerical strings to the placeholders in the 'dim' property defined in the geometry file. The first string is matched to the first placeholder, the second to
 the second placeholder, and so on. A similar strategy is followed to match integer numbers to the placeholders in the 'dim' property defined in the geometry file.
 For a full explanation of how the internal layout of the data file can be  described in the geometry file, please see \fBman crystfel_geometry\fR.
 
+You can use \fBlist_events\fR to prepare a list of each event in one or more input files.  Note that you only need to do this if you need to perform some sorting or filtering on this list.  If you want to process every event in a file, simply specify the filename in the input file.
+
+
 .SH BUGS
 ReAx indexing is experimental.  It works very nicely for some people, and crashes for others.  In a future version, it will be improved and fully supported.
 
@@ -410,4 +409,6 @@ You should have received a copy of the GNU General Public License along with Cry
 .BR crystfel_geometry (5),
 .BR cell_explorer (1),
 .BR process_hkl (1),
-.BR partialator (1)
+.BR partialator (1),
+.BR list_events (1),
+.BR whirligig (1)
diff --git a/doc/man/partial_sim.1 b/doc/man/partial_sim.1
index 67b05edc..dbf2a5ba 100644
--- a/doc/man/partial_sim.1
+++ b/doc/man/partial_sim.1
@@ -15,7 +15,6 @@ partial_sim \- calculate partial reflections
 .BR partial_sim
 \fB-o\fR \fIsimulated.stream\fR
 \fB-g\fR \fIgeometry.geom\fR
-\fB-b\fR \fIxrays.beam\fR
 \fB-p\fR \fIunitcell.pdb\fR
 [\fIoptions\fR] \fB...\fR
 
diff --git a/doc/man/partialator.1 b/doc/man/partialator.1
index 75fc34f2..63fb062c 100644
--- a/doc/man/partialator.1
+++ b/doc/man/partialator.1
@@ -93,20 +93,29 @@ Include a reflection in the output only if it appears at least least \fIn\fR tim
 
 The available partiality models are:
 
-.IP \fBsphere\fR
+.IP \fBscsphere\fR
 .PD
 The volume of intersection between a sphere centered on each reciprocal lattice
-point, and the part of reciprocal space excited by the Ewald sphere taking into
-account the finite bandwidth and convergence angle.  A Lorentz factor will also
-be used, proportional to the distance between the limiting Ewald spheres.
+point and the part of reciprocal space excited by the Ewald sphere taking into
+account the finite bandwidth and convergence angle.  A "source coverage factor"
+will be included to take into account the spectral brightness of the effective
+source for the reflection.
 
-For a full description including diagrams, see T. A. White et al., Acta Cryst.
-D69 (2013) p1231-1240.
+This model is similar to that described in Acta Cryst. D69 (2013) p1231-1240,
+and in Phil. Trans. Roy. Soc. B 369 (2014) 20130330, except that the "Lorentz
+factor" described there is no longer treated as a separate factor.
 
-.IP \fBunity\fR
+
+.IP \fBscgaussian\fR
 .PD
-Fix all partialities at 1, and use no Lorentz factor at all.
+As \fBscsphere\fR, except that the shape of the scattering density centered on
+each reciprocal lattice point is taken to be a 3D Gaussian distribution instead
+of a sphere.  The standard deviation of the distribution will be the profile
+radius (determined by indexamajig) divided by 2.6.
 
+.IP \fBunity\fR
+.PD
+Fix all partialities at 1.
 
 
 .SH BUGS