This is CrystFEL 0.6.20.6.2

9 files changed, 233 insertions, 212 deletions

diff --git a/ChangeLog b/ChangeLog
index 49f5e830..86788368 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,34 @@
+CrystFEL version 0.6.2, 21st March 2016
+---------------------------------------
+
+- partialator: Stability improvements to scaling algorithm, default --push-res
+  value changed from zero to infinity (no resolution cutoff by default).
+- partialator: Added --custom-split, --max-rel-B and --output-every-cycle
+- Added "retry" indexing method to improve indexing rate.
+- Per-pixel saturation values can now be used (needed for CSPAD dual-gain mode).
+- Fixed a regression with indexing using MOSFLM introduced near 0.6.1.
+- indexamajig: Fixed a bug which occasionally caused corrupted stream output
+  ("Failed while reading reflections").
+- indexamajig: gradient fitting of background now optional and OFF by default.
+- indexamajig: Added --profile for speed monitoring.
+- indexamajig: Added "felix" indexing method.
+- scripts/ave-resolution: Show resolution histogram.
+- scripts/turbo-index: Added a missing line
+- hdfsee: Improve quality of display by rounding pixel coordinates.
+- hdfsee: Can now safely save new geometry file over old one.
+- Target unit cell is now written to the stream.
+- pattern_sim: Fixed bugs in --gpu and --powder.
+- compare_hkl: Avoid double counting of reflections when calculating anomalous
+  figures of merit.
+- geoptimiser: Major refactoring
+- ambigator: fixes for a few problem when using very large numbers of correlation
+  coefficients or patterns.
+- Some crashes and hangs in asdf were fixed.
+- The "random" partiality model was improved.
+- Prototype for prepare_indexing() changed, otherwise only additions were made
+  to the API.
+
+
 CrystFEL version 0.6.1, 3rd August 2015
 ---------------------------------------
 
diff --git a/Makefile.am b/Makefile.am
index b1a477c4..5a916cad 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -29,7 +29,7 @@ TESTS = tests/list_check $(MERGE_CHECKS) $(PARTIAL_CHECKS) \
         tests/prediction_gradient_check
 
 EXTRA_DIST += $(MERGE_CHECKS) $(PARTIAL_CHECKS)
-EXTRA_DIST += relnotes-0.6.1 announcement-0.6.1
+EXTRA_DIST += relnotes-0.6.2 announcement-0.6.2
 
 if BUILD_HDFSEE
 bin_PROGRAMS += src/hdfsee
diff --git a/announcement-0.6.1 b/announcement-0.6.1
deleted file mode 100644
index 85137402..00000000
--- a/announcement-0.6.1
+++ /dev/null
@@ -1,43 +0,0 @@
-Dear CrystFEL users and interested crystallographers,
-
-CrystFEL version 0.6.1 has been released.  The most interesting changes
-in this version are the new "prediction refinement" stage in
-"indexamajig" and improvements to the scaling and post-refinement
-program "partialator".  There are also a couple of new indexing methods.
-
-The prediction refinement stage improves the quality of spot prediction
-and hence the initial parameter estimates of the spot partialities.  It
-is also able to update the central beam position.
-
-The improvements to partialator include a significantly more stable
-scaling procedure.  Partialator is no longer considered experimental as
-of this version, so please try it and send feedback.  With the
-combination of this and the new prediction refinement, many users are
-already finding significant improvements in data quality.  See the
-release notes (link below) and the updated tutorial on the CrystFEL
-website for more details.
-
-The new indexing methods are "asdf", which is an algorithm built into
-CrystFEL which therefore requires no external programs, and
-"mosflm-cell" which uses the prior unit cell algorithm in recent (7.2.0
-or later) versions of mosflm.  "mosflm-cell" is now the default
-behaviour for mosflm - if it doesn't work for you, explicitly specify
-"mosflm-nocell" to restore the old behaviour.
-
-In addition, there are many smaller improvements, new features and bug
-fixes.
-
-Thanks for all of your past and future feedback and contributions, and
-your continued use of CrystFEL.  As ever, please contact me directly if
-you would prefer to be unsubscribed from this mailing list.
-
-CrystFEL website:
-http://www.desy.de/~twhite/crystfel
-
-Release notes for version 0.6.1:
-http://www.desy.de/~twhite/crystfel/relnotes-0.6.1
-
-CrystFEL tutorial:
-http://www.desy.de/~twhite/crystfel/tutorial
-
-Tom
diff --git a/announcement-0.6.2 b/announcement-0.6.2
new file mode 100644
index 00000000..e92ce453
--- /dev/null
+++ b/announcement-0.6.2
@@ -0,0 +1,44 @@
+Dear CrystFEL users and interested crystallographers,
+
+CrystFEL version 0.6.2 has been released.  The most interesting changes
+in this version are improvements to the stability of scaling in partialator,
+the addition of "custom dataset splitting" for time-resolved/isomorphous
+replacement experiments and more, "retry" indexing to increase the indexing
+rate, and support for per-pixel saturation values needed to correctly handle
+dual-gain CSPAD data.
+
+The improvements to scaling stability should improve the results with most
+data sets, but particularly those where there have been problems with large
+numbers of crystals getting rejected due to excessively large relative B
+factors.
+
+"Custom dataset splitting" enables you to process multiple datasets in one
+large scaling (or post-refinement) job, and separate them afterwards.  This
+eliminates any possible concerns over uniqueness of the solution with these new
+data processing methods.
+
+"Retry" indexing is where the indexing method is repeated, if it fails the first
+time, after deleting some of the weakest peaks.  In many cases, this leads to a
+successfully indexed pattern.  This can sometimes double the indexing rate!
+However, it makes it slower to process if the indexing rate is low, and you can
+disable it by adding "-noretry" to all the indexing methods.
+
+See the release notes for more discussion of these improvements.
+
+In addition, there are many smaller improvements, new features and bug
+fixes.  See the ChangeLog for details.
+
+Thanks for all of your past and future feedback and contributions, and
+your continued use of CrystFEL.  As ever, please contact me directly if
+you would prefer to be unsubscribed from this mailing list.
+
+CrystFEL website:
+http://www.desy.de/~twhite/crystfel
+
+Release notes for version 0.6.2:
+http://www.desy.de/~twhite/crystfel/relnotes-0.6.2
+
+CrystFEL tutorial:
+http://www.desy.de/~twhite/crystfel/tutorial
+
+Tom
diff --git a/configure.ac b/configure.ac
index 9d4d7edf..8044a145 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,6 +1,6 @@
 dnl Process this file with autoconf to produce a configure script.
 
-AC_INIT([crystfel],[0.6.1],[taw@physics.org])
+AC_INIT([crystfel],[0.6.2],[taw@physics.org])
 AC_CONFIG_AUX_DIR([build-aux])
 AC_CONFIG_HEADERS([config.h])
 AM_INIT_AUTOMAKE([subdir-objects])
diff --git a/doc/reference/libcrystfel/CrystFEL-docs.sgml b/doc/reference/libcrystfel/CrystFEL-docs.sgml
index 9ef5d945..407f23b4 100644
--- a/doc/reference/libcrystfel/CrystFEL-docs.sgml
+++ b/doc/reference/libcrystfel/CrystFEL-docs.sgml
@@ -8,7 +8,7 @@
   <bookinfo>
     <title>CrystFEL Reference Manual</title>
     <releaseinfo>
-      For libcrystfel from CrystFEL 0.6.1.
+      For libcrystfel from CrystFEL 0.6.2.
     </releaseinfo>
     <abstract>
     This is the internal documentation for CrystFEL.  Unless you are looking at
diff --git a/libcrystfel/Makefile.am b/libcrystfel/Makefile.am
index f2760dac..d7142f6a 100644
--- a/libcrystfel/Makefile.am
+++ b/libcrystfel/Makefile.am
@@ -1,6 +1,6 @@
 lib_LTLIBRARIES = libcrystfel.la
 libcrystfel_la_LIBADD = ../lib/libgnu.la @LIBCRYSTFEL_LIBS@ $(PTY_LIB)
-libcrystfel_la_LDFLAGS = -version-info 7:0:0
+libcrystfel_la_LDFLAGS = -version-info 8:0:0
 
 libcrystfel_la_SOURCES = src/reflist.c src/utils.c src/cell.c src/detector.c \
                          src/thread-pool.c src/image.c src/hdf5-file.c \
diff --git a/relnotes-0.6.1 b/relnotes-0.6.1
deleted file mode 100644
index 4c189cd6..00000000
--- a/relnotes-0.6.1
+++ /dev/null
@@ -1,165 +0,0 @@
-CrystFEL - Crystallography with a FEL
--------------------------------------
-
-Release notes for version 0.6.1
-
-Copyright © 2012-2015 Deutsches Elektronen-Synchrotron DESY,
-                      a research centre of the Helmholtz Association.
-
-Authors:
-  Thomas White <taw@physics.org>
-  Richard Kirian <rkirian@asu.edu>
-  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
-  Andrew Aquila <andrew.aquila@cfel.de>
-  Andrew Martin <andrew.martin@desy.de>
-  Lorenzo Galli <lorenzo.galli@desy.de>
-  Chun Hong Yoon <chun.hong.yoon@desy.de>
-  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
-  Karol Nass <karol.nass@desy.de>
-  Nadia Zatsepin <nadia.zatsepin@asu.edu>
-  Anton Barty <anton.barty@desy.de>
-  Cornelius Gati <cornelius.gati@desy.de>
-  Fedor Chervinskii <fedor.chervinskii@gmail.com>
-  Alexandra Tolstikova <alexandra.tolstikova@desy.de>
-  Wolfgang Brehm <wolfgang.brehm@gmail.com>
-  Valerio Mariani <valerio.mariani@desy.de>
-  Parker de Waal <Parker.deWaal@vai.org>
-  Takanori Nakane <nakane.t@gmail.com>
-  Keitaro Yamashita <k.yamashita@spring8.or.jp>
-  Oleksandr Yefanov <oleksandr.yefanov@cfel.de>
-
-CrystFEL is free software: you can redistribute it and/or modify it under the
-terms of the GNU General Public License as published by the Free Software
-Foundation, either version 3 of the License, or (at your option) any later
-version.
-
-CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
-WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
-PARTICULAR PURPOSE.  See the GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along with
-CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
-
-
-Overview
---------
-
-The most important new features in this version of CrystFEL are:
-
-- Introduction of "prediction refinement".
-
-- Scaling and post-refinement with "partialator".
-
-- New indexing methods: mosflm-cell and asdf.
-
-- process_hkl --even-only and --odd-only instead of "alternate-stream".
-
-These new features have individual sections below.  In addition, there are many
-other new developments.  See the ChangeLog or the changes page on the website
-for more details.  There were, of course, the usual large number of smaller
-refinements and bug fixes.
-
-
-Introduction of "prediction refinement"
----------------------------------------
-
-A new refinement step has been introduced between the indexing and prediction/
-integration stages of indexamajig.  The refinement aims to maximise the
-agreement between the predicted spot positions and the peak positions found by
-the peak search step, at the same time as putting the peaks as close to the
-Bragg condition as possible.  This increases the accuracy of spot prediction,
-which has positive effects on the downstream processing stages.
-
-The prediction refinement stage allows the central beam position to move
-relative to the entire detector.  The required offset for each crystal is
-recorded in the stream and can be visualised using scripts/detector-shift.
-
-If, somehow, the prediction refinement causes problems for you, you can disable
-it using the indexamajig option "--no-refine".
-
-
-Scaling and post-refinement with "partialator"
-----------------------------------------------
-
-The scaling and post-refinement program partialator is no longer considered
-experimental in this version.  Many users are already finding significant
-improvements in their data simply by applying scaling without partiality:
-
-$ partialator -i my.stream -o my.hkl -y mypg --model=unity --iterations=3
-
-A further improvement is sometimes obtained by applying partialities and doing
-full post-refinement:
-
-$ partialator -i my.stream -o my.hkl -y mypg --model=scsphere --iterations=3
-
-Note that partialator applies a per-crystal resolution cutoff to stabilise the
-scaling calculation.  Therefore, you should experiment with different values
-for "--push-res".
-
-Further improvements to partiality modelling, scaling and merging will be
-included in the near future.
-
-
-New indexing methods
---------------------
-
-This version of CrystFEL can make use of the known unit cell algorithm included
-with very recent (7.2.0 or later) versions of mosflm.  The known unit cell
-algorithm is used by default if you provide lattice parameters to indexamajig.
-If this isn't what you want, or if it doesn't work well for you, use
-"mosflm-nocell" as your indexing method to restore the old behaviour.
-
-A new indexing method, asdf, has been added.  This algorithm is built into
-CrystFEL directly, so no temporary files or external programs are needed.
-
-The old indexing methods "grainspotter" and "reax" have been removed.
-
-
-process_hkl --even-only and --odd-only
---------------------------------------
-
-When using process_hkl for merging, you no longer need to use "alternate-stream"
-to split the stream into two parts in order to calculate figures of merit such
-as Rsplit.  Instead, simply use the "--even-only" and "--odd-only" options to
-merge only even-numbered and odd-numbered crystals from the stream:
-
-$ process_hkl -i my.stream -o my.hkl1 -y mypg --even-only
-$ process_hkl -i my.stream -o my.hkl2 -y mypg --odd-only
-$ compare_hkl my.hkl1 my.hkl2 -y mypg --fom=rsplit [..etc..]
-
-Note that the numbering of crystals for this purpose is simply according to the
-order they appear in the stream.  For most purposes, this is effectively random.
-
-Note further that partialator writes the ".hkl1" and ".hkl2" split-merged
-datasets for you automatically.
-
-
-API changes
------------
-
-The following changes have been made to the libcrystfel API:
-
-New functions:
-	- asdf_prepare(), run_asdf(), asdf_cleanup()
-	- cell_get_volume()
-	- crystal_{get,set}_Bfac()
-	- crystal_{get,set,add}_notes()
-	- detector_has_clen_references()
-	- {x,y,r}_gradient()
-	- refine_prediction()
-	- refine_radius()
-	- {get,set}_panel()
-	- {get,set}_flag()
-	- solve_svd()
-
-Removed functions:
-	- select_intersections()
-	- update_partialities_2()
-	- get_excitation_error()
-	- extract_f_from_stuff()
-	- grainspotter_{prepare,index,cleanup}()
-	- reax_{prepare,index,cleanup}()
-
-Changed function prototypes:
-	- integrate_all_4()
-	- set_detector_pos()
diff --git a/relnotes-0.6.2 b/relnotes-0.6.2
new file mode 100644
index 00000000..daf48f3f
--- /dev/null
+++ b/relnotes-0.6.2
@@ -0,0 +1,154 @@
+CrystFEL - Crystallography with a FEL
+-------------------------------------
+
+Release notes for version 0.6.2
+
+Copyright © 2012-2016 Deutsches Elektronen-Synchrotron DESY,
+                      a research centre of the Helmholtz Association.
+
+Authors:
+  Thomas White <taw@physics.org>
+  Richard Kirian <rkirian@asu.edu>
+  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
+  Andrew Aquila <andrew.aquila@cfel.de>
+  Andrew Martin <andrew.martin@desy.de>
+  Lorenzo Galli <lorenzo.galli@desy.de>
+  Chun Hong Yoon <chun.hong.yoon@desy.de>
+  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
+  Karol Nass <karol.nass@desy.de>
+  Nadia Zatsepin <nadia.zatsepin@asu.edu>
+  Anton Barty <anton.barty@desy.de>
+  Cornelius Gati <cornelius.gati@desy.de>
+  Fedor Chervinskii <fedor.chervinskii@gmail.com>
+  Alexandra Tolstikova <alexandra.tolstikova@desy.de>
+  Wolfgang Brehm <wolfgang.brehm@gmail.com>
+  Valerio Mariani <valerio.mariani@desy.de>
+  Parker de Waal <Parker.deWaal@vai.org>
+  Takanori Nakane <nakane.t@gmail.com>
+  Keitaro Yamashita <k.yamashita@spring8.or.jp>
+  Oleksandr Yefanov <oleksandr.yefanov@cfel.de>
+
+CrystFEL is free software: you can redistribute it and/or modify it under the
+terms of the GNU General Public License as published by the Free Software
+Foundation, either version 3 of the License, or (at your option) any later
+version.
+
+CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
+PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with
+CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
+
+
+Overview
+--------
+
+The most important new features in this version of CrystFEL are:
+
+- Stability improvements to the scaling algorithm in partialator.
+
+- Addition of "custom dataset splitting" in partialator.
+
+- "retry" indexing.
+
+- Per-pixel saturation values
+
+These changes have sections below.  In addition, there were many bug fixes and
+smaller improvements.
+
+
+Stability improvements in partialator
+-------------------------------------
+
+The scaling algorithm in partialator has been altered such that only the strong
+reflections contribute to the scaling factors at all.  This reduces the
+"drifting" of scale factors which happened before, and helps avoid the situation
+where a very large number of crystals get rejected with large B factors.
+Of course, all reflections contribute to the final merged data set as before.
+
+In CrystFEL 0.6.1, we needed to reduce the amount of noise going into the
+scaling algorithm to make it work.  This was done by including reflections only
+up to the apparent (conservative) resolution limit of each crystal made by
+indexamajig, and could be overridden by using the "--push-res" option of
+partialator.  The default --push-res value was zero, which meant to ignore any
+reflections higher than the limit.  With the stabilised algorithm, this is no
+longer necessary, and hence the default --push-res value has been changed to
+infinity, i.e. no cutoff at all.  You can restore the old behaviour with
+--push-res=0.
+
+
+Custom dataset splitting in partialator
+---------------------------------------
+
+When performing a time-resolved or isomorphous replacement experiment (plus some
+other types of experiment), it is preferable to ensure that the data for all
+time points has been processed identically.  Rather than processing each time
+point independently with separate runs of partialator, it is better to process
+them all together and do the splitting into time points just before the final
+output.  Consider, for example, the case of simple scaling (without a B
+factor): when merging independently, the resulting datasets would probably end
+up with different overall scaling factors.  When comparing the results, you
+would need to take this difference into account.  In practice, most programs
+can do that job easily, but what about if a B factor is included?  And what if
+partialities are included - how unique is the solution?
+
+Partialator's new "custom dataset splitting" option allows you to provide a
+text file containing a list of filenames, event numbers and dataset
+names, one event (detector frame) per line.  All crystals will be refined
+together, but they will be merged according to the dataset names you give.  The
+parameters (scaling factors, partialities etc) determined during the joint
+refinement will be applied.  For each dataset, a separate pair of split
+half-datasets will also be written, allowing you to calculate figures of merit
+such as Rsplit and CC1/2 for each one.
+
+
+"retry" indexing
+----------------
+
+If an indexing method fails to index a pattern (or if the solution doesn't get
+through the prediction refinement stage), this version of CrystFEL will delete
+the weakest few peaks and try again.  This can increase the indexing yield a lot
+(sometimes a factor of two!), but decreases the speed.  If you'd rather keep
+things fast, add "-noretry" to each of your indexing methods.
+
+
+Per-pixel saturation values
+---------------------------
+
+When using dual-gain mode on the CSPAD (or any other detector which offers a
+similar feature), our current advice is to correct the pixel values in one of
+the regions by multiplying them by the relative gain factor.  This will give
+image data which looks good, but the saturation values will be different in the
+two regions.  To handle this, CrystFEL 0.6.2 allows you to prepare a map, in
+HDF5 format, giving the saturation value for each pixel individually.  The
+script "gaincal-to-saturation-map" can be used to create a suitable file from
+the gain map (e.g. as generated and used by Cheetah).  Specify the filename and
+HDF5 path of the saturation map using saturation_map and saturation_map_file
+in the geometry file - these work exactly like the bad pixel map.  Reflections
+containing any pixel above this value will be rejected, in addition to any
+reflections which get rejected by the usual saturation criteria (so remove any
+max_adu directives if you want the saturation map to be the only decider!).
+
+
+API changes
+-----------
+
+The following changes have been made to the libcrystfel API:
+
+New functions:
+	- write_cell()
+	- find_orig_panel_number()
+	- felix_{prepare,index,cleanup}()
+	- sort_peaks()
+	- remove_flagged_crystals()
+	- open_stream_for_write_3()
+
+Removed functions:
+	(none)
+
+Changed function prototypes:
+	- prepare_indexing()
+
+"data", "flags" and "twotheta" were removed from "struct image", and "sat" was
+added.