This is CrystFEL 0.6.30.6.3

9 files changed, 252 insertions, 204 deletions

diff --git a/ChangeLog b/ChangeLog
index 0c8d596e..ba504e5d 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,5 @@
-Changes in this development version
------------------------------------
+CrystFEL version 0.6.3, 24th July 2017
+--------------------------------------
 
 - indexamajig --peaks=peakfinder8 was added (Valerio Mariani, Oleksandr Yefanov)
 - CBF files are now supported as input.
diff --git a/Makefile.am b/Makefile.am
index fda28fbc..78456727 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -29,7 +29,7 @@ TESTS = tests/list_check $(MERGE_CHECKS) $(PARTIAL_CHECKS) \
         tests/prediction_gradient_check
 
 EXTRA_DIST += $(MERGE_CHECKS) $(PARTIAL_CHECKS)
-EXTRA_DIST += relnotes-0.6.2 announcement-0.6.2
+EXTRA_DIST += relnotes-0.6.3 announcement-0.6.3
 
 if BUILD_HDFSEE
 bin_PROGRAMS += src/hdfsee
diff --git a/announcement-0.6.2 b/announcement-0.6.2
deleted file mode 100644
index e92ce453..00000000
--- a/announcement-0.6.2
+++ /dev/null
@@ -1,44 +0,0 @@
-Dear CrystFEL users and interested crystallographers,
-
-CrystFEL version 0.6.2 has been released.  The most interesting changes
-in this version are improvements to the stability of scaling in partialator,
-the addition of "custom dataset splitting" for time-resolved/isomorphous
-replacement experiments and more, "retry" indexing to increase the indexing
-rate, and support for per-pixel saturation values needed to correctly handle
-dual-gain CSPAD data.
-
-The improvements to scaling stability should improve the results with most
-data sets, but particularly those where there have been problems with large
-numbers of crystals getting rejected due to excessively large relative B
-factors.
-
-"Custom dataset splitting" enables you to process multiple datasets in one
-large scaling (or post-refinement) job, and separate them afterwards.  This
-eliminates any possible concerns over uniqueness of the solution with these new
-data processing methods.
-
-"Retry" indexing is where the indexing method is repeated, if it fails the first
-time, after deleting some of the weakest peaks.  In many cases, this leads to a
-successfully indexed pattern.  This can sometimes double the indexing rate!
-However, it makes it slower to process if the indexing rate is low, and you can
-disable it by adding "-noretry" to all the indexing methods.
-
-See the release notes for more discussion of these improvements.
-
-In addition, there are many smaller improvements, new features and bug
-fixes.  See the ChangeLog for details.
-
-Thanks for all of your past and future feedback and contributions, and
-your continued use of CrystFEL.  As ever, please contact me directly if
-you would prefer to be unsubscribed from this mailing list.
-
-CrystFEL website:
-http://www.desy.de/~twhite/crystfel
-
-Release notes for version 0.6.2:
-http://www.desy.de/~twhite/crystfel/relnotes-0.6.2
-
-CrystFEL tutorial:
-http://www.desy.de/~twhite/crystfel/tutorial
-
-Tom
diff --git a/announcement-0.6.3 b/announcement-0.6.3
new file mode 100644
index 00000000..db8d0adb
--- /dev/null
+++ b/announcement-0.6.3
@@ -0,0 +1,35 @@
+Dear CrystFEL users and interested crystallographers,
+
+It's been a long time since the last CrystFEL release, but we have not been
+resting.  CrystFEL version 0.6.3 has been released.
+
+This version incorporates a large number of "backstage" improvements and
+alterations.  The most interesting changes for users are:
+
+- The incorporation of "peak finder 8" from Cheetah (also used in OnDA)
+
+- Support for images in CBF format
+
+- Incorporation of "TakeTwo" indexing algorithm
+
+- Hit finder functionality (skip frames with few peaks)
+
+- Better support for three-dimensional detector geometries
+
+See the release notes for more discussion of these improvements, and the
+ChangeLog for full details of the other improvements.
+
+Thanks for all of your past and future feedback and contributions, and
+your continued use of CrystFEL.  As ever, please contact me directly if
+you would prefer to be unsubscribed from this mailing list.
+
+CrystFEL website:
+https://www.desy.de/~twhite/crystfel
+
+Release notes for version 0.6.3:
+https://www.desy.de/~twhite/crystfel/relnotes-0.6.3
+
+CrystFEL tutorial:
+https://www.desy.de/~twhite/crystfel/tutorial
+
+Tom
diff --git a/configure.ac b/configure.ac
index 2a5f40cf..41071139 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,6 +1,6 @@
 dnl Process this file with autoconf to produce a configure script.
 
-AC_INIT([crystfel],[0.6.2],[taw@physics.org])
+AC_INIT([crystfel],[0.6.3],[taw@physics.org])
 AC_CONFIG_AUX_DIR([build-aux])
 AC_CONFIG_HEADERS([config.h])
 AM_INIT_AUTOMAKE([subdir-objects])
diff --git a/doc/reference/libcrystfel/CrystFEL-docs.sgml b/doc/reference/libcrystfel/CrystFEL-docs.sgml
index 407f23b4..048f4167 100644
--- a/doc/reference/libcrystfel/CrystFEL-docs.sgml
+++ b/doc/reference/libcrystfel/CrystFEL-docs.sgml
@@ -8,7 +8,7 @@
   <bookinfo>
     <title>CrystFEL Reference Manual</title>
     <releaseinfo>
-      For libcrystfel from CrystFEL 0.6.2.
+      For libcrystfel from CrystFEL 0.6.3.
     </releaseinfo>
     <abstract>
     This is the internal documentation for CrystFEL.  Unless you are looking at
diff --git a/libcrystfel/Makefile.am b/libcrystfel/Makefile.am
index fef2e11f..24cb7358 100644
--- a/libcrystfel/Makefile.am
+++ b/libcrystfel/Makefile.am
@@ -1,6 +1,6 @@
 lib_LTLIBRARIES = libcrystfel.la
 libcrystfel_la_LIBADD = ../lib/libgnu.la @LIBCRYSTFEL_LIBS@ $(PTY_LIB)
-libcrystfel_la_LDFLAGS = -version-info 8:0:0
+libcrystfel_la_LDFLAGS = -version-info 9:0:0
 
 libcrystfel_la_SOURCES = src/reflist.c src/utils.c src/cell.c src/detector.c \
                          src/thread-pool.c src/image.c src/hdf5-file.c \
diff --git a/relnotes-0.6.2 b/relnotes-0.6.2
deleted file mode 100644
index daf48f3f..00000000
--- a/relnotes-0.6.2
+++ /dev/null
@@ -1,154 +0,0 @@
-CrystFEL - Crystallography with a FEL
--------------------------------------
-
-Release notes for version 0.6.2
-
-Copyright © 2012-2016 Deutsches Elektronen-Synchrotron DESY,
-                      a research centre of the Helmholtz Association.
-
-Authors:
-  Thomas White <taw@physics.org>
-  Richard Kirian <rkirian@asu.edu>
-  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
-  Andrew Aquila <andrew.aquila@cfel.de>
-  Andrew Martin <andrew.martin@desy.de>
-  Lorenzo Galli <lorenzo.galli@desy.de>
-  Chun Hong Yoon <chun.hong.yoon@desy.de>
-  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
-  Karol Nass <karol.nass@desy.de>
-  Nadia Zatsepin <nadia.zatsepin@asu.edu>
-  Anton Barty <anton.barty@desy.de>
-  Cornelius Gati <cornelius.gati@desy.de>
-  Fedor Chervinskii <fedor.chervinskii@gmail.com>
-  Alexandra Tolstikova <alexandra.tolstikova@desy.de>
-  Wolfgang Brehm <wolfgang.brehm@gmail.com>
-  Valerio Mariani <valerio.mariani@desy.de>
-  Parker de Waal <Parker.deWaal@vai.org>
-  Takanori Nakane <nakane.t@gmail.com>
-  Keitaro Yamashita <k.yamashita@spring8.or.jp>
-  Oleksandr Yefanov <oleksandr.yefanov@cfel.de>
-
-CrystFEL is free software: you can redistribute it and/or modify it under the
-terms of the GNU General Public License as published by the Free Software
-Foundation, either version 3 of the License, or (at your option) any later
-version.
-
-CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
-WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
-PARTICULAR PURPOSE.  See the GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along with
-CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
-
-
-Overview
---------
-
-The most important new features in this version of CrystFEL are:
-
-- Stability improvements to the scaling algorithm in partialator.
-
-- Addition of "custom dataset splitting" in partialator.
-
-- "retry" indexing.
-
-- Per-pixel saturation values
-
-These changes have sections below.  In addition, there were many bug fixes and
-smaller improvements.
-
-
-Stability improvements in partialator
--------------------------------------
-
-The scaling algorithm in partialator has been altered such that only the strong
-reflections contribute to the scaling factors at all.  This reduces the
-"drifting" of scale factors which happened before, and helps avoid the situation
-where a very large number of crystals get rejected with large B factors.
-Of course, all reflections contribute to the final merged data set as before.
-
-In CrystFEL 0.6.1, we needed to reduce the amount of noise going into the
-scaling algorithm to make it work.  This was done by including reflections only
-up to the apparent (conservative) resolution limit of each crystal made by
-indexamajig, and could be overridden by using the "--push-res" option of
-partialator.  The default --push-res value was zero, which meant to ignore any
-reflections higher than the limit.  With the stabilised algorithm, this is no
-longer necessary, and hence the default --push-res value has been changed to
-infinity, i.e. no cutoff at all.  You can restore the old behaviour with
---push-res=0.
-
-
-Custom dataset splitting in partialator
----------------------------------------
-
-When performing a time-resolved or isomorphous replacement experiment (plus some
-other types of experiment), it is preferable to ensure that the data for all
-time points has been processed identically.  Rather than processing each time
-point independently with separate runs of partialator, it is better to process
-them all together and do the splitting into time points just before the final
-output.  Consider, for example, the case of simple scaling (without a B
-factor): when merging independently, the resulting datasets would probably end
-up with different overall scaling factors.  When comparing the results, you
-would need to take this difference into account.  In practice, most programs
-can do that job easily, but what about if a B factor is included?  And what if
-partialities are included - how unique is the solution?
-
-Partialator's new "custom dataset splitting" option allows you to provide a
-text file containing a list of filenames, event numbers and dataset
-names, one event (detector frame) per line.  All crystals will be refined
-together, but they will be merged according to the dataset names you give.  The
-parameters (scaling factors, partialities etc) determined during the joint
-refinement will be applied.  For each dataset, a separate pair of split
-half-datasets will also be written, allowing you to calculate figures of merit
-such as Rsplit and CC1/2 for each one.
-
-
-"retry" indexing
-----------------
-
-If an indexing method fails to index a pattern (or if the solution doesn't get
-through the prediction refinement stage), this version of CrystFEL will delete
-the weakest few peaks and try again.  This can increase the indexing yield a lot
-(sometimes a factor of two!), but decreases the speed.  If you'd rather keep
-things fast, add "-noretry" to each of your indexing methods.
-
-
-Per-pixel saturation values
----------------------------
-
-When using dual-gain mode on the CSPAD (or any other detector which offers a
-similar feature), our current advice is to correct the pixel values in one of
-the regions by multiplying them by the relative gain factor.  This will give
-image data which looks good, but the saturation values will be different in the
-two regions.  To handle this, CrystFEL 0.6.2 allows you to prepare a map, in
-HDF5 format, giving the saturation value for each pixel individually.  The
-script "gaincal-to-saturation-map" can be used to create a suitable file from
-the gain map (e.g. as generated and used by Cheetah).  Specify the filename and
-HDF5 path of the saturation map using saturation_map and saturation_map_file
-in the geometry file - these work exactly like the bad pixel map.  Reflections
-containing any pixel above this value will be rejected, in addition to any
-reflections which get rejected by the usual saturation criteria (so remove any
-max_adu directives if you want the saturation map to be the only decider!).
-
-
-API changes
------------
-
-The following changes have been made to the libcrystfel API:
-
-New functions:
-	- write_cell()
-	- find_orig_panel_number()
-	- felix_{prepare,index,cleanup}()
-	- sort_peaks()
-	- remove_flagged_crystals()
-	- open_stream_for_write_3()
-
-Removed functions:
-	(none)
-
-Changed function prototypes:
-	- prepare_indexing()
-
-"data", "flags" and "twotheta" were removed from "struct image", and "sat" was
-added.
diff --git a/relnotes-0.6.3 b/relnotes-0.6.3
new file mode 100644
index 00000000..11a1f93b
--- /dev/null
+++ b/relnotes-0.6.3
@@ -0,0 +1,211 @@
+CrystFEL - Crystallography with a FEL
+-------------------------------------
+
+Release notes for version 0.6.3
+
+Copyright © 2012-2017 Deutsches Elektronen-Synchrotron DESY,
+                      a research centre of the Helmholtz Association.
+
+Authors:
+  Thomas White <taw@physics.org>
+  Richard Kirian <rkirian@asu.edu>
+  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
+  Andrew Aquila <andrew.aquila@cfel.de>
+  Andrew Martin <andrew.martin@desy.de>
+  Lorenzo Galli <lorenzo.galli@desy.de>
+  Chun Hong Yoon <chun.hong.yoon@desy.de>
+  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
+  Karol Nass <karol.nass@desy.de>
+  Nadia Zatsepin <nadia.zatsepin@asu.edu>
+  Anton Barty <anton.barty@desy.de>
+  Cornelius Gati <cornelius.gati@desy.de>
+  Fedor Chervinskii <fedor.chervinskii@gmail.com>
+  Alexandra Tolstikova <alexandra.tolstikova@desy.de>
+  Wolfgang Brehm <wolfgang.brehm@gmail.com>
+  Valerio Mariani <valerio.mariani@desy.de>
+  Parker de Waal <Parker.deWaal@vai.org>
+  Takanori Nakane <nakane.t@gmail.com>
+  Keitaro Yamashita <k.yamashita@spring8.or.jp>
+  Oleksandr Yefanov <oleksandr.yefanov@cfel.de>
+  Steve Aplin <steve.aplin@desy.de>
+  Helen Ginn <helen@strubi.ox.ac.uk>
+
+CrystFEL is free software: you can redistribute it and/or modify it under the
+terms of the GNU General Public License as published by the Free Software
+Foundation, either version 3 of the License, or (at your option) any later
+version.
+
+CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
+PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with
+CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
+
+
+Overview
+--------
+
+The most important new features in this version of CrystFEL are:
+
+- Support for CBF files as input
+
+- TakeTwo indexing algorithm
+
+- Improved peak detection and basic hitfinder functionality
+
+- Detector panels not perpendicular to X-ray beam, and "rail direction"
+
+These changes have sections below.  In addition, there were many bug fixes and
+other improvements.  See the "ChangeLog" or the changes page on the CrystFEL
+website for details.
+
+
+Support for CBF files as input
+------------------------------
+
+Crystallographic Binary Files (CBF) can now be used as input for indexamajig
+and hdfsee.  To enable this functionality, you must have "CBFlib" installed.
+Some distributions (eg Fedora) include CBFlib in their standard repositories -
+simply install "cbflib-devel" or similar.  Then compile and install CrystFEL as
+usual.  If CBFlib is installed somewhere non-standard, e.g. if you has to
+install it manually, you will need to give the path with the "--with-cbflib-dir"
+option to "./configure".
+
+After that, you can simply give the names of CBF files in the indexamajig input,
+and everything should work as usual!  There are some catches, for example you
+cannot yet use a bad pixel mask with CBF files, but these will be fixed in a
+future release.
+
+
+TakeTwo indexing algorithm
+--------------------------
+
+This version of CrystFEL includes the TakeTwo indexing algorithm by Helen Ginn.
+TakeTwo is a new indexing algorithm, designed specifically for diffraction
+snapshots and operating completely differently to conventional FFT-based methods.
+
+To use it, simply use "taketwo" as the indexing method, or add it to the list of
+indexing methods you have already:
+   $ indexamajig --indexing=taketwo
+   $ indexamajig --indexing=mosflm,taketwo,dirax
+
+If you use TakeTwo, please take careful note of the citation reminder which will
+be shown to you, and cite the following paper in any resulting publications:
+   Ginn et al., Acta Cryst. (2016). D72, 956-965
+
+TakeTwo is considered experimental in this version of CrystFEL.  We welcome any
+feedback, and will use the information to improve its performance for future
+versions.
+
+
+Improved peak detection and basic hitfinder functionality
+---------------------------------------------------------
+
+"Peak finder 8" from Cheetah, also used in OnDA, has been incorporated into
+CrystFEL.  It uses a radially varying background level to find peaks, which
+works very well with most diffraction data.
+
+Simply use "--peaks=peakfinder8", and set --threshold, --min-snr,
+--min-pix-count and --local-bg-radius.  See the manual for other options.
+
+Indexamajig can now skip over patterns which have a small number of peaks.  To
+use this, use the option "--min-peaks=N", where N is the minimum number of peaks
+before indexamajig will proceed to indexing (and hopefully integration) with the
+pattern.
+
+By combining the improved peak detection with the hitfinder functionality and
+CBF reading, you can process data from several different types of detector,
+without converting the files to HDF5 format or using any external program for
+hit finding!
+
+If the hit rate is very low, you might want to make CrystFEL stream smaller by
+excluding non-hits from it using "--no-non-hits-in-stream".  Without this
+option, standard information about the non-hits (e.g. photon energy and peaks
+found) will be recorded.
+
+
+Detector panels not perpendicular to X-ray beam, and "rail direction"
+----------------------------------------------------------------------
+
+CrystFEL now supports three-dimensional detector geometry.  The "fs" and "ss"
+vectors (which are the real-space directions of the fast scan and slow scan
+directions in the image data) can now contain a z component as well as x and y.
+
+For many detectors, the "rail" direction, along which the detector moves when
+the "camera length" (sample-detector distance) is adjusted, is not perpendicular
+to the detector plane.  The geometry file now allows you to specify this "rail"
+direction, containing x, y and z components.  See the crystfel_geometry manual
+page for more details.
+
+
+API changes
+-----------
+
+The following changes have been made to the libcrystfel API.  The biggest
+changes are the switch from the old "hdfile" API (which is still supported),
+to the new "imagefile" API, while allows multiple file types to be used.
+The indexing system was also updated, and there were changes to the detector
+geometry system, which no longer uses the "data slab" representation internally.
+
+New functions:
+	- crystal_{get,set}_det_shift()
+	- panel_number()
+	- get_detector_geometry_2()
+	- fill_in_adu()
+	- adjust_centering_for_rail()
+	- get_peaks_2()
+	- get_peaks_cxi_2()
+	- imagefile_open()
+	- imagefile_read()
+	- imagefile_read_simple()
+	- imagefile_get_hdfile()
+	- imagefile_get_type()
+	- imagefile_copy_fields()
+	- imagefile_close()
+	- {new,free}_imagefile_field_list()
+	- add_imagefile_field()
+	- get_indm_from_string()
+	- setup_indexing()
+	- index_pattern_2()
+	- search_peaks_peakfinder8()
+	- reflist_add_command_and_version()
+	- reflist_{add,get}_notes()
+	- write_chunk_2()
+	- pointgroup_warning()
+
+New structure definitions:
+	- struct image
+	- struct imagefile_field_list
+
+Removed functions:
+	- get_q()
+	- find_panel()
+	- find_panel_number()
+	- fill_in_values()
+	- partial_event_substitution()
+	- build_indexer_list()
+	- prepare_indexing()
+
+Changed function prototypes:
+	- in_bad_region()
+	- simple_geometry()
+	- reverse_2d_mapping()
+	- hdfile_set_image()
+	- image_add_feature()
+	- image_feature_closest()
+	- image_reflection_closest()
+	- index_pattern()
+	- cleanup_indexing()
+	- write_chunk()
+
+Changed structure definitions:
+	- struct panel: removed {min,max}_{fs,ss}, added adu_per_photon,
+	    added fsz,ssz, rail_{x,y,z} and clen_for_centering
+	- struct detector: removed max_{fs,ss}
+	- struct imagefeature: replaced "pn" with "p"
+	- struct beam_params: "copyme" is not an imagefile_field_list
+	    (previously: copy_hdf5_field)
+	- struct image: "copyme" is not an imagefile_field_list
+	    (previously: copy_hdf5_field).  width and height remoted.
+