This is CrystFEL 0.7.00.7.0

9 files changed, 225 insertions, 257 deletions

diff --git a/ChangeLog b/ChangeLog
index 2625c6a9..4562a94b 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,21 +1,22 @@
-Changes in this development version
------------------------------------
+CrystFEL  version 0.7.0, 17th May 2018
+--------------------------------------
 
-- indexamajig: Auto-detection of indexing methods was added.
+- indexamajig: Auto-detection of indexing methods
 - indexamajig: New options system - removes confusing "comb","bad","retry" etc.
 - Most programs now use the symmetry in the reflection files, allowing "-y"
+  options to be omitted.
 - Ginn partiality model was added.
 - partialator: Switch to gradient-free minimiser.
+- geoptimiser: Fixes for rotation and new features for AGIPD.
+- Speed and accuracy improvements for TakeTwo indexing (Helen Ginn).
 - process_hkl, partialator and ambigator now include the audit information from
   the stream in their output files.
 - pattern_sim: Write orientation and lattice basis vectors into HDF5 file.
 - hdfsee: Add event ID and peaks/preds/rings to exported image filenames.
-  options to be omitted.
 - indexamajig --serial-start was added (useful when using turbo-index).
-- indexamajig --overpredict was added (useful for partiality
-  modelling/post-refinement)
+- indexamajig --overpredict was added (needed for post-refinement)
 - pattern_sim --flat was added (new reflection profile option)
-- cell_explorer: Show number of matching cells on terminal.
+- cell_explorer: Now shows the number of matching cells on terminal.
 - Felix options were simplified.
 - scripts/sum-hdf5-files and eiger-badmap were added
 
diff --git a/Makefile.am b/Makefile.am
index f5d38c2a..a669b1dd 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -29,7 +29,7 @@ TESTS = tests/list_check $(MERGE_CHECKS) $(PARTIAL_CHECKS) \
         tests/prediction_gradient_check tests/scaling_check
 
 EXTRA_DIST += $(MERGE_CHECKS) $(PARTIAL_CHECKS)
-EXTRA_DIST += relnotes-0.6.3 announcement-0.6.3
+EXTRA_DIST += relnotes-0.7.0 announcement-0.7.0
 
 if BUILD_HDFSEE
 bin_PROGRAMS += src/hdfsee
diff --git a/announcement-0.6.3 b/announcement-0.6.3
deleted file mode 100644
index db8d0adb..00000000
--- a/announcement-0.6.3
+++ /dev/null
@@ -1,35 +0,0 @@
-Dear CrystFEL users and interested crystallographers,
-
-It's been a long time since the last CrystFEL release, but we have not been
-resting.  CrystFEL version 0.6.3 has been released.
-
-This version incorporates a large number of "backstage" improvements and
-alterations.  The most interesting changes for users are:
-
-- The incorporation of "peak finder 8" from Cheetah (also used in OnDA)
-
-- Support for images in CBF format
-
-- Incorporation of "TakeTwo" indexing algorithm
-
-- Hit finder functionality (skip frames with few peaks)
-
-- Better support for three-dimensional detector geometries
-
-See the release notes for more discussion of these improvements, and the
-ChangeLog for full details of the other improvements.
-
-Thanks for all of your past and future feedback and contributions, and
-your continued use of CrystFEL.  As ever, please contact me directly if
-you would prefer to be unsubscribed from this mailing list.
-
-CrystFEL website:
-https://www.desy.de/~twhite/crystfel
-
-Release notes for version 0.6.3:
-https://www.desy.de/~twhite/crystfel/relnotes-0.6.3
-
-CrystFEL tutorial:
-https://www.desy.de/~twhite/crystfel/tutorial
-
-Tom
diff --git a/announcement-0.7.0 b/announcement-0.7.0
new file mode 100644
index 00000000..619dc570
--- /dev/null
+++ b/announcement-0.7.0
@@ -0,0 +1,37 @@
+Dear CrystFEL users and interested crystallographers,
+
+CrystFEL version 0.7.0 has been released.
+
+This version adds several exciting new features while greatly simplifying the
+user experience.  The most interesting changes are:
+
+- New options system for indexamajig, removing most of the profusion of indexing
+  method flags "comb", "bad", "retry" and so on.  Indexamajig can also now
+  automatically determine which indexing methods can be used if you don't tell
+  it which ones to use.  In addition, the options for the Felix indexing method
+  have been simplified.
+
+- The spectrum-based partiality model and numerical post-refinement algorithm
+  from Ginn et al. [Acta D71 2015 p1400] has been incorporated.
+
+- The symmetry of the merged reflection list is now automatically tracked through
+  the subsequent data processing stages, so you no longer need to give the "-y"
+  option to compare_hkl, check_hkl and other programs.
+
+See the release notes for more discussion of these improvements, and the
+ChangeLog for full details of the other improvements.
+
+Thanks for all of your past and future feedback and contributions, and
+your continued use of CrystFEL.  As ever, please contact me directly if
+you would prefer to be unsubscribed from this mailing list.
+
+CrystFEL website:
+https://www.desy.de/~twhite/crystfel
+
+Release notes for version 0.7.0:
+https://www.desy.de/~twhite/crystfel/relnotes-0.7.0
+
+CrystFEL tutorial:
+https://www.desy.de/~twhite/crystfel/tutorial
+
+Tom
diff --git a/configure.ac b/configure.ac
index 30e803f7..ef003372 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,6 +1,6 @@
 dnl Process this file with autoconf to produce a configure script.
 
-AC_INIT([crystfel],[0.6.3],[taw@physics.org])
+AC_INIT([crystfel],[0.7.0],[taw@physics.org])
 AC_CONFIG_AUX_DIR([build-aux])
 AC_CONFIG_HEADERS([config.h])
 AM_INIT_AUTOMAKE([subdir-objects])
diff --git a/doc/reference/libcrystfel/CrystFEL-docs.sgml b/doc/reference/libcrystfel/CrystFEL-docs.sgml
index 048f4167..6a1dabee 100644
--- a/doc/reference/libcrystfel/CrystFEL-docs.sgml
+++ b/doc/reference/libcrystfel/CrystFEL-docs.sgml
@@ -8,7 +8,7 @@
   <bookinfo>
     <title>CrystFEL Reference Manual</title>
     <releaseinfo>
-      For libcrystfel from CrystFEL 0.6.3.
+      For libcrystfel from CrystFEL 0.7.0.
     </releaseinfo>
     <abstract>
     This is the internal documentation for CrystFEL.  Unless you are looking at
diff --git a/libcrystfel/Makefile.am b/libcrystfel/Makefile.am
index 24cb7358..1d24d353 100644
--- a/libcrystfel/Makefile.am
+++ b/libcrystfel/Makefile.am
@@ -1,6 +1,6 @@
 lib_LTLIBRARIES = libcrystfel.la
 libcrystfel_la_LIBADD = ../lib/libgnu.la @LIBCRYSTFEL_LIBS@ $(PTY_LIB)
-libcrystfel_la_LDFLAGS = -version-info 9:0:0
+libcrystfel_la_LDFLAGS = -version-info 10:0:0
 
 libcrystfel_la_SOURCES = src/reflist.c src/utils.c src/cell.c src/detector.c \
                          src/thread-pool.c src/image.c src/hdf5-file.c \
diff --git a/relnotes-0.6.3 b/relnotes-0.6.3
deleted file mode 100644
index 11a1f93b..00000000
--- a/relnotes-0.6.3
+++ /dev/null
@@ -1,211 +0,0 @@
-CrystFEL - Crystallography with a FEL
--------------------------------------
-
-Release notes for version 0.6.3
-
-Copyright © 2012-2017 Deutsches Elektronen-Synchrotron DESY,
-                      a research centre of the Helmholtz Association.
-
-Authors:
-  Thomas White <taw@physics.org>
-  Richard Kirian <rkirian@asu.edu>
-  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
-  Andrew Aquila <andrew.aquila@cfel.de>
-  Andrew Martin <andrew.martin@desy.de>
-  Lorenzo Galli <lorenzo.galli@desy.de>
-  Chun Hong Yoon <chun.hong.yoon@desy.de>
-  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
-  Karol Nass <karol.nass@desy.de>
-  Nadia Zatsepin <nadia.zatsepin@asu.edu>
-  Anton Barty <anton.barty@desy.de>
-  Cornelius Gati <cornelius.gati@desy.de>
-  Fedor Chervinskii <fedor.chervinskii@gmail.com>
-  Alexandra Tolstikova <alexandra.tolstikova@desy.de>
-  Wolfgang Brehm <wolfgang.brehm@gmail.com>
-  Valerio Mariani <valerio.mariani@desy.de>
-  Parker de Waal <Parker.deWaal@vai.org>
-  Takanori Nakane <nakane.t@gmail.com>
-  Keitaro Yamashita <k.yamashita@spring8.or.jp>
-  Oleksandr Yefanov <oleksandr.yefanov@cfel.de>
-  Steve Aplin <steve.aplin@desy.de>
-  Helen Ginn <helen@strubi.ox.ac.uk>
-
-CrystFEL is free software: you can redistribute it and/or modify it under the
-terms of the GNU General Public License as published by the Free Software
-Foundation, either version 3 of the License, or (at your option) any later
-version.
-
-CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
-WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
-PARTICULAR PURPOSE.  See the GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along with
-CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
-
-
-Overview
---------
-
-The most important new features in this version of CrystFEL are:
-
-- Support for CBF files as input
-
-- TakeTwo indexing algorithm
-
-- Improved peak detection and basic hitfinder functionality
-
-- Detector panels not perpendicular to X-ray beam, and "rail direction"
-
-These changes have sections below.  In addition, there were many bug fixes and
-other improvements.  See the "ChangeLog" or the changes page on the CrystFEL
-website for details.
-
-
-Support for CBF files as input
-------------------------------
-
-Crystallographic Binary Files (CBF) can now be used as input for indexamajig
-and hdfsee.  To enable this functionality, you must have "CBFlib" installed.
-Some distributions (eg Fedora) include CBFlib in their standard repositories -
-simply install "cbflib-devel" or similar.  Then compile and install CrystFEL as
-usual.  If CBFlib is installed somewhere non-standard, e.g. if you has to
-install it manually, you will need to give the path with the "--with-cbflib-dir"
-option to "./configure".
-
-After that, you can simply give the names of CBF files in the indexamajig input,
-and everything should work as usual!  There are some catches, for example you
-cannot yet use a bad pixel mask with CBF files, but these will be fixed in a
-future release.
-
-
-TakeTwo indexing algorithm
---------------------------
-
-This version of CrystFEL includes the TakeTwo indexing algorithm by Helen Ginn.
-TakeTwo is a new indexing algorithm, designed specifically for diffraction
-snapshots and operating completely differently to conventional FFT-based methods.
-
-To use it, simply use "taketwo" as the indexing method, or add it to the list of
-indexing methods you have already:
-   $ indexamajig --indexing=taketwo
-   $ indexamajig --indexing=mosflm,taketwo,dirax
-
-If you use TakeTwo, please take careful note of the citation reminder which will
-be shown to you, and cite the following paper in any resulting publications:
-   Ginn et al., Acta Cryst. (2016). D72, 956-965
-
-TakeTwo is considered experimental in this version of CrystFEL.  We welcome any
-feedback, and will use the information to improve its performance for future
-versions.
-
-
-Improved peak detection and basic hitfinder functionality
----------------------------------------------------------
-
-"Peak finder 8" from Cheetah, also used in OnDA, has been incorporated into
-CrystFEL.  It uses a radially varying background level to find peaks, which
-works very well with most diffraction data.
-
-Simply use "--peaks=peakfinder8", and set --threshold, --min-snr,
---min-pix-count and --local-bg-radius.  See the manual for other options.
-
-Indexamajig can now skip over patterns which have a small number of peaks.  To
-use this, use the option "--min-peaks=N", where N is the minimum number of peaks
-before indexamajig will proceed to indexing (and hopefully integration) with the
-pattern.
-
-By combining the improved peak detection with the hitfinder functionality and
-CBF reading, you can process data from several different types of detector,
-without converting the files to HDF5 format or using any external program for
-hit finding!
-
-If the hit rate is very low, you might want to make CrystFEL stream smaller by
-excluding non-hits from it using "--no-non-hits-in-stream".  Without this
-option, standard information about the non-hits (e.g. photon energy and peaks
-found) will be recorded.
-
-
-Detector panels not perpendicular to X-ray beam, and "rail direction"
-----------------------------------------------------------------------
-
-CrystFEL now supports three-dimensional detector geometry.  The "fs" and "ss"
-vectors (which are the real-space directions of the fast scan and slow scan
-directions in the image data) can now contain a z component as well as x and y.
-
-For many detectors, the "rail" direction, along which the detector moves when
-the "camera length" (sample-detector distance) is adjusted, is not perpendicular
-to the detector plane.  The geometry file now allows you to specify this "rail"
-direction, containing x, y and z components.  See the crystfel_geometry manual
-page for more details.
-
-
-API changes
------------
-
-The following changes have been made to the libcrystfel API.  The biggest
-changes are the switch from the old "hdfile" API (which is still supported),
-to the new "imagefile" API, while allows multiple file types to be used.
-The indexing system was also updated, and there were changes to the detector
-geometry system, which no longer uses the "data slab" representation internally.
-
-New functions:
-	- crystal_{get,set}_det_shift()
-	- panel_number()
-	- get_detector_geometry_2()
-	- fill_in_adu()
-	- adjust_centering_for_rail()
-	- get_peaks_2()
-	- get_peaks_cxi_2()
-	- imagefile_open()
-	- imagefile_read()
-	- imagefile_read_simple()
-	- imagefile_get_hdfile()
-	- imagefile_get_type()
-	- imagefile_copy_fields()
-	- imagefile_close()
-	- {new,free}_imagefile_field_list()
-	- add_imagefile_field()
-	- get_indm_from_string()
-	- setup_indexing()
-	- index_pattern_2()
-	- search_peaks_peakfinder8()
-	- reflist_add_command_and_version()
-	- reflist_{add,get}_notes()
-	- write_chunk_2()
-	- pointgroup_warning()
-
-New structure definitions:
-	- struct image
-	- struct imagefile_field_list
-
-Removed functions:
-	- get_q()
-	- find_panel()
-	- find_panel_number()
-	- fill_in_values()
-	- partial_event_substitution()
-	- build_indexer_list()
-	- prepare_indexing()
-
-Changed function prototypes:
-	- in_bad_region()
-	- simple_geometry()
-	- reverse_2d_mapping()
-	- hdfile_set_image()
-	- image_add_feature()
-	- image_feature_closest()
-	- image_reflection_closest()
-	- index_pattern()
-	- cleanup_indexing()
-	- write_chunk()
-
-Changed structure definitions:
-	- struct panel: removed {min,max}_{fs,ss}, added adu_per_photon,
-	    added fsz,ssz, rail_{x,y,z} and clen_for_centering
-	- struct detector: removed max_{fs,ss}
-	- struct imagefeature: replaced "pn" with "p"
-	- struct beam_params: "copyme" is not an imagefile_field_list
-	    (previously: copy_hdf5_field)
-	- struct image: "copyme" is not an imagefile_field_list
-	    (previously: copy_hdf5_field).  width and height remoted.
-
diff --git a/relnotes-0.7.0 b/relnotes-0.7.0
new file mode 100644
index 00000000..186694a0
--- /dev/null
+++ b/relnotes-0.7.0
@@ -0,0 +1,176 @@
+CrystFEL - Data processing for serial crystallography
+-----------------------------------------------------
+
+Release notes for version 0.7.0
+
+Copyright © 2012-2018 Deutsches Elektronen-Synchrotron DESY,
+                      a research centre of the Helmholtz Association.
+
+Authors:
+  Thomas White <taw@physics.org>
+  Richard Kirian <rkirian@asu.edu>
+  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
+  Andrew Aquila <andrew.aquila@cfel.de>
+  Andrew Martin <andrew.martin@desy.de>
+  Lorenzo Galli <lorenzo.galli@desy.de>
+  Chun Hong Yoon <chun.hong.yoon@desy.de>
+  Karol Nass <karol.nass@desy.de>
+  Nadia Zatsepin <nadia.zatsepin@asu.edu>
+  Anton Barty <anton.barty@desy.de>
+  Cornelius Gati <cornelius.gati@desy.de>
+  Fedor Chervinskii <fedor.chervinskii@gmail.com>
+  Alexandra Tolstikova <alexandra.tolstikova@desy.de>
+  Wolfgang Brehm <wolfgang.brehm@gmail.com>
+  Valerio Mariani <valerio.mariani@desy.de>
+  Parker de Waal <Parker.deWaal@vai.org>
+  Takanori Nakane <nakane.t@gmail.com>
+  Keitaro Yamashita <k.yamashita@spring8.or.jp>
+  Oleksandr Yefanov <oleksandr.yefanov@cfel.de>
+  Steve Aplin <steve.aplin@desy.de>
+  Helen Ginn <helen@strubi.ox.ac.uk>
+  Thomas Grant <tgrant@hwi.buffalo.edu>
+  Mamoru Suzuki <mamoru.suzuki@protein.osaka-u.ac.jp>
+
+CrystFEL is free software: you can redistribute it and/or modify it under the
+terms of the GNU General Public License as published by the Free Software
+Foundation, either version 3 of the License, or (at your option) any later
+version.
+
+CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
+PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with
+CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
+
+
+Overview
+--------
+
+The most important new features in this version of CrystFEL are:
+
+- New options system for indexamajig, removing most of the profusion of indexing
+  method flags "comb", "bad", "retry" and so on.  Indexamajig can also now
+  automatically determine which indexing methods can be used if you don't tell
+  it which ones to use.  In addition, the options for the Felix indexing method
+  have now been greatly simplified.
+
+- The spectrum-based partiality model and numerical post-refinement algorithm
+  from Ginn et al [Acta D71 2015 p1400] was incorporated.
+
+- The symmetry of the merged reflection list is now tracked through automatically,
+  so you no longer need to give the "-y" option to compare_hkl and check_hkl.
+
+These changes have sections below.  In addition, there were many bug fixes and
+other improvements.  See the "ChangeLog" or the changes page on the CrystFEL
+website for details.
+
+
+New indexamajig options system
+------------------------------
+
+The new version of indexamajig will automatically determine which indexing
+methods to use based on the prior information (e.g. if a target unit cell was
+provided) and which indexing programs are available on the computer.  To enable
+the automatic determination, simply omit any "--indexing=" option to indexamajig.
+Of course, you can override the automatic choice by specifying --indexing, just
+as it worked before.
+
+To options for finer control of the indexing process, which in previous versions
+were enabled or disabled by adding flags to the indexing methods
+(e.g. "mosflm-noretry"), now apply to all indexing methods at once and are
+selected using new options, listed below.
+
+The indexing method should contain only the method itself and prior information
+modifiers ('cell' or 'latt').  This simplifies things greatly by removing the
+very long indexing method descriptions ("mosflm-raw-latt-noretry-refine-nomulti"
+and so on).
+
+- To disable prediction refinement ('norefine'), use --no-refine.
+- To check cell axes only ('axes'), use --no-cell-combinations.
+- To disable all unit cell checks ('raw'), use --no-check-cell.
+- To disable peak alignment check ('bad'), use --no-check-peaks.
+- To disable indexing retry ('noretry'), use --no-retry.
+- To enable multi-lattice indexing by 'delete and retry', use --multi.
+
+
+Spectrum-based partiality model and post-refinement
+---------------------------------------------------
+
+This version includes an implementation of the spectrum-based partiality model
+described by Ginn et al., Acta D71 2015 p1400 and associated numerical post-
+refinement algorithm.  To use it, use "partialator --model=xsphere".  The old
+models (scsphere and scgaussian) have been removed.  You will probably need to
+experiment with changing the bandwidth and reflection radius, using
+--force-bandwidth and --force-radius.  Add --no-pr to disable post-refinement,
+while still using the partiality estimates.  To use post-refinement, omit --no-pr,
+but you should also run indexamajig with "--overpredict".  You may also need to
+disable B factor scaling in partialator with "--no-Bscale".
+
+If you use post-refinement, the new scripts "plot-pr" and "plot-contourmap" can
+be used to visualise the distributions of partialities and the contour maps of
+residual against parameters such as crystal rotation angle.  The data for these
+graphs is in the "pr-logs" folder which will be created by partialator.
+
+Partiality modelling and post-refinement should still be considered experimental
+when applied to real data.  The success of post-refinement depends on many
+factors, and only has a chance of working if there are no other problems with
+the dataset (e.g. inaccurate detector geometry).  Further improvements should
+appear in future versions.
+
+
+Automatic tracking of reflection list symmetry
+----------------------------------------------
+
+The point group symmetry used for merging a reflection list has been stored in
+the reflection file (.hkl) for a long time.  However, version 0.7.0 of CrystFEL
+uses the stored symmetry when performing tasks such as calculating figures of
+merit using check_hkl.  You can simply omit the "-y" options which were needed
+previously.  Of course, you can override the symmetry by specifying -y as before.
+
+
+API changes
+-----------
+
+The following changes have been made to the libcrystfel API.
+
+New functions:
+	- asdf_probe() and _probe for all other indexing methods.
+	- compare_cells()
+	- crystal_get_cell_const()
+	- crystal_get_profile_radius()
+	- predict_to_res()
+	- calculate_partialities()
+	- update_predictions()
+	- is_cbf_file()
+	- get_indm_from_string_2()
+	- detect_indexing_methods()
+	- integrate_all_5()
+	- read_reflections_2()
+	- {get,set}_{khalf,kpred,exerr}()
+	- first_refl_const(), next_refl_const()
+	- open_stream_for_write_4()
+	- stream_has_old_indexers()
+	- stream_audit_info()
+
+Removed functions:
+	- find_intersections()
+	- find_intersections_to_res()
+	- update_partilities()
+	- write_reflections_to_file()
+	- read_reflections_from_file()
+	- {get,sh}_partial()
+
+Changed function prototypes:
+	- asdf_prepare() and _prepare for all other indexing methods.
+	- cell_new_from_cell() (input UnitCell is now const)
+	- crystal_copy() (input Crystal is now const)
+	- x_gradient(), y_gradient()
+	- remove_flagged_crystals() (now returns number of crystals removed)
+	- setup_indexing()
+
+Changed structure/enum definitions:
+	- PartialityModel: removed PMODEL_SCSPHERE and PMODEL_SCGAUSSIAN,
+	  added PMODEL_XSPHERE
+	- gparam: added GPARAM_ANG1, GPARAM_ANG2, GPARAM_WAVELENGTH, GPARAM_EOL
+	- Added IndexingFlags