This is CrystFEL 0.6.10.6.1

8 files changed, 217 insertions, 263 deletions

diff --git a/ChangeLog b/ChangeLog
index b20db9af..49f5e830 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,5 @@
-Changes in this development version of CrystFEL
------------------------------------------------
+CrystFEL version 0.6.1, 3rd August 2015
+---------------------------------------
 
 - partialator: big improvements, now works well on most data sets.
 - indexamajig: prediction refinement added.
@@ -9,11 +9,14 @@ Changes in this development version of CrystFEL
 - process_hkl --even-only and --odd-only were added.
 - External bad pixel masks can now be used (see "mask_file" in geometry file).
 - indexamajig: Can now use --int-diag with -j n where n>1
+- Image data filters (e.g. --noise-filter) were fixed.
+- indexamajig sandbox was redesigned and made more robust.
 - compare_hkl now uses a B-factor for scaling the datasets.
 - pattern_sim: spectrum normalisation added.
 - scripts/ave-resolution and scripts/create-frame-number added.
 - scripts/plot-predict-refine and scripts/plot-radius-resolution added.
 - scripts/detector-shift added.
+- scripts/turbo-index added.
 - "reax" and "grainspotter" indexing methods removed.
 - Many other small but important bug fixes and improvements to data quality.
 
diff --git a/Makefile.am b/Makefile.am
index a6acd357..32cc8a63 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -29,7 +29,7 @@ TESTS = tests/list_check $(MERGE_CHECKS) $(PARTIAL_CHECKS) \
         tests/prediction_gradient_check
 
 EXTRA_DIST += $(MERGE_CHECKS) $(PARTIAL_CHECKS)
-EXTRA_DIST += relnotes-0.6.0
+EXTRA_DIST += relnotes-0.6.1 announcement-0.6.1
 
 if BUILD_HDFSEE
 bin_PROGRAMS += src/hdfsee
diff --git a/announcement-0.6.1 b/announcement-0.6.1
new file mode 100644
index 00000000..85137402
--- /dev/null
+++ b/announcement-0.6.1
@@ -0,0 +1,43 @@
+Dear CrystFEL users and interested crystallographers,
+
+CrystFEL version 0.6.1 has been released.  The most interesting changes
+in this version are the new "prediction refinement" stage in
+"indexamajig" and improvements to the scaling and post-refinement
+program "partialator".  There are also a couple of new indexing methods.
+
+The prediction refinement stage improves the quality of spot prediction
+and hence the initial parameter estimates of the spot partialities.  It
+is also able to update the central beam position.
+
+The improvements to partialator include a significantly more stable
+scaling procedure.  Partialator is no longer considered experimental as
+of this version, so please try it and send feedback.  With the
+combination of this and the new prediction refinement, many users are
+already finding significant improvements in data quality.  See the
+release notes (link below) and the updated tutorial on the CrystFEL
+website for more details.
+
+The new indexing methods are "asdf", which is an algorithm built into
+CrystFEL which therefore requires no external programs, and
+"mosflm-cell" which uses the prior unit cell algorithm in recent (7.2.0
+or later) versions of mosflm.  "mosflm-cell" is now the default
+behaviour for mosflm - if it doesn't work for you, explicitly specify
+"mosflm-nocell" to restore the old behaviour.
+
+In addition, there are many smaller improvements, new features and bug
+fixes.
+
+Thanks for all of your past and future feedback and contributions, and
+your continued use of CrystFEL.  As ever, please contact me directly if
+you would prefer to be unsubscribed from this mailing list.
+
+CrystFEL website:
+http://www.desy.de/~twhite/crystfel
+
+Release notes for version 0.6.1:
+http://www.desy.de/~twhite/crystfel/relnotes-0.6.1
+
+CrystFEL tutorial:
+http://www.desy.de/~twhite/crystfel/tutorial
+
+Tom
diff --git a/configure.ac b/configure.ac
index 0971acb6..9d4d7edf 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,6 +1,6 @@
 dnl Process this file with autoconf to produce a configure script.
 
-AC_INIT([crystfel],[0.6.0],[taw@physics.org])
+AC_INIT([crystfel],[0.6.1],[taw@physics.org])
 AC_CONFIG_AUX_DIR([build-aux])
 AC_CONFIG_HEADERS([config.h])
 AM_INIT_AUTOMAKE([subdir-objects])
diff --git a/doc/reference/libcrystfel/CrystFEL-docs.sgml b/doc/reference/libcrystfel/CrystFEL-docs.sgml
index 6c7baa74..9ef5d945 100644
--- a/doc/reference/libcrystfel/CrystFEL-docs.sgml
+++ b/doc/reference/libcrystfel/CrystFEL-docs.sgml
@@ -8,7 +8,7 @@
   <bookinfo>
     <title>CrystFEL Reference Manual</title>
     <releaseinfo>
-      For libcrystfel from CrystFEL 0.6.0.
+      For libcrystfel from CrystFEL 0.6.1.
     </releaseinfo>
     <abstract>
     This is the internal documentation for CrystFEL.  Unless you are looking at
diff --git a/libcrystfel/Makefile.am b/libcrystfel/Makefile.am
index 1a0c3efe..77e5991b 100644
--- a/libcrystfel/Makefile.am
+++ b/libcrystfel/Makefile.am
@@ -1,6 +1,6 @@
 lib_LTLIBRARIES = libcrystfel.la
 libcrystfel_la_LIBADD = ../lib/libgnu.la @LIBCRYSTFEL_LIBS@ $(PTY_LIB)
-libcrystfel_la_LDFLAGS = -version-info 6:0:0
+libcrystfel_la_LDFLAGS = -version-info 7:0:0
 
 libcrystfel_la_SOURCES = src/reflist.c src/utils.c src/cell.c src/detector.c \
                          src/thread-pool.c src/image.c src/hdf5-file.c \
diff --git a/relnotes-0.6.0 b/relnotes-0.6.0
deleted file mode 100644
index 22efa56d..00000000
--- a/relnotes-0.6.0
+++ /dev/null
@@ -1,257 +0,0 @@
-CrystFEL - Crystallography with a FEL
--------------------------------------
-
-Release notes for version 0.6.0
-
-Copyright © 2012-2015 Deutsches Elektronen-Synchrotron DESY,
-                      a research centre of the Helmholtz Association.
-
-Authors:
-  Thomas White <taw@physics.org>
-  Richard Kirian <rkirian@asu.edu>
-  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
-  Andrew Aquila <andrew.aquila@cfel.de>
-  Andrew Martin <andrew.martin@desy.de>
-  Lorenzo Galli <lorenzo.galli@desy.de>
-  Chun Hong Yoon <chun.hong.yoon@desy.de>
-  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
-  Karol Nass <karol.nass@desy.de>
-  Nadia Zatsepin <nadia.zatsepin@asu.edu>
-  Anton Barty <anton.barty@desy.de>
-  Cornelius Gati <cornelius.gati@desy.de>
-  Fedor Chervinskii <fedor.chervinskii@gmail.com>
-  Alexandra Tolstikova <alexandra.tolstikova@desy.de>
-  Wolfgang Brehm <wolfgang.brehm@gmail.com>
-  Valerio Mariani <valerio.mariani@desy.de>
-  Parker de Waal <Parker.deWaal@vai.org>
-  Takanori Nakane <nakane.t@gmail.com>
-  Keitaro Yamashita <k.yamashita@spring8.or.jp>
-  Oleksandr Yefanov <oleksandr.yefanov@cfel.de>
-
-CrystFEL is free software: you can redistribute it and/or modify it under the
-terms of the GNU General Public License as published by the Free Software
-Foundation, either version 3 of the License, or (at your option) any later
-version.
-
-CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
-WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
-PARTICULAR PURPOSE.  See the GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along with
-CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
-
-
-Overview
---------
-
-The most important new features in this version of CrystFEL are:
-
-- Support for multi-event HDF5 files (e.g. CXI format as used by CXIDB)
-
-- geoptimiser: a new tool for precisely optimising the detector geometry
-
-- Removal of beam files / automatic determination of spot prediction parameters
-
-- whirligig: a new tool for finding clusters of similarly-oriented
-   crystal snapshots, e.g. for finding "mini rotation series" in data
-   from slow extrusion sample delivery methods.
-
-- Introduction of "CrystFEL unit cell files".
-
-These new features have individual sections below.  In addition, there are many
-other new developments.  See the ChangeLog or the changes page on the website
-for more details.  There were, of course, the usual large number of smaller
-refinements and bug fixes.
-
-
-Support for multi-event HDF5 files
-----------------------------------
-
-CrystFEL's handling of HDF5 files has been made much more flexible.  Most
-importantly, it now offers the ability to handle HDF5 files which contain more
-than one event (frame).  Until now, CrystFEL has required that each event be
-contained in its own file, which can place a lot of unnecessary strain on
-filesystems.  Recent versions of Cheetah allow you to create larger files which
-contain many events each using the CXI format (see http://www.cxidb.org for more
-details).  CrystFEL now supports this as well as almost any reasonable
-multi-event HDF5 layout.
-
-If you choose to continue using many small files, you should not notice much
-difference.  However, some small updates will be required.  Firstly,
-indexamajig's "-e" and "--image" command-line parameters have been removed.
-Instead, you need to edit your geometry file and add a line such as this near
-the top:
-
-        data = /data/rawdata
-
-Geometry files now contain all the information needed to interpret the contents
-of the HDF5 files as a physical setup, including the photon energy.  Beam
-parameter files have been removed.  See below for more information about this.
-
-When using multi-event files in CXI format, the peak lists are read differently.
-Use --peaks=cxi to retrieve peak lists from CXI files.
-
-hdfsee has been extended to support multi-event files, including navigation
-between events.  You should always provide a geometry file on the command line,
-otherwise it won't know how to interpret the contents of your HDF5 file.
-
-A new tool in CrystFEL 0.6.0, list_events, is provided to simplify the process
-of creating lists of individual events.  However, most users will simply be
-able to list the multi-event filenames themselves in the input to indexamajig,
-which will then process all of the events in the file.  See "man indexamajig"
-and "man list_events" for more details.
-
-Finally, it may be necessary to update your check-near-bragg and
-check-peak-detection scripts.  Simply make a fresh copy from the CrystFEL
-"scripts" folder or download them from the website.
-
-
-geoptimiser: a tool for optimising the detector geometry
---------------------------------------------------------
-
-CrystFEL 0.6.0 introduces a new tool, geoptimiser, for optimising the detector
-geometry.  You simply give it a stream containing indexing results obtained with
-the approximate geometry, and it gives you a refined geometry.  It can refine
-panel translations, rotations and camera lengths.
-
-To use geoptimiser, you need to add some extra information about the
-mechanical construction of your detector.  This is used to constrain the
-refinement appropriately, for example to ensure that panels which share sensor
-silicon do not move relative to one another.  If you're processing CSPAD data,
-you can simply copy all the rigid_group lines from one of the CSPAD geometry
-file examples (folder doc/examples) into your own geometry file.
-
-Refer to "man geoptimiser" and "man crystfel_geometry" for more information.
-
-
-Removal of beam files / automatic determination of spot prediction parameters
------------------------------------------------------------------------------
-
-As of version 0.6.0, "beam files" have been removed completely.  This should
-simplify usage for most people and remove a lot of ambiguity.  Programs which
-need X-ray beam parameters, such as pattern_sim, now have additional command-
--line arguments to provide them.  indexamajig now determines these parameters
-automatically for the purposes of spot prediction.
-
-We are interested in feedback about the automatic spot prediction parameter
-determination.  If it appears not to work well for you, you can restore the old
-behaviour by using the new command-line options for indexamajig:
---fix-profile-radius, --fix-bandwidth and --fix-divergence.  Simply set these
-parameters to the values you had in your old beam file.
-
-The photon energy, or information about where to get it, now needs to be in the
-geometry file.  This can be done with a line like this:
-
-       photon_energy = /LCLS/photon_energy_eV
-
-or, for a fixed value:
-
-       photon_energy = 8300
-
-
-whirligig: a tool for finding "mini rotation series"
-----------------------------------------------------
-
-CrystFEL 0.6.0 introduces yet another tool, whirligig, which can be used for
-locating "mini rotation series" in the output from indexamajig.  This might be
-used to perform an experiment similar to that described by Gati et al.,  IUCrJ
-1 (2014) p87.  In this initial version, whirligig finds runs of consecutive
-frames which contain crystals in similar orientations.  It writes the
-corresponding filenames and event/crystal identifiers to log files, which might
-be useful for further analysis.
-
-This is the program described by Nogly et al., IUCrJ 2 (2015).  Refer to "man
-whirligig" for usage information.
-
-
-Introduction of CrystFEL unit cell files
-----------------------------------------
-
-CrystFEL offers you the ability to index patterns using Bravais lattice
-information but without unit cell parameters, for example: "index these using
-only tetragonal primitive lattices, but any parameters".  However, it's awkward
-to give this information using a PDB file: you essentially have to "trick" it
-into interpreting the parameters correctly, then throw away the parameter
-information.
-
-Version 0.6.0 of CrystFEL introduces a new way of specifying unit cell
-information.  These new unit cell files look like this:
-
-         CrystFEL unit cell file version 1.0
-
-         lattice_type = cubic
-         centering = I
-
-         a = 66.2 A
-         b = 66.2 A
-         c = 66.2 A
-
-         al = 90.0 deg
-         be = 90.0 deg
-         ga = 90.0 deg
-
-In the event that you want to specify the lattice type information alone, you
-can simply omit the cell parameters:
-
-         CrystFEL unit cell file version 1.0
-
-         lattice_type = tetragonal
-         centering = P
-	 unique_axis = c
-
-Note that a unique axis must be specified for all types of cell where this makes
-sense, and can be omitted otherwise.  This was done in the first example above,
-which is cubic and therefore has no unique axis.
-
-You can, of course, continue to use PDB files just like before.
-
-
-API changes
------------
-
-The following changes have been made to the libcrystfel API:
-
-New functions:
-	- The event API (see libcrystfel/src/events.h)
-	- load_cell_from_file()
-	- cell_has_parameters()
-	- find_orig_panel()
-	- crystal_{get,set}_num_implausible_reflections() were added
-	- panel_is_in_rigid_group()
-	- rigid_group_is_in_collection()
-	- single_panel_data_source()
-	- find_rigid_group_collection_by_name()
-	- write_detector_geometry_2()
-	- find_intersections_to_res()
-	- sphere_fraction()
-	- gaussian_fraction()
-	- hdf5_read2()
-	- check_path_existence()
-	- get_peaks_cxi()
-	- hdfile_get_value()
-	- fill_event_list()
-	- image_reflection_closest()
-	- intmat_identity()
-	- extract_f_from_stuff()
-
-Removed functions:
-	- cell_set_cartesian_{a,b,c}()
-	- cell_{get,set}_pointgroup()
-	- twod_mapping()
-	- get_value()
-
-Changed function prototypes:
-	- record_image()
-	- get_detector_geometry()
-	- fill_in_values()
-	- write_detector_geometry()
-	- hdf5_write_image()
-	- hdf5_read()
-	- hdfile_set_image()
-	- hdfile_get_string_value()
-	- {get,set}_partial()
-	- write_chunk()
-	- prepare_indexing() and {dirax,mosflm,reax,grainspotter,xds}_prepare()
-
-beam-parameters.h was removed, and "struct beam_params" is now defined in
-image.h.
diff --git a/relnotes-0.6.1 b/relnotes-0.6.1
new file mode 100644
index 00000000..4c189cd6
--- /dev/null
+++ b/relnotes-0.6.1
@@ -0,0 +1,165 @@
+CrystFEL - Crystallography with a FEL
+-------------------------------------
+
+Release notes for version 0.6.1
+
+Copyright © 2012-2015 Deutsches Elektronen-Synchrotron DESY,
+                      a research centre of the Helmholtz Association.
+
+Authors:
+  Thomas White <taw@physics.org>
+  Richard Kirian <rkirian@asu.edu>
+  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
+  Andrew Aquila <andrew.aquila@cfel.de>
+  Andrew Martin <andrew.martin@desy.de>
+  Lorenzo Galli <lorenzo.galli@desy.de>
+  Chun Hong Yoon <chun.hong.yoon@desy.de>
+  Kenneth Beyerlein <kenneth.beyerlein@desy.de>
+  Karol Nass <karol.nass@desy.de>
+  Nadia Zatsepin <nadia.zatsepin@asu.edu>
+  Anton Barty <anton.barty@desy.de>
+  Cornelius Gati <cornelius.gati@desy.de>
+  Fedor Chervinskii <fedor.chervinskii@gmail.com>
+  Alexandra Tolstikova <alexandra.tolstikova@desy.de>
+  Wolfgang Brehm <wolfgang.brehm@gmail.com>
+  Valerio Mariani <valerio.mariani@desy.de>
+  Parker de Waal <Parker.deWaal@vai.org>
+  Takanori Nakane <nakane.t@gmail.com>
+  Keitaro Yamashita <k.yamashita@spring8.or.jp>
+  Oleksandr Yefanov <oleksandr.yefanov@cfel.de>
+
+CrystFEL is free software: you can redistribute it and/or modify it under the
+terms of the GNU General Public License as published by the Free Software
+Foundation, either version 3 of the License, or (at your option) any later
+version.
+
+CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
+PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with
+CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
+
+
+Overview
+--------
+
+The most important new features in this version of CrystFEL are:
+
+- Introduction of "prediction refinement".
+
+- Scaling and post-refinement with "partialator".
+
+- New indexing methods: mosflm-cell and asdf.
+
+- process_hkl --even-only and --odd-only instead of "alternate-stream".
+
+These new features have individual sections below.  In addition, there are many
+other new developments.  See the ChangeLog or the changes page on the website
+for more details.  There were, of course, the usual large number of smaller
+refinements and bug fixes.
+
+
+Introduction of "prediction refinement"
+---------------------------------------
+
+A new refinement step has been introduced between the indexing and prediction/
+integration stages of indexamajig.  The refinement aims to maximise the
+agreement between the predicted spot positions and the peak positions found by
+the peak search step, at the same time as putting the peaks as close to the
+Bragg condition as possible.  This increases the accuracy of spot prediction,
+which has positive effects on the downstream processing stages.
+
+The prediction refinement stage allows the central beam position to move
+relative to the entire detector.  The required offset for each crystal is
+recorded in the stream and can be visualised using scripts/detector-shift.
+
+If, somehow, the prediction refinement causes problems for you, you can disable
+it using the indexamajig option "--no-refine".
+
+
+Scaling and post-refinement with "partialator"
+----------------------------------------------
+
+The scaling and post-refinement program partialator is no longer considered
+experimental in this version.  Many users are already finding significant
+improvements in their data simply by applying scaling without partiality:
+
+$ partialator -i my.stream -o my.hkl -y mypg --model=unity --iterations=3
+
+A further improvement is sometimes obtained by applying partialities and doing
+full post-refinement:
+
+$ partialator -i my.stream -o my.hkl -y mypg --model=scsphere --iterations=3
+
+Note that partialator applies a per-crystal resolution cutoff to stabilise the
+scaling calculation.  Therefore, you should experiment with different values
+for "--push-res".
+
+Further improvements to partiality modelling, scaling and merging will be
+included in the near future.
+
+
+New indexing methods
+--------------------
+
+This version of CrystFEL can make use of the known unit cell algorithm included
+with very recent (7.2.0 or later) versions of mosflm.  The known unit cell
+algorithm is used by default if you provide lattice parameters to indexamajig.
+If this isn't what you want, or if it doesn't work well for you, use
+"mosflm-nocell" as your indexing method to restore the old behaviour.
+
+A new indexing method, asdf, has been added.  This algorithm is built into
+CrystFEL directly, so no temporary files or external programs are needed.
+
+The old indexing methods "grainspotter" and "reax" have been removed.
+
+
+process_hkl --even-only and --odd-only
+--------------------------------------
+
+When using process_hkl for merging, you no longer need to use "alternate-stream"
+to split the stream into two parts in order to calculate figures of merit such
+as Rsplit.  Instead, simply use the "--even-only" and "--odd-only" options to
+merge only even-numbered and odd-numbered crystals from the stream:
+
+$ process_hkl -i my.stream -o my.hkl1 -y mypg --even-only
+$ process_hkl -i my.stream -o my.hkl2 -y mypg --odd-only
+$ compare_hkl my.hkl1 my.hkl2 -y mypg --fom=rsplit [..etc..]
+
+Note that the numbering of crystals for this purpose is simply according to the
+order they appear in the stream.  For most purposes, this is effectively random.
+
+Note further that partialator writes the ".hkl1" and ".hkl2" split-merged
+datasets for you automatically.
+
+
+API changes
+-----------
+
+The following changes have been made to the libcrystfel API:
+
+New functions:
+	- asdf_prepare(), run_asdf(), asdf_cleanup()
+	- cell_get_volume()
+	- crystal_{get,set}_Bfac()
+	- crystal_{get,set,add}_notes()
+	- detector_has_clen_references()
+	- {x,y,r}_gradient()
+	- refine_prediction()
+	- refine_radius()
+	- {get,set}_panel()
+	- {get,set}_flag()
+	- solve_svd()
+
+Removed functions:
+	- select_intersections()
+	- update_partialities_2()
+	- get_excitation_error()
+	- extract_f_from_stuff()
+	- grainspotter_{prepare,index,cleanup}()
+	- reax_{prepare,index,cleanup}()
+
+Changed function prototypes:
+	- integrate_all_4()
+	- set_detector_pos()