From 87fd650abf11da75e7abd2ff90821cf7cb041b91 Mon Sep 17 00:00:00 2001 From: Thomas White Date: Tue, 15 May 2018 15:32:09 +0200 Subject: This is CrystFEL 0.7.0 --- ChangeLog | 15 +- Makefile.am | 2 +- announcement-0.6.3 | 35 ----- announcement-0.7.0 | 37 +++++ configure.ac | 2 +- doc/reference/libcrystfel/CrystFEL-docs.sgml | 2 +- libcrystfel/Makefile.am | 2 +- relnotes-0.6.3 | 211 --------------------------- relnotes-0.7.0 | 176 ++++++++++++++++++++++ 9 files changed, 225 insertions(+), 257 deletions(-) delete mode 100644 announcement-0.6.3 create mode 100644 announcement-0.7.0 delete mode 100644 relnotes-0.6.3 create mode 100644 relnotes-0.7.0 diff --git a/ChangeLog b/ChangeLog index 2625c6a9..4562a94b 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,21 +1,22 @@ -Changes in this development version ------------------------------------ +CrystFEL version 0.7.0, 17th May 2018 +-------------------------------------- -- indexamajig: Auto-detection of indexing methods was added. +- indexamajig: Auto-detection of indexing methods - indexamajig: New options system - removes confusing "comb","bad","retry" etc. - Most programs now use the symmetry in the reflection files, allowing "-y" + options to be omitted. - Ginn partiality model was added. - partialator: Switch to gradient-free minimiser. +- geoptimiser: Fixes for rotation and new features for AGIPD. +- Speed and accuracy improvements for TakeTwo indexing (Helen Ginn). - process_hkl, partialator and ambigator now include the audit information from the stream in their output files. - pattern_sim: Write orientation and lattice basis vectors into HDF5 file. - hdfsee: Add event ID and peaks/preds/rings to exported image filenames. - options to be omitted. - indexamajig --serial-start was added (useful when using turbo-index). -- indexamajig --overpredict was added (useful for partiality - modelling/post-refinement) +- indexamajig --overpredict was added (needed for post-refinement) - pattern_sim --flat was added (new reflection profile option) -- cell_explorer: Show number of matching cells on terminal. +- cell_explorer: Now shows the number of matching cells on terminal. - Felix options were simplified. - scripts/sum-hdf5-files and eiger-badmap were added diff --git a/Makefile.am b/Makefile.am index f5d38c2a..a669b1dd 100644 --- a/Makefile.am +++ b/Makefile.am @@ -29,7 +29,7 @@ TESTS = tests/list_check $(MERGE_CHECKS) $(PARTIAL_CHECKS) \ tests/prediction_gradient_check tests/scaling_check EXTRA_DIST += $(MERGE_CHECKS) $(PARTIAL_CHECKS) -EXTRA_DIST += relnotes-0.6.3 announcement-0.6.3 +EXTRA_DIST += relnotes-0.7.0 announcement-0.7.0 if BUILD_HDFSEE bin_PROGRAMS += src/hdfsee diff --git a/announcement-0.6.3 b/announcement-0.6.3 deleted file mode 100644 index db8d0adb..00000000 --- a/announcement-0.6.3 +++ /dev/null @@ -1,35 +0,0 @@ -Dear CrystFEL users and interested crystallographers, - -It's been a long time since the last CrystFEL release, but we have not been -resting. CrystFEL version 0.6.3 has been released. - -This version incorporates a large number of "backstage" improvements and -alterations. The most interesting changes for users are: - -- The incorporation of "peak finder 8" from Cheetah (also used in OnDA) - -- Support for images in CBF format - -- Incorporation of "TakeTwo" indexing algorithm - -- Hit finder functionality (skip frames with few peaks) - -- Better support for three-dimensional detector geometries - -See the release notes for more discussion of these improvements, and the -ChangeLog for full details of the other improvements. - -Thanks for all of your past and future feedback and contributions, and -your continued use of CrystFEL. As ever, please contact me directly if -you would prefer to be unsubscribed from this mailing list. - -CrystFEL website: -https://www.desy.de/~twhite/crystfel - -Release notes for version 0.6.3: -https://www.desy.de/~twhite/crystfel/relnotes-0.6.3 - -CrystFEL tutorial: -https://www.desy.de/~twhite/crystfel/tutorial - -Tom diff --git a/announcement-0.7.0 b/announcement-0.7.0 new file mode 100644 index 00000000..619dc570 --- /dev/null +++ b/announcement-0.7.0 @@ -0,0 +1,37 @@ +Dear CrystFEL users and interested crystallographers, + +CrystFEL version 0.7.0 has been released. + +This version adds several exciting new features while greatly simplifying the +user experience. The most interesting changes are: + +- New options system for indexamajig, removing most of the profusion of indexing + method flags "comb", "bad", "retry" and so on. Indexamajig can also now + automatically determine which indexing methods can be used if you don't tell + it which ones to use. In addition, the options for the Felix indexing method + have been simplified. + +- The spectrum-based partiality model and numerical post-refinement algorithm + from Ginn et al. [Acta D71 2015 p1400] has been incorporated. + +- The symmetry of the merged reflection list is now automatically tracked through + the subsequent data processing stages, so you no longer need to give the "-y" + option to compare_hkl, check_hkl and other programs. + +See the release notes for more discussion of these improvements, and the +ChangeLog for full details of the other improvements. + +Thanks for all of your past and future feedback and contributions, and +your continued use of CrystFEL. As ever, please contact me directly if +you would prefer to be unsubscribed from this mailing list. + +CrystFEL website: +https://www.desy.de/~twhite/crystfel + +Release notes for version 0.7.0: +https://www.desy.de/~twhite/crystfel/relnotes-0.7.0 + +CrystFEL tutorial: +https://www.desy.de/~twhite/crystfel/tutorial + +Tom diff --git a/configure.ac b/configure.ac index 30e803f7..ef003372 100644 --- a/configure.ac +++ b/configure.ac @@ -1,6 +1,6 @@ dnl Process this file with autoconf to produce a configure script. -AC_INIT([crystfel],[0.6.3],[taw@physics.org]) +AC_INIT([crystfel],[0.7.0],[taw@physics.org]) AC_CONFIG_AUX_DIR([build-aux]) AC_CONFIG_HEADERS([config.h]) AM_INIT_AUTOMAKE([subdir-objects]) diff --git a/doc/reference/libcrystfel/CrystFEL-docs.sgml b/doc/reference/libcrystfel/CrystFEL-docs.sgml index 048f4167..6a1dabee 100644 --- a/doc/reference/libcrystfel/CrystFEL-docs.sgml +++ b/doc/reference/libcrystfel/CrystFEL-docs.sgml @@ -8,7 +8,7 @@ CrystFEL Reference Manual - For libcrystfel from CrystFEL 0.6.3. + For libcrystfel from CrystFEL 0.7.0. This is the internal documentation for CrystFEL. Unless you are looking at diff --git a/libcrystfel/Makefile.am b/libcrystfel/Makefile.am index 24cb7358..1d24d353 100644 --- a/libcrystfel/Makefile.am +++ b/libcrystfel/Makefile.am @@ -1,6 +1,6 @@ lib_LTLIBRARIES = libcrystfel.la libcrystfel_la_LIBADD = ../lib/libgnu.la @LIBCRYSTFEL_LIBS@ $(PTY_LIB) -libcrystfel_la_LDFLAGS = -version-info 9:0:0 +libcrystfel_la_LDFLAGS = -version-info 10:0:0 libcrystfel_la_SOURCES = src/reflist.c src/utils.c src/cell.c src/detector.c \ src/thread-pool.c src/image.c src/hdf5-file.c \ diff --git a/relnotes-0.6.3 b/relnotes-0.6.3 deleted file mode 100644 index 11a1f93b..00000000 --- a/relnotes-0.6.3 +++ /dev/null @@ -1,211 +0,0 @@ -CrystFEL - Crystallography with a FEL -------------------------------------- - -Release notes for version 0.6.3 - -Copyright © 2012-2017 Deutsches Elektronen-Synchrotron DESY, - a research centre of the Helmholtz Association. - -Authors: - Thomas White - Richard Kirian - Kenneth Beyerlein - Andrew Aquila - Andrew Martin - Lorenzo Galli - Chun Hong Yoon - Kenneth Beyerlein - Karol Nass - Nadia Zatsepin - Anton Barty - Cornelius Gati - Fedor Chervinskii - Alexandra Tolstikova - Wolfgang Brehm - Valerio Mariani - Parker de Waal - Takanori Nakane - Keitaro Yamashita - Oleksandr Yefanov - Steve Aplin - Helen Ginn - -CrystFEL is free software: you can redistribute it and/or modify it under the -terms of the GNU General Public License as published by the Free Software -Foundation, either version 3 of the License, or (at your option) any later -version. - -CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY -WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A -PARTICULAR PURPOSE. See the GNU General Public License for more details. - -You should have received a copy of the GNU General Public License along with -CrystFEL. If not, see . - - -Overview --------- - -The most important new features in this version of CrystFEL are: - -- Support for CBF files as input - -- TakeTwo indexing algorithm - -- Improved peak detection and basic hitfinder functionality - -- Detector panels not perpendicular to X-ray beam, and "rail direction" - -These changes have sections below. In addition, there were many bug fixes and -other improvements. See the "ChangeLog" or the changes page on the CrystFEL -website for details. - - -Support for CBF files as input ------------------------------- - -Crystallographic Binary Files (CBF) can now be used as input for indexamajig -and hdfsee. To enable this functionality, you must have "CBFlib" installed. -Some distributions (eg Fedora) include CBFlib in their standard repositories - -simply install "cbflib-devel" or similar. Then compile and install CrystFEL as -usual. If CBFlib is installed somewhere non-standard, e.g. if you has to -install it manually, you will need to give the path with the "--with-cbflib-dir" -option to "./configure". - -After that, you can simply give the names of CBF files in the indexamajig input, -and everything should work as usual! There are some catches, for example you -cannot yet use a bad pixel mask with CBF files, but these will be fixed in a -future release. - - -TakeTwo indexing algorithm --------------------------- - -This version of CrystFEL includes the TakeTwo indexing algorithm by Helen Ginn. -TakeTwo is a new indexing algorithm, designed specifically for diffraction -snapshots and operating completely differently to conventional FFT-based methods. - -To use it, simply use "taketwo" as the indexing method, or add it to the list of -indexing methods you have already: - $ indexamajig --indexing=taketwo - $ indexamajig --indexing=mosflm,taketwo,dirax - -If you use TakeTwo, please take careful note of the citation reminder which will -be shown to you, and cite the following paper in any resulting publications: - Ginn et al., Acta Cryst. (2016). D72, 956-965 - -TakeTwo is considered experimental in this version of CrystFEL. We welcome any -feedback, and will use the information to improve its performance for future -versions. - - -Improved peak detection and basic hitfinder functionality ---------------------------------------------------------- - -"Peak finder 8" from Cheetah, also used in OnDA, has been incorporated into -CrystFEL. It uses a radially varying background level to find peaks, which -works very well with most diffraction data. - -Simply use "--peaks=peakfinder8", and set --threshold, --min-snr, ---min-pix-count and --local-bg-radius. See the manual for other options. - -Indexamajig can now skip over patterns which have a small number of peaks. To -use this, use the option "--min-peaks=N", where N is the minimum number of peaks -before indexamajig will proceed to indexing (and hopefully integration) with the -pattern. - -By combining the improved peak detection with the hitfinder functionality and -CBF reading, you can process data from several different types of detector, -without converting the files to HDF5 format or using any external program for -hit finding! - -If the hit rate is very low, you might want to make CrystFEL stream smaller by -excluding non-hits from it using "--no-non-hits-in-stream". Without this -option, standard information about the non-hits (e.g. photon energy and peaks -found) will be recorded. - - -Detector panels not perpendicular to X-ray beam, and "rail direction" ----------------------------------------------------------------------- - -CrystFEL now supports three-dimensional detector geometry. The "fs" and "ss" -vectors (which are the real-space directions of the fast scan and slow scan -directions in the image data) can now contain a z component as well as x and y. - -For many detectors, the "rail" direction, along which the detector moves when -the "camera length" (sample-detector distance) is adjusted, is not perpendicular -to the detector plane. The geometry file now allows you to specify this "rail" -direction, containing x, y and z components. See the crystfel_geometry manual -page for more details. - - -API changes ------------ - -The following changes have been made to the libcrystfel API. The biggest -changes are the switch from the old "hdfile" API (which is still supported), -to the new "imagefile" API, while allows multiple file types to be used. -The indexing system was also updated, and there were changes to the detector -geometry system, which no longer uses the "data slab" representation internally. - -New functions: - - crystal_{get,set}_det_shift() - - panel_number() - - get_detector_geometry_2() - - fill_in_adu() - - adjust_centering_for_rail() - - get_peaks_2() - - get_peaks_cxi_2() - - imagefile_open() - - imagefile_read() - - imagefile_read_simple() - - imagefile_get_hdfile() - - imagefile_get_type() - - imagefile_copy_fields() - - imagefile_close() - - {new,free}_imagefile_field_list() - - add_imagefile_field() - - get_indm_from_string() - - setup_indexing() - - index_pattern_2() - - search_peaks_peakfinder8() - - reflist_add_command_and_version() - - reflist_{add,get}_notes() - - write_chunk_2() - - pointgroup_warning() - -New structure definitions: - - struct image - - struct imagefile_field_list - -Removed functions: - - get_q() - - find_panel() - - find_panel_number() - - fill_in_values() - - partial_event_substitution() - - build_indexer_list() - - prepare_indexing() - -Changed function prototypes: - - in_bad_region() - - simple_geometry() - - reverse_2d_mapping() - - hdfile_set_image() - - image_add_feature() - - image_feature_closest() - - image_reflection_closest() - - index_pattern() - - cleanup_indexing() - - write_chunk() - -Changed structure definitions: - - struct panel: removed {min,max}_{fs,ss}, added adu_per_photon, - added fsz,ssz, rail_{x,y,z} and clen_for_centering - - struct detector: removed max_{fs,ss} - - struct imagefeature: replaced "pn" with "p" - - struct beam_params: "copyme" is not an imagefile_field_list - (previously: copy_hdf5_field) - - struct image: "copyme" is not an imagefile_field_list - (previously: copy_hdf5_field). width and height remoted. - diff --git a/relnotes-0.7.0 b/relnotes-0.7.0 new file mode 100644 index 00000000..186694a0 --- /dev/null +++ b/relnotes-0.7.0 @@ -0,0 +1,176 @@ +CrystFEL - Data processing for serial crystallography +----------------------------------------------------- + +Release notes for version 0.7.0 + +Copyright © 2012-2018 Deutsches Elektronen-Synchrotron DESY, + a research centre of the Helmholtz Association. + +Authors: + Thomas White + Richard Kirian + Kenneth Beyerlein + Andrew Aquila + Andrew Martin + Lorenzo Galli + Chun Hong Yoon + Karol Nass + Nadia Zatsepin + Anton Barty + Cornelius Gati + Fedor Chervinskii + Alexandra Tolstikova + Wolfgang Brehm + Valerio Mariani + Parker de Waal + Takanori Nakane + Keitaro Yamashita + Oleksandr Yefanov + Steve Aplin + Helen Ginn + Thomas Grant + Mamoru Suzuki + +CrystFEL is free software: you can redistribute it and/or modify it under the +terms of the GNU General Public License as published by the Free Software +Foundation, either version 3 of the License, or (at your option) any later +version. + +CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A +PARTICULAR PURPOSE. See the GNU General Public License for more details. + +You should have received a copy of the GNU General Public License along with +CrystFEL. If not, see . + + +Overview +-------- + +The most important new features in this version of CrystFEL are: + +- New options system for indexamajig, removing most of the profusion of indexing + method flags "comb", "bad", "retry" and so on. Indexamajig can also now + automatically determine which indexing methods can be used if you don't tell + it which ones to use. In addition, the options for the Felix indexing method + have now been greatly simplified. + +- The spectrum-based partiality model and numerical post-refinement algorithm + from Ginn et al [Acta D71 2015 p1400] was incorporated. + +- The symmetry of the merged reflection list is now tracked through automatically, + so you no longer need to give the "-y" option to compare_hkl and check_hkl. + +These changes have sections below. In addition, there were many bug fixes and +other improvements. See the "ChangeLog" or the changes page on the CrystFEL +website for details. + + +New indexamajig options system +------------------------------ + +The new version of indexamajig will automatically determine which indexing +methods to use based on the prior information (e.g. if a target unit cell was +provided) and which indexing programs are available on the computer. To enable +the automatic determination, simply omit any "--indexing=" option to indexamajig. +Of course, you can override the automatic choice by specifying --indexing, just +as it worked before. + +To options for finer control of the indexing process, which in previous versions +were enabled or disabled by adding flags to the indexing methods +(e.g. "mosflm-noretry"), now apply to all indexing methods at once and are +selected using new options, listed below. + +The indexing method should contain only the method itself and prior information +modifiers ('cell' or 'latt'). This simplifies things greatly by removing the +very long indexing method descriptions ("mosflm-raw-latt-noretry-refine-nomulti" +and so on). + +- To disable prediction refinement ('norefine'), use --no-refine. +- To check cell axes only ('axes'), use --no-cell-combinations. +- To disable all unit cell checks ('raw'), use --no-check-cell. +- To disable peak alignment check ('bad'), use --no-check-peaks. +- To disable indexing retry ('noretry'), use --no-retry. +- To enable multi-lattice indexing by 'delete and retry', use --multi. + + +Spectrum-based partiality model and post-refinement +--------------------------------------------------- + +This version includes an implementation of the spectrum-based partiality model +described by Ginn et al., Acta D71 2015 p1400 and associated numerical post- +refinement algorithm. To use it, use "partialator --model=xsphere". The old +models (scsphere and scgaussian) have been removed. You will probably need to +experiment with changing the bandwidth and reflection radius, using +--force-bandwidth and --force-radius. Add --no-pr to disable post-refinement, +while still using the partiality estimates. To use post-refinement, omit --no-pr, +but you should also run indexamajig with "--overpredict". You may also need to +disable B factor scaling in partialator with "--no-Bscale". + +If you use post-refinement, the new scripts "plot-pr" and "plot-contourmap" can +be used to visualise the distributions of partialities and the contour maps of +residual against parameters such as crystal rotation angle. The data for these +graphs is in the "pr-logs" folder which will be created by partialator. + +Partiality modelling and post-refinement should still be considered experimental +when applied to real data. The success of post-refinement depends on many +factors, and only has a chance of working if there are no other problems with +the dataset (e.g. inaccurate detector geometry). Further improvements should +appear in future versions. + + +Automatic tracking of reflection list symmetry +---------------------------------------------- + +The point group symmetry used for merging a reflection list has been stored in +the reflection file (.hkl) for a long time. However, version 0.7.0 of CrystFEL +uses the stored symmetry when performing tasks such as calculating figures of +merit using check_hkl. You can simply omit the "-y" options which were needed +previously. Of course, you can override the symmetry by specifying -y as before. + + +API changes +----------- + +The following changes have been made to the libcrystfel API. + +New functions: + - asdf_probe() and _probe for all other indexing methods. + - compare_cells() + - crystal_get_cell_const() + - crystal_get_profile_radius() + - predict_to_res() + - calculate_partialities() + - update_predictions() + - is_cbf_file() + - get_indm_from_string_2() + - detect_indexing_methods() + - integrate_all_5() + - read_reflections_2() + - {get,set}_{khalf,kpred,exerr}() + - first_refl_const(), next_refl_const() + - open_stream_for_write_4() + - stream_has_old_indexers() + - stream_audit_info() + +Removed functions: + - find_intersections() + - find_intersections_to_res() + - update_partilities() + - write_reflections_to_file() + - read_reflections_from_file() + - {get,sh}_partial() + +Changed function prototypes: + - asdf_prepare() and _prepare for all other indexing methods. + - cell_new_from_cell() (input UnitCell is now const) + - crystal_copy() (input Crystal is now const) + - x_gradient(), y_gradient() + - remove_flagged_crystals() (now returns number of crystals removed) + - setup_indexing() + +Changed structure/enum definitions: + - PartialityModel: removed PMODEL_SCSPHERE and PMODEL_SCGAUSSIAN, + added PMODEL_XSPHERE + - gparam: added GPARAM_ANG1, GPARAM_ANG2, GPARAM_WAVELENGTH, GPARAM_EOL + - Added IndexingFlags -- cgit v1.2.3