1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
|
.\"
.\" partialator man page
.\"
.\" Copyright © 2012-2021 Deutsches Elektronen-Synchrotron DESY,
.\" a research centre of the Helmholtz Association.
.\"
.\" Part of CrystFEL - crystallography with a FEL
.\"
.TH PARTIALATOR 1
.SH NAME
partialator \- scaling and post-refinement of partial reflections
.SH SYNOPSIS
.PP
.B partialator
\fB-i\fR \fIinput.stream\fR
\fB-o\fR \fIoutput.hkl\fR
\fB-y\fR \fIpointgroup\fR
[\fBoptions\fR] \fB...\fR
.PP
.B partialator
\fB--help\fR
.SH DESCRIPTION
\fBpartialator\fR merges reflections by scaling and post refinement, accounting
for the partialities of the reflections. That means that it models the geometry
of diffraction for each pattern (crystal orientation, unit cell parameters,
X-ray bandwidth and so on) and attempts to optimise the geometrical parameters
to make the fully integrated intensities calculated using the model agree as
closely as possible between the many patterns.
See \fBUSAGE CASES\fR below for examples of commands to merge reflections in different
ways, for example with and without partiality or scaling.
In addition to the output reflection list, \fBpartialator\fR will write a file called partialator.params which contains the scaling factors determined for each crystal. After each iteration, a file will be written called pgraph-iter\fIn\fR.dat which contains the observed and calculated partialities for a randomly chosen set of "free" reflections which were not included in the refinement.
\fBpartialator\fR will also write datasets merged using two halves of the total data using the same filename as you give for \fB-o\fR or \fB--output\fR, with \fB1\fR and \fB2\fR appended. For example, \fBpartialator.hkl1\fR and \fBpartialator.hkl2\fR. Compare these two files with each other to calculate figures of merit such as Rsplit and CC1/2. See the section on \fBCUSTOM DATASET SPLITTING\fR for more discussion on this topic.
.SH OPTIONS
.PD 0
.IP "\fB-i\fR \fIfilename\fR"
.IP \fB--input=\fR\fIfilename\fR
.PD
Give the name of the input stream.
.PD 0
.IP "\fB-o\fR \fIfilename\fR"
.IP \fB--output=\fR\fIfilename\fR
.PD
Give the name of the output file. The default is \fB--output=partialator.hkl\fR.
.PD 0
.IP "\fB-y\fR \fIpointgroup\fR"
.IP \fB--symmetry=\fR\fIpointgroup\fR
.PD
Merge according to symmetry \fIpointgroup\fR.
.PD 0
.IP "\fB-n\fR \fIn\fR"
.IP \fB--iterations=\fR\fIn\fR
.PD
Run \fIn\fR cycles of scaling and post refinement.
.PD 0
.IP \fB--no-scale\fR
.PD
Disable the scaling part of the refinement calculation.
.PD 0
.IP \fB--no-Bscale\fR
.PD
Disable the Debye-Waller part of the scaling calculation.
.PD 0
.IP \fB--no-pr\fR
.PD
Disable the orientation/physics model part of the refinement calculation.
.PD 0
.IP \fB--no-deltacchalf\fR
.PD
Disable rejection based on deltaCChalf.
.PD 0
.IP "\fB-m\fR \fImodel\fR"
.IP \fB--model=\fR\fImodel\fR
.PD
Specify the partiality model. See the list below for possible choices.
.PD 0
.IP "\fB-j\fR \fIn\fR"
.PD
Run \fIn\fR analyses in parallel.
.PD 0
.IP \fB--polarisation=\fItype\fR
.PD
Specify the polarisation of the incident radiation. \fItype\fR can be \fBhoriz\fR or \fBvert\fR to indicate 100% polarisation of the electric field in the horizontal plane or vertical plane respectively. Setting \fItype\fR to \fBnone\fR completely disables the polarisation correction (see the note below). Alternatively, \fItype\fR can be a direction followed by a percentage polarisation fraction. For example, \fB45deg90\fR means that 90% of the radiation is polarised with its electric field in a direction 45 degrees from horizontal, and \fB10deg100\fR means that all the radiation is polarised at 10 degrees from horizontal. The angle is specified clockwise from horizontal as viewed along the beam direction, i.e. as shown by the CrystFEL GUI. The beam is unpolarised when the fraction is 50% (equal parts of the radiation have their electric field in the specified plane). If the polarisation fraction is 100%, it can be omitted. For example \fB10deg\fR or \fBhoriz\fR.
Note that \fB--polarisation=none\fR is not the same as, for example, \fB--polarisation=vert50\fR. In the first case, the polarisation correction will be completely disabled. In the other case, the incident beam will be unpolarised, but the polarisation of the diffracted radiation will still be corrected for (the factor of (1+cos^2(2theta))/2 or, equivalently, (2-sin^2(2theta))/2).
The default is \fB--polarisation=horiz\fR.
.PD 0
.IP \fB--no-polarisation\fR
.PD
Synonym for \fB--polarisation=none\fR.
.PD 0
.IP \fB--max-adu=\fR\fIn\fR
.PD
Include reflections only if their peak values were less than \fIn\fR. That means, \fIn\fR is the saturation value of the detector. The default is infinity, i.e. no cutoff.
.PD 0
.IP \fB--min-res=\fR\fIn\fR
.PD
Merge crystals only if they diffract to beyond \fIn\fR Angstroms resolution. The default is infinity, i.e. all crystals are included. The resolution is taken from the diffraction_resolution_limit line in the stream.
.PD 0
.IP \fB--min-measurements=\fR\fIn\fR
.PD
Include a reflection in the output only if it appears at least least \fIn\fR times. The default is \fB--min-measurements=2\fR.
.PD 0
.IP \fB--push-res=\fIn\fR
.PD
Merge reflections which are up to \fIn\fR nm^-1 higher than the apparent resolution limit of each individual crystal. \fIn\fR can be negative to merge \fIlower\fR than the apparent resolution limit. The default is \fB--push-res=inf\fR, which means no resolution cutoff at all.
.PD 0
.IP \fB--start-after=\fR\fIn\fR
.PD
Ignore the first \fIn\fR crystals in the input. The default is \fB--start-after=0\fR, i.e. start at the beginning.
.PD 0
.IP \fB--stop-after=\fR\fIn\fR
.PD
Stop processing after \fIn\fR crystals have been successfully read. The default is \fB--stop-after=0\fR, which means to process all the patterns from the start point to the end of the input (see \fB--start-after\fR).
.PD 0
.IP \fB--no-free\fR
.PD
Disable cross-validation (for testing only).
.PD 0
.IP \fB--custom-split=\fIfilename\fR
.PD
Read a list of filenames, event IDs and dataset IDs from \fIfilename\fR. See the section on \fBCUSTOM DATASET SPLITTING\fR below.
.PD 0
.IP \fB--max-rel-B=\fIn\fR
.PD
Reject crystals if the absolute values of their relative Debye-Waller ("B") factors are more than \fIn\fR square Angstroms. The default is \fB--max-rel-B=100\fR.
.PD 0
.IP \fB--output-every-cycle\fR
.PD
Write out the per-crystal parameters and reflection lists after every cycle of refinement, instead of only at the end. The intermediate reflection lists and parameter filenames will be prefixed with \fBiter\fIN\fB_\fR (note the trailing underscore), where \fIN\fR is the iteration number. If you use \fB--custom-split\fR, intermediate results will also be output for each custom dataset.
.PD 0
.IP \fB--no-logs\fR
.PD
Do not write the extensive log files needed for plotting contour maps and spectrum graphs. This makes the process a lot faster, but you probably do want these logs to check that post-refinement is working reasonably.
.PD 0
.IP \fB--log-folder=\fIfolder\fR
.PD
Specify the location of the log folder (see \fB--no-logs\fR). The default is \fB--log-folder=pr-logs\fR.
.PD 0
.IP "\fB-w\fR \fIpg\fR"
.PD
Set the apparent symmetry of the crystals. The ambiguity operator will be determined by comparing this to the actual symmetry.
.IP
If you prefer (or the scenario demands it), you can specify the ambiguity operator directly using \fB--operator\fR.
.PD 0
.IP \fB--operator=\fR\fIop\fR
.PD
Specify the indexing ambiguity operator. Example: \fB--operator=k,h,-l\fR.
.IP
If you prefer, you can specify the ambiguity operator by specifying the apparent symmetry using \fB-w\fR.
.PD 0
.IP \fB--force-bandwidth=\fIbw\fR
.IP \fB--force-radius=\fIR\fR
.IP \fB--force-lambda=\fIl\fR
.PD
Set the X-ray bandwidth, initial profile radius or wavelength for all crystals before proceeding, overriding the values from the stream. Bandwidth is given as a fraction, i.e. \fB--force-bandwidth=0.0013\fR means 0.13 percent (approximate FWHM). Radius is given in nm^-1. Wavelength is given in Angstroms.
.PD 0
.IP \fB--harvest-file=\fIfn\fR
.PD
Write a list of parameters to \fIfn\fR, in JSON format. This is intended to be used for harvesting data into a database system.
.SH PARTIALITY MODELS
The available partiality models are:
.IP \fBxsphere\fR
.PD
The volume of intersection between a sphere centered on each reciprocal lattice point and the part of reciprocal space excited by the Ewald sphere taking into account the finite bandwidth of the incident X-rays. A "source coverage factor" is included to take into account the spectral brightness of the effective source for the reflection. The X-ray spectrum is modelled as a super-Gaussian with exponent 1.5, and the overlap integral is evaluated numerically.
This model is the same as that described in Acta Cryst. D71 (2015) p1400.
.IP \fBunity\fR
.PD
Fix all partialities at 1.
.IP \fBoffset\fR
.PD
Treat the radiation as perfectly monochromatic, and calculate the partiality as exp(-t^2/R^2), where t is the excitation error and R is the reflection radius (taking mosaicity into account). This model is similar to that used by XDS.
.IP \fBggpm\fR
.PD
An analytical overlap integral modelling the reflection profiles as Gaussian functions, and the radiation spectrum as a sum of Gaussians.
.SH USAGE CASES
.IP "Merging without scaling, partialities or post-refinement:"
.PD
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=unity --iterations=0\fR
.IP "Merging without partialities or post-refinement, but with scaling:"
.PD
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=unity --iterations=1\fR
.IP
(Use a higher number of iterations to increase the accuracy of scaling, but at a cost of more CPU time and possibly more rejected crystals)
.IP "Merging with partialities, but without post-refinement and without scaling:"
.PD
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=xsphere --iterations=0\fR
.IP "Merging with partialities, with scaling but without post-refinement:"
.PD
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=xsphere --iterations=1 --no-pr\fR
.IP "Merging with partialities, post-refinement and scaling:"
.PD
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=xsphere --iterations=1\fR
.IP
(Use a higher number of iterations to increase the accuracy of scaling and post-refinement, but at a cost of more CPU time and possibly more rejected crystals)
.IP "Merging with partialities and post-refinement, but without scaling:"
.PD
This would be a strange thing to want to do, however:
.IP
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=xsphere --iterations=1 --no-scale\fR
.IP
(Use a higher number of iterations to increase the accuracy of post-refinement, but at a cost of more CPU time and possibly more rejected crystals)
.SH CUSTOM DATASET SPLITTING
When performing a time-resolved experiment (for example), it is preferable to ensure that the data for all time points has been processed identically. Rather than processing each time point independently with separate runs of partialator, it is better to process them all together and do the splitting into time points just before the final output. Consider, for example, the case of simple scaling (without a B factor): when merging independently, the resulting datasets would probably end up with different overall scaling factors. When comparing the results, you would need to take this difference into account. In practice, most programs can do that job easily, but what about if a B factor is included? And what if partialities are included - how unique is the solution?
With \fBpartialator --custom-split\fR, you can provide a separate text file containing a list of filenames, event numbers and \fIdataset names\fR, one event (detector frame) per line, with each field separated by exactly one space. For each unique \fIdataset name\fR, a separate reflection list will be output. All crystals will be refined together, but they will be merged according to the dataset names you give. The parameters (scaling factors, partialities etc) determined during the joint refinement will be applied. For each dataset, a separate pair of split half-datasets will also be written, allowing you to calculate figures of merit such as Rsplit and CC1/2 for each one.
If the overall output filename (given with \fB-o\fR or \fB--output\fR) were \fBmerged.hkl\fR, then a dataset named \fIdataset\fR would be written to \fBmerged-\fIdataset\fB.hkl\fR. The corresponding half-datasets would be written to \fBmerged-\fIdataset\fB.hkl1\fR and \fBmerged-\fIdataset\fB.hkl2\fR.
Note that the filenames and event names must match \fBexactly\fR what is written into the stream as the \fBImage filename\fR and \fBEvent\fR, taking into account options such as \fBindexamajig --prefix\fR and \fB--basename\fR. You should therefore check that the numbers of crystals in each dataset, which will be written on the terminal by partialator, match your expectations and that no patterns have been "lost". There is no requirement for every event in the list to appear in the stream, nor for every event in the stream to belong to one of the datasets. If an event is listed for more than one dataset, the results are "undefined".
If you do not have event IDs for your data, i.e. if you have one detector frame per input file, simply leave out the event IDs from the custom split file.
Finally, note that the main and all custom split datasets, and also all the half-datasets, are subject to --min-measurements.
.SH AUTHOR
This page was written by Thomas White.
.SH REPORTING BUGS
Report bugs to <taw@physics.org>, or visit <http://www.desy.de/~twhite/crystfel>.
.SH COPYRIGHT AND DISCLAIMER
Copyright © 2012-2021 Deutsches Elektronen-Synchrotron DESY, a research centre of the Helmholtz Association.
.P
partialator, and this manual, are part of CrystFEL.
.P
CrystFEL is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
.P
CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
.P
You should have received a copy of the GNU General Public License along with CrystFEL. If not, see <http://www.gnu.org/licenses/>.
.SH SEE ALSO
.BR crystfel (7),
.BR indexamajig (1),
.BR process_hkl (1)
|