2.1. Selection of compression schemes

The topic of data compression is generally a vast and well researched area with many algorithms available for different types of data. Detailed reviews of mostly lossless methods applied to different scientific datasets can be found in the literature (Delaunay et al., 2019; Duwe et al., 2020). Here, we focus on data-compression methods applicable to or available in the context of serial crystallography.

Compression schemes typically reduce data volume by exploiting symmetries or redundancies in the data. Thus, the best type of compression to use for a given application depends on the nature of the data being compressed and the information deemed important to retain. Identifying a universal lossy compression scheme is thus difficult since the choice depends on what information must be retained and what can be discarded. For example, high-quality JPEG-type compression is widely used in medical imaging (Wiseman & Fredj, 2001), and compression factors of 6–8 can be obtained without affecting the diagnosis using conventional JPEG 2000 (Rabbani & Joshi, 2002; Marone et al., 2020) and JPEG XR (https://jpeg.org/jpegxr/index.html) compression methods. The appearance of compression artifacts after applying lossy compression is not so important here since JPEG is designed to preserve common visual features of such images. However, JPEG compression is poor for diffraction pattern analysis because the intensity of scientific interest is concentrated in localized Bragg peaks rather than image-like features. In X-ray tomography, azimuthal regrouping of input images followed by CBF byte-offset compression has proven its effectiveness (Kieffer et al., 2018), thereby exploiting the known rotational properties of a tomographic dataset. However, this assumption often cannot be made for other types of data.

In this paper, we focus our attention on the following compression schemes suitable to reduce the volume of SX data. Specifically, we consider the following.

(1) Lossless compression algorithms commonly available for the HDF5 library including gzip, bzip2, zstd and lz4 with and without bit-shuffle, and different combinations of Blosc (lz4, lz4hc, blosclz, snappy, zlib and zstd).

(2) Data-reduction schemes including non-hit rejection (saving only the diffraction patterns with detectable crystal diffraction), measuring less data (reduced number of frames) and saving only found peaks in diffraction patterns.

(3) Lossy compression schemes including binning (effectively increasing the pixel size and reducing the number of pixels at the detector), quantization (saving only several discrete levels of intensity) and quantization with data rearrangement (saving the intensity value of each pixel re-arranged as a single byte).

We study the effect of each compression type on resultant data and structural model quality obtained with test SFX datasets.

2.2. Selection of appropriate data quality metrics

Accurately assessing the potential loss of scientific content resulting from the application of lossy data compression necessitates a comprehensive set of metrics specifically designed to measure the extent of information loss. It is necessary to quantify not only whether the quality of the final molecular structure is affected, but also whether the ability to perform any of the many intermediate analysis steps, for example, peak finding or estimating background signal, is compromised due to loss of data quality.

The determination of macromolecular structures from X-ray diffraction patterns is by now a mature and well developed topic with accepted data quality metrics applied at various stages along the data-processing pipeline (Karplus & Diederichs, 2012, 2015; Assmann et al., 2016). Therefore, we use the following data quality metrics and guides already established for SX and conventional macromolecular crystallography (MX) data processing:

(1) Measured data quality and reproducibility: I/σ(I), completeness, R_split, CC* or CC_1/2.

(2) Model refinement quality (R values): R_free/R_work.

(3) Visual inspection of the reconstructed electron density maps.

(4) The strength of the anomalous diffraction signal and the ability to perform de novo phasing and/or structure refinement.

The first set of metrics evaluates the quality and reproducibility of the merged data and is usually plotted as a function of the resolution (Karplus & Diederichs, 2012, 2015). These metrics are also used for selecting an optimal high-resolution cut-off. One important consideration to use these metrics: the resolution of the measured dataset should not be limited by the geometry of the experiment. Although these metrics can offer insights into data quality when structure refinement is unavailable, note that they alone cannot guarantee the successful reconstruction of electron density or the accuracy of any resulting structure (Diederichs & Karplus, 1997).

The quality of the derived structural maps is of paramount importance for interpreting protein structures. Therefore, it is important to compare how well the derived structure matches the experimental data. For this purpose, the model-refinement R value (the second metric from the aforementioned list) measures the agreement between the observed and calculated structure factor moduli (Brünger, 1992; Karplus & Diederichs, 2012). Going one step further, protein crystallographers introduced the R_free metric which measures how well a subset of data omitted from the refinement process is explained by the final model to indicate possible over-fitting or incorrect results.

Ultimately, the quality of the reconstructed map is interpreted by an expert. Here, it is occasionally observed that refinement quality assessed by R values alone can lead to incorrect conclusions about the model quality. For example, expert inspection of the structure may reveal nonphysical chemistry such as overlaps of atoms or bonds, or conversely the presence or absence of extra electron density such as structural solvent which must be included in the structural model to further improve agreement with the experiment. Thus, the third method of visually inspecting the quality of the reconstructed electron density is vital. Unfortunately, it is not so easy to automate this method, and it requires an expert to decide if the electron density looks reasonable.

Therefore, we turn to an additional method based on the strength of the anomalous signal and the ability to perform ab initio reconstruction of the structure as an effective method for determining whether information has been lost during data manipulation. The anomalous signal in single/multi-wavelength anomalous diffraction (SAD/MAD) datasets is usually weak, and the method of de novo phasing can work only if the error in the determination of the structure factors is lower than the Bijvoet differences. This method of structure determination is known to require a higher data quality than, for example, molecular replacement, making it a good test of whether lossy compression has adversely affected the data. To further increase the sensitivity of this approach, we start with a subset of the data that is only large enough for the de novo phasing pipeline to work, so that any meaningful deterioration in data quality will lead to an inability to reconstruct the structure (Nass et al., 2020).

We have used all four quality checks to test different lossy compression algorithms. Some applications, such as binding screening studies, may be more robust to small data errors and suffice if R_free/R_work are exceeded (metric 2), whereas cutting-edge cases such as de novo structure determination of sensitive time-resolved structural studies may require a higher data fidelity such as those measured by anomalous differences (metric 4). We further hope this suggested set of metrics and guides will be used to compare other compression schemes not studied here, ideally using the same open datasets.

2.3. Selection of test datasets

An ideal protein crystal diffraction pattern measured using a noiseless detector would be very sparse, consisting of bright Bragg peaks with zero background. Such a diffraction pattern is easily compressible by most existing lossless compression algorithms. By contrast, in real crystallography experiments the background recorded in each diffraction pattern is quite high and is often comparable to the strength of the measured Bragg peaks. Statistical noise in the background leads to significant intensity differences between neighboring pixels. Furthermore, the integrating detectors used at XFELs do not count incoming photons but rather accumulate the charge deposited in a single femtosecond-duration exposure. Accumulated charge including intrinsic electronic noise sources is not directly converted to individual photon counts but estimated after the detector is read out. Experience shows that compression of experimental SX diffraction data rarely reaches a compression factor better than 5 using lossless algorithms.

To capture these challenges, we selected four representative SX datasets covering a range of detector technologies (both counting and integrating), photon sources (including synchrotron and FELs) and varied sample-delivery methods. These methods include tape drive (Henkel et al., 2023; Zielinski et al., 2022), lipidic cubic phase (LCP) jet (Weierstall et al., 2014) and gas dynamic virtual nozzle (GVDN) jet (DePonte et al., 2008). The dataset specifics are enumerated in Table 1. Notably, three of the datasets listed are available for download from the CXIDB (Maia, 2012), and the associated code, deposited in GitHub (https://github.com/galchenm/binningANDcompression.git), facilitates the examination of all compression methods detailed in this study. The selection of tested samples (Table 1) is purposeful, aiming to evaluate diverse algorithms across different unit cells. Detailed unit-cell parameters and space groups can be found in Table S1 of the supporting information.

Table 1 Datasets used for the different tests Facility (beamline) Energy of X-rays (keV) Detector Data type Sample delivery Background level Samples PDB entry, CXIDB ID Reference Petra III (P11) 12 EIGER2 16M (counting, photons) Integer Tape drive Low Lysozyme, lactamase, ferritin, MPro – Unpublished SwissFEL (Alvra) 4.57 JUNGFRAU 16M (integrating, photons) Integers LCP High Thaumatin 6s19, CXIDB 104 (runs 12–41) Nass et al. (2020) LCLS (CXI) 9.4 CS-PAD 2.3M (integrating, ADUs) Integers GDVN Moderate Lysozyme 4et8, CXIDB 17 (runs 305–396) Boutet et al. (2012) EuXFEL (SPB) 9.3 AGIPD 1M (integrating, ADUs) Floating point GDVN Moderate Lysozyme, granulovirus 6ftr, CXIDB 98 (runs 96–98) Yefanov et al. (2019)

These test cases cover a representative sample of current protein crystallography datasets, including cryo-crystallography, where the background is high and a counting detector is usually used (similar to the second test dataset). In our discussion, we have selected comparisons between datasets that we found to best illustrate the challenges posted for different algorithms based on our practical experience working with different compression schemes and datasets. This avoids the combinatorial explosion of testing all algorithms against all datasets, enabling us to focus our attention on the key issues rather than presenting large tables of exhaustive comparisons.

3.1. Lossless compression

Lossless compression approaches are commonly used for compressing scientific data. By definition, no information is lost and the original data can be restored verbatim. The only question is therefore the achievable compression rate (CR) and speed, which are both highly dependent on the statistical properties of the data to be compressed.

We observe that lossless compression schemes vary significantly in effectiveness depending on the experimental SX data to which they are applied. From our sample datasets, two extreme cases for lossless compression were the EIGER 16M detector and the AGIPD. The EIGER 16M is a counting detector that registers zero counts without incident photons and integer values corresponding to the number of photons incident on a pixel. On the other hand, the AGIPD is an integrating detector with three different gain stages for which calibrated data are stored in a floating-point format, and the value in a pixel might be even negative due to the subtraction of non-constant dark signal. For integer data (EIGER 16M, CS-PAD or AGIPD rounded to integer values), compression ratios of higher than 4 can be achieved using zstd or bzip2, and in some cases can be higher than 10. Commonly available in HDF5, gzip compression (level 6) demonstrates quite good results. Conversely, the achievable compression ratio for floating point type data (AGIPD detector) reaches only 1.3 with gzip level 6.

A complete table with the results of the lossless compression algorithms tested against our reference datasets can be found in Table S2. We observe that the conversion to photons (integers) is an important factor in determining compression efficiency even if this conversion is itself a form of lossy compression, as expected.

Another important parameter for lossless compression is speed, which is especially critical for online data processing and real time compression (Fig. S1 of the supporting information). The tests were performed using blocks of 1000 frames from the two extreme cases above: EIGER 16M and AGIPD 1M detectors. The results indicate that most of the tested compression algorithms demonstrated similar performances, with the exception of bzip2, which usually produces the best compression ratio. But the best compromise of the compression/decompression speed versus compression ratio was observed for the blosc, zstd and bit-shuffle algorithms. For more information, see Fig. S1.

3.2. Rejection of non-hits (vetoing)

In serial crystallography, it is common that not all detector frames contain crystal diffraction. This is due to the sample being passed across the X-ray beam and intersecting with the beam at random. Whether any individual detector frame contains diffraction is a matter of chance. The hit rate, namely the fraction of measured frames containing useful crystal diffraction, is one of the important characteristics of an SX experiment and is related to the sample concentration, flow rate and relative size of the X-ray beam, among other factors. In practice, the hit rate during experiments is frequently rather low, typically ranging between 0.1 and 10%. This observation leads to the obvious conclusion that the volume of data can be reduced by discarding data frames without any crystal diffraction.

The urgent question is how to save collected data: do we need to store only frames containing obvious crystal diffraction, or should all the data be retained? Could we expect that more advanced data-processing algorithms could help to obtain better results in the future? To answer these questions, we re-processed the first high-resolution SFX dataset measured in 2011 (Boutet et al., 2012) using a recent SX processing pipeline (CrystFEL version 0.10.1 and Phenix version 1.13). This reprocessing was performed in three different ways.

(1) Using the data as deposited at the CXIDB for the original paper (third row in Table 1).

(2) Repeating processing from raw data archived to tape for all measured data using modern detector calibration and hit-finding methods applied to all data frames and re-processed with the new algorithms.

(3) Repeating processing as described in (2), but using only the same data frames as deposited for the original paper.

The structure refinement of reprocessed data were performed with phenix.refine (Phenix version 1.13) with parameters such as `xray_data.high_resolution = 1.6' and `xray_data.low_resolution = 20' using PDB entry 6ftr (Wiedorn et al., 2018) as the search model. The results are summarized in Table 2.

From Table 2 we can see that the original paper reported lower-quality results than after any re-processing of the data performed now. Note that the results in Table 2 are limited by the geometry of the experiment – the resolution of 1.49 Å corresponds to the corners of the detector (Fig. S2). Determining the resolution cut-off point is primarily a subjective process. However, we consider all the quality metrics mentioned above in order to reach a reasonable compromise. The reconstructed electron density and the structural model between the original and the re-processed data are shown in Fig. 1 (additional examples can be found in Fig. S3). One example given (Asp52) is an active-site residue essential for the enzyme mechanism of lysozyme (Held & Van Smaalen, 2014). Re-processing data results in electron density maps that allow us to resolve alternative conformations of Asp52, which allows more accurate interpretation of biological function. The quality improvement is not surprising since Boutet et al. (2012) reported the first-ever high-resolution SFX structure, and data-processing software has significantly improved in the following years. There are several reasons for the improvement in result quality: new algorithms for indexing (including indexing multiple crystals per pattern) and integration implemented in CrystFEL (White et al., 2016), better knowledge of the detector geometry (Yefanov et al., 2015) and a different strategy for background subtraction (see Fig. S4). Note that the re-processing of the deposited MTZ file (PDB entry 4et8) using a more recent version of Phenix did not result in substantial improvements in the reconstructed structure: published R_free/R_work = 0.229/0.196 versus Phenix (version 1.20) R_free/R_work = 0.2109/0.1730 (additional comparison of the results achieved using different versions of Phenix are presented in Table S3). Based on this observation, we can conclude that the significant improvement observed in the reconstructed structure can be attributed to the processing of actual diffraction frames.

Figure 1 Comparison of electron density maps (2Fo − Fc, contour level σ = 1.5) of lysozyme in the original structure (PDB entry 4et8; 1.90 Å yellow mesh and model) and the reprocessed data using all frames (1.49 Å, blue mesh and model). (a) and (b) Active-site residue Asp52 could be modeled with an alternative conformation using the reprocessed data. (c) and (d) Another section of the structure around Tyr23 with the same maps as described above (but with contour level σ = 0.8) shows more detailed density for the aromatic amino acids when using the reprocessed data.

An important observation is that an improved identification of frames containing crystal diffraction (the `hit finding' step) did not lead to a significant improvement in data quality. This is because hits are identified based on the simple metric of the presence of Bragg peaks. Most of the additional patterns found by repeating hit finding on all raw data only served to identify frames containing weak diffraction corresponding to small crystals or crystals hit by the tail of the X-ray beam. Weak patterns provide less information than the strong patterns found in the initial analysis, especially at high resolution, thus contributing little to improving the quality of the refined structure.

As a check, we reprocessed several other datasets from the same beam time as well as from some other LCLS beam times (not reported in detail here). Comparison with the originally published datasets always demonstrated that new processing would lead to significantly better results: we have observed the improvement of the achievable resolution by up to 1 Å: from 3.5 to 2.5 Å in the case of a protein with a large unit cell (article in preparation). However, repeating the selection of data frames to process (hit finding) usually resulted in marginal improvement, if any.

We therefore conclude that for strongly diffracting crystals it is indeed a good compromise to save only frames with clear diffraction peaks but to save these data in the `raw' data format so that improvements in detector calibration can be applied from the raw data later.

For the case of weakly diffracting crystals, the situation is more complicated because there is always the potential that weakly diffracting frames may not be found (Ayyer et al., 2015) and additional information may be identified in the diffraction patterns after the experiment, such as diffuse scattering outside of Bragg peaks (Ayyer et al., 2016). The decision as to whether this justifies the retention of all weakly diffracting data is one that each experiment team will have to make.

3.3. Measuring less data

Measuring more data frames in an SX experiment generally leads to higher-quality results due to averaging a greater number of observations. But an associated question during any experiment is: when have sufficient data been collected to answer the scientific question at hand? Measuring only enough data to answer a scientific question reduces experiment time and minimizes the amount of data collected to only the amount needed. Indeed, one of the common questions during SX beam time is when to stop data collection. In order to address this question, it is necessary to assess the impact of reducing the number of measured frames (Tolstikova et al., 2019; Zielinski et al., 2022).

The effect of measuring fewer data frames can readily be checked for any single dataset by integrating progressively smaller data subsets. Fig. 2 shows the quality metrics CC* and R_split versus resolution as well as R_free/R_work metrics for the different subsets of lactamase data measured during one of the tape-drive (Beyerlein et al., 2017; Henkel et al., 2023; Zielinski et al., 2022) SX experiments at the P11 beamline of PETRA3. The initial dataset consists of 200 000 diffraction patterns and is processed as smaller subsets equivalent to less measurement time (Table 3). This dataset was collected as a single 25 min acquisition with an EIGER2 X 16M detector operated at 133 Hz.

Figure 2 Data quality metrics CC* (from 0 to 1, higher is better) and Rsplit (up to 100%, lower is better) for the different fractions of the measured dataset of lactamase (first row in Table 1). The insets show the histogram of achievable resolution for each pattern.

The degradation in data quality with fewer patterns is rather obvious from Fig. 2 and is to be expected given the fact that redundant measurements improve statistical metrics such as CC*/R_split and thus the quality of the obtained data. We can conclude that while the strategy of limiting measurement time is understandable, measuring more data always improves data quality in line with known statistics. Therefore, the lossy reduction idea to save space just by measuring fewer data is, in fact, not optimal because it results in lower quality (see Fig. 2). On the other hand, the improvement of the resolution achievable using 1563 patterns (1/128) versus all 200 000 patterns is from 1.8 to 1.58 Å (0.22 Å difference). Therefore, the decision to halt data acquisition should be made based on the specific scientific inquiries of the study.

For the evaluation of data quality corresponding to the number of indexed lattices, the stream after ambigator (White et al., 2016) was split randomly at 1/4, 1/8, 1/16, 1/32, 1/64 and 1/128 and then subjected to scaling and merging. Phenix (Adams et al., 2010) (`phenix.reflection_file_editor') was used to add the same set of R_free-flags to each resulting dataset, and all datasets were refined with phenix.refine, using the same starting model, parameters and resolution cut-off (as set by the highest-resolution shell still containing useful data for the 1/128 dataset). Polygon (Urzhumtseva et al., 2009), MolProbity (Davis et al., 2007) and Coot were used for valid­ation of the final model.

3.4. Storing only detectable Bragg peaks

Another proposed data-reduction scheme is to save only the information around peaks found in each measured diffraction pattern. The idea is that only Bragg peak information affects the structure, so it should only be necessary to save information around the Bragg peaks.

Fig. 2 shows that adopting such a strategy will limit the achievable resolution. For this dataset, if we limit ourselves to only using the found peaks, the achievable per pattern resolution would reach 2 Å (5 nm⁻¹) for only a very small number of patterns (see the resolution histogram in the inset), while the entire dataset achieves a resolution of 1.58 Å according to CC* cut-off decision. It is well known that redundant measurement of weak data improves the overall resolution achieved beyond the resolution, at which peaks can be detected before integrating (Gati et al., 2017). As a consequence, compression schemes based on saving full detector data only around detectable peaks (Underwood et al., 2023) will artificially limit the resolution. For example, by processing the stream file from Table 2 to include only reflections from each pattern found in the initial peak search, the resulting resolution dropped to 1.62 Å and R_work/R_free increased to 0.236/0.292. In other words, we conclude that retaining data from only detected peaks noticeably decreases the structure quality.

3.5. Binning to lower the number of detector pixels

Reducing the pixel count by binning data to fewer pixels is commonly used when it is known that the detector has a finer pixel pitch than is strictly necessary for the current measurements. For example, when the beamline is equipped with a 16M detector but a 4M detector would suffice for the experiment, it is possible to bin the data after measurement. Alternatively, if the detector is capable of selectively recording a specific region of interest (ROI), and the beamline geometry permits moving the detector closer to the sample, it becomes feasible to save solely the designated small ROI. In both cases the separation between Bragg peaks has to remain adequate for data processing and the shape of the Bragg peaks need not be resolved. For experiments with a monochromatic X-ray beam, such minimum distance should be on the order of 5–10 pixels. Therefore, many datasets, especially for proteins with small unit-cell parameters, can be binned.

Binned data (Table S4 and Fig. S5) indicate that 2 × 2 pixel binning for the tested datasets measured with 16M detector did not degrade the data quality for the samples we used, but the data volume was reduced by a factor of 3–4. The one caveat is that it might be more difficult to detect the peaks after binning; therefore, we have developed a procedure in which the positions of the peaks are found before binning, recalculated into the coordinates of the binned image and saved within the output HDF5 file. Those saved positions can be later used for indexing.

3.6. Quantization of detector output

The quantization of data refers to the reduction of the bit depth of the saved data to a lower number of discrete values, reducing unnecessary precision in the stored values such as remapping 32 bit integers to 16 or 8 bit values, or converting floating point data to integers. In photon science, a common form of quantization is the conversion of the electrical signal to photon counts. This is performed in the electronics of counting detectors (PILATUS, EIGER), where each pixel directly counts the number of incident photons at high speed. As previously noted, such data compresses well using lossless compression schemes compared with data saved in floating point format. And, in general, data with fewer discrete values compress better using most of the lossless compression schemes.

Since counting detectors are not suited for the short pulse lengths found at XFEL sources, integrating detectors that integrate the deposited charge in each pixel during the exposure are used. Converting the deposited charge into the number of incident photons helps to reduce the data precision required; however, this operation relies on good calibration of detectors and is not necessarily a trivial task, there is a tendency to save actual digitizer readout for later photon conversion.

Our tests on quantization indicate that reducing the data precision of integrating detectors can be highly effective at enabling data compression. Results are presented in Table S5, where we test not only conversion to photons but also more aggressive reduction of data precision. For the AGIPD detector, even a quite high quantization level [1024 analog-to-digital units (ADUs) per quantum, which corresponds to approximately 14 photons at 9 keV] still achieves reasonably good data quality: R_free/R_work of 0.1753/0.1543 with a compression ratio of 64, compared with R_free/R_work of 0.1670/0.1497 for the original data.

To assess the impact of quantization in comparison with the previously outlined strategy of collecting a reduced dataset, we conducted the following test: quantization to 64 and 1024 ADUs was applied for both the entire dataset (comprising 190 000 diffraction patterns) and its 1/16 fraction (refer to Table S6 and Fig. S6). Although the volume of the 1/16 subset was nearly equivalent to the volume of the data rounded to 1024 ADUs, the data quality was superior in the latter dataset. This is evident in the higher achievable resolution and improved statistics even at lower resolutions (see Fig. S6). The findings demonstrate that prioritizing the acquisition of a sufficient number of patterns is more important than precise recording of the diffracted intensities.

3.7. Non-uniform quantization

An even higher compression ratio can be achieved by selecting the levels for quantization in a non-uniform way. Diffraction from crystals usually consists of some background (typically smooth) and rather sharp Bragg peaks. As mentioned earlier, a high dynamic range is usually needed to record such diffraction, with the intensity of the Bragg peaks varying from rather high (at low resolution) to very low (at high resolution). However, though single photon counting may be useful in weak reflections, it is not necessarily needed in the bright Bragg peaks. For this reason, special X-ray detectors (AGIPD, JUNGFRAU, ePIX) were developed that have variable gains per pixel to be able to record single photons at low flux as well as very high intensities (up to 10 000 photons per pixel) at high flux in a single image.

The tolerable relative error of the peak intensity drives the required precision. Thus, at low photon counts the quantization levels are rather dense, while at the higher fluxes, the levels are comparatively sparse, in proportion to the counting noise. In the measured data this can be achieved by keeping the value of just a few of the most significant bits (starting from the first non-zero bit) in the integer representation of the measured intensity. For positive values, the simplest method is to preserve the most significant bit with the value of `1' and set all other bits to 0. To obtain better results, rounding to the nearest value of 1, 2 or 3 most significant bits is utilized instead of truncation (see the examples in Table S7). The histograms in Fig. 3 provide an illustrative example of pixel intensities at different distances from the center of the detector. Note that this rounding technique alone does not decrease the data size. However, the modified data become highly compressible using various lossless compression algorithms (as indicated in Table 4).

Figure 3 Plot of quality metrics Rsplit and CC* for the original data of thaumatin (see Table 1), binned and rounded to 1, 2, 3 of the most significant bits. Under the plot, the histograms of found peak intensities over 1/d (peakograms) for different datasets are presented.

The proposed compression applied to the data discussed previously had almost no influence on data quality metrics such as CC* or R_free/R_work (see Table S8). Therefore, for this test, we have chosen the technique that is much more sensitive to data quality: SAD. We have used the thaumatin dataset (second row in Table 1). The structures for different datasets after applying lossy compression algorithms of thaumatin were solved by SAD phasing and refinement using the Crank2 pipeline (Skubák & Pannu, 2013) with default settings. As can be seen from Fig. 3, the quality of the data did not degrade much after applying rounding to several of the most significant bits. The results presented in Table 4 demonstrate that even the SAD data can still be used successfully if only two significant bits are saved (more statistics can be found in Table S9 and the reconstructed structures in Fig. S7). At the same time, saving just the single most significant bit is not enough for the same dataset – as can be seen from the last row of Table 4, the R_free/R_work are very high in this case.

Retaining only the most significant bits is quite similar to the way the data are represented by floating-point numbers, thus we have also represented the integer data in a floating-point-like way – we have converted the 32 bit integers into 8 bit floating-point values: one bit for the sign, 5 bits for the exponent and 2 more bits for the mantissa. From the numbers in Table 4, one can see that such conversion allows one to compress data even better. One very important benefit of the proposed lossy compression scheme is its speed. The truncation of the least significant bits requires very little computation. Indeed, the conversion may lend itself to implementation directly in hardware, such that it could even be realized within the detector.