Write a NeXuS HDF5 File with links to external data

HDF5 files may contain links to data (or groups) in other files. This can be used to advantage to refer to data in existing HDF5 files and create NeXus-compliant data files. Here, we show such an example, using the same counts v. two_theta data from the examples above.

We use the HDF5 external file links with NeXus data files.

f[local_addr] = h5py.ExternalLink(external_file_name, external_addr)

where f is an open h5py.File() object in which we will create the new link, local_addr is an HDF5 path address, external_file_name is the name (relative or absolute) of an existing HDF5 file, and external_addr is the HDF5 path address of the existing data in the external_file_name to be linked.

file: external_angles.hdf5

Take for example, the structure of external_angles.hdf5, a simple HDF5 data file that contains just the two_theta angles in an HDF5 dataset at the root level of the file. Although this is a valid HDF5 data file, it is not a valid NeXus data file:

angles:float64[31] = [17.926079999999999, '...', 17.92108]
  @units = degrees

file: external_counts.hdf5

The data in the file external_angles.hdf5 might be referenced from another HDF5 file (such as external_counts.hdf5) by an HDF5 external link. [1] Here is an example of the structure:

entry:NXentry
  instrument:NXinstrument
  detector:NXdetector
    counts:NX_INT32[31] = [1037, '...', 1321]
      @units = counts
    two_theta --> file="external_angles.hdf5", path="/angles"

[1]

see these URLs for further guidance on HDF5 external links: https://portal.hdfgroup.org/display/HDF5/H5L_CREATE_EXTERNAL, http://docs.h5py.org/en/stable/high/group.html#external-links

file: external_master.hdf5

A valid NeXus data file could be created that refers to the data in these files without making a copy of the data files themselves.

Note

It is necessary for all these files to be located together in the same directory for the HDF5 external file links to work properly.`

To be a valid NeXus file, it must contain a NXentry group. For the files above, it is simple to make a master file that links to the data we desire, from structure that we create. We then add the group attributes that describe the default plottable data:

data:NXdata
  @signal = counts
  @axes = "two_theta"
  @two_theta_indices = 0

Here is (the basic structure of) external_master.hdf5, an example:

entry:NXentry
@default = data
  instrument --> file="external_counts.hdf5", path="/entry/instrument"
  data:NXdata
    @signal = counts
    @axes = "two_theta"
     @two_theta = 0
    counts --> file="external_counts.hdf5", path="/entry/instrument/detector/counts"
    two_theta --> file="external_angles.hdf5", path="/angles"

source code: external_example_write.py

Here is the complete code of a Python program, using h5py to write a NeXus-compliant HDF5 file with links to data in other HDF5 files.

external_example_write.py: Write using HDF5 external links

nexusformath5py

#!/usr/bin/env python
"""
Writes a NeXus HDF5 file using h5py with links to data in other HDF5 files.

This example is based on ``writer_2_1``.
"""

from pathlib import Path

import h5py
import numpy

from nexusformat.nexus import (NXdata, NXdetector, NXentry, NXfield,
                               NXinstrument, NXlink, nxopen)

FILE_HDF5_MASTER = "external_master.hdf5"
FILE_HDF5_ANGLES = "external_angles.hdf5"
FILE_HDF5_COUNTS = "external_counts.hdf5"

# ---------------------------

# get some data
filename = str(Path(__file__).absolute().parent.parent / "simple_example.dat")
buffer = numpy.loadtxt(filename).T
tthData = buffer[0]  # float[]
countsData = numpy.asarray(buffer[1], "int32")  # int[]

# put the angle data in an external (non-NeXus) HDF5 data file
with h5py.File(FILE_HDF5_ANGLES, "w") as f:
    ds = f.create_dataset("angles", data=tthData)
    ds.attrs["units"] = "degrees"

# put the detector counts in an external HDF5 data file
# with *incomplete* NeXus structure (no NXdata group)
with nxopen(FILE_HDF5_COUNTS, "w") as f:
    f["entry"] = NXentry()
    f["entry/instrument"] = NXinstrument()
    f["entry/instrument/detector"] = NXdetector()
    f["entry/instrument/detector/counts"] = NXfield(countsData, units="counts")
    f["entry/instrument/detector/two_theta"] = NXlink("/angles",
                                                      FILE_HDF5_ANGLES)

# create a master NeXus HDF5 file
with nxopen(FILE_HDF5_MASTER, "w") as f:
    f["entry"] = NXentry()
    counts = NXlink("/entry/instrument/detector/counts", FILE_HDF5_COUNTS,
                    name="counts")
    two_theta = NXlink("/angles", FILE_HDF5_ANGLES, name="two_theta")
    f["entry/data"] = NXdata(counts, two_theta)
    f["entry/data"].set_default()
    f["entry/instrument"] = NXlink("/entry/instrument", FILE_HDF5_COUNTS)

downloads

The Python code and files related to this section may be downloaded from the following table.

file	description
external_angles_h5dump.txt	h5dump analysis of external_angles.hdf5
external_angles.hdf5	HDF5 file written by external_example_write
external_angles_structure.txt	punx tree analysis of external_angles.hdf5
external_counts_h5dump.txt	h5dump analysis of external_counts.hdf5
external_counts.hdf5	HDF5 file written by external_example_write
external_counts_structure.txt	punx tree analysis of external_counts.hdf5
external_example_write.py	h5py code to write external linking examples
nexusformat/external_example_write.py	nexusformat code to write external linking examples
external_master_h5dump.txt	h5dump analysis of external_master.hdf5
external_master.hdf5	NeXus file written by external_example_write
external_master_structure.txt	punx tree analysis of external_master.hdf5