Fetching data¶

In this tutorial we will fetch some of the data. All public Human Organ Atlas data is available through a read-only public Google Cloud Storage bucket. in a chunked data format, making it easy to access sub-volumes of data.

Getting a remote data store¶

First we'll fetch a Dataset object from the inventory.

In [1]:

import hoa_tools.dataset
import hoa_tools.inventory

import hoa_tools.dataset import hoa_tools.inventory

In [2]:

inventory = hoa_tools.inventory.load_inventory()
dataset_name = inventory.index[0]
print(dataset_name)

inventory = hoa_tools.inventory.load_inventory() dataset_name = inventory.index[0] print(dataset_name)

S-20-28_kidney_VOI-02_6.51um_bm05

In [3]:

dataset = hoa_tools.dataset.get_dataset(dataset_name)
print(dataset)

dataset = hoa_tools.dataset.get_dataset(dataset_name) print(dataset)

Dataset(name=S-20-28_kidney_VOI-02_6.51um_bm05)

The Dataset.remote_array property gives us access to a remote data array for a dataset. The level parameter lets us choose a downsampling level: level=0 is the original resolution dataset, and each time the level goes up by one the dataset is downsampled by a factor of 2. All datasets are downsampled to level=4, so lets get a remote store for the lowest resolution copy of the data.

In [4]:

data_array = dataset.data_array(downsample_level=4)
data_array

data_array = dataset.data_array(downsample_level=4) data_array

Out[4]:

<xarray.DataArray 'S-20-28_kidney_VOI-02_6.51um_bm05' (z: 270, y: 217, x: 241)> Size: 28MB
dask.array<array, shape=(270, 217, 241), dtype=uint16, chunksize=(128, 128, 128), chunktype=numpy.ndarray>
Coordinates:
  * z        (z) float64 2kB 0.0 104.2 208.3 ... 2.781e+04 2.791e+04 2.802e+04
  * y        (y) float64 2kB 0.0 104.2 208.3 ... 2.229e+04 2.239e+04 2.25e+04
  * x        (x) float64 2kB 0.0 104.2 208.3 ... 2.479e+04 2.489e+04 2.5e+04

xarray.DataArray

'S-20-28_kidney_VOI-02_6.51um_bm05'

• z: 270
• y: 217
• x: 241

• dask.array<chunksize=(128, 128, 128), meta=np.ndarray>

  Array Chunk Bytes 26.93 MiB 4.00 MiB Shape (270, 217, 241) (128, 128, 128) Dask graph 12 chunks in 2 graph layers Data type uint16 numpy.ndarray

• Coordinates: (3)
  • z
    (z)
    float64
    0.0 104.2 ... 2.791e+04 2.802e+04

    units :

    μm

    array([    0.  ,   104.16,   208.32, ..., 27810.72, 27914.88, 28019.04],
          shape=(270,))

  • y
    (y)
    float64
    0.0 104.2 ... 2.239e+04 2.25e+04

    units :

    μm

    array([    0.  ,   104.16,   208.32, ..., 22290.24, 22394.4 , 22498.56],
          shape=(217,))

  • x
    (x)
    float64
    0.0 104.2 ... 2.489e+04 2.5e+04

    units :

    μm

    array([    0.  ,   104.16,   208.32, ..., 24790.08, 24894.24, 24998.4 ],
          shape=(241,))

Here we can see that the array is read-only, 16 bit, and has shape (477, 238, 238). At this point no data has been downloaded - to download data you need to index the remote array. As an example, lets fetch and show a slice in the x-y plane::

In [5]:

middle_slice = data_array.isel(z=100)
im = middle_slice.plot(cmap="Grays_r")
im.axes.set_aspect("equal")

middle_slice = data_array.isel(z=100) im = middle_slice.plot(cmap="Grays_r") im.axes.set_aspect("equal")

If you want to save a part of the array for later analysis, see the xarray guide to reading and writing files.