refinements to direct access page

* added section on getting file URLs using idc-index
* updated all examples with the details on getting bucket file URLs using idc-index
* replaced direct use of tag numbers with keyword lookup
* updated importance of offset tables to clarify it is about SM

This notebook contains all of the code samples for convenient testing - I plan to
update this notebook and add it to IDC-Tutorials.

Author: fedorov

data/downloading-data/direct-loading.md

@@ -5,6 +5,25 @@ DICOM files in the IDC are stored as "blobs" on the cloud, with one copy housed
 
 [Pydicom][2] is popular library for working with DICOM files in Python. Its [dcmread][3] function is able to accept any "file-like" object, meaning you can read a file straight from a cloud blob if you know its path. See [this page](../organization-of-data/files-and-metadata.md#storage-buckets) for information on finding the paths of the blobs for DICOM objects in IDC. The `dcmread` function also has some other options that allow you to control what is read. For example you can choose to read only the metadata and not the pixel data, or read only certain attributes. In the following two sections, we demonstrate these abilities using first Google Cloud Storage blobs and then AWS S3 blobs.
 
+##### Mapping IDC DICOM series to bucket URLs
+
+All of the image data available from IDC is replicated between public Google Cloud Storage (GCS) and AWS buckets. pip-installable [idc-index](https://github.com/imagingdatacommons/idc-index) package provides convenience functions to get URLs of the files corresponding to a given DICOM series.
+
+```python
+from idc_index import IDCClient
+
+# create IDCClient() for looking up bucket URLs
+idc_client = IDCClient()
+
+# get the list of GCS file URLs in Google bucket from SeriesInstanceUID
+gcs_file_urls = idc_client.get_series_file_URLs(seriesInstanceUID="1.3.6.1.4.1.14519.5.2.1.131619305319442714547556255525285829796",
+                                   source_bucket_location="gcs")
+
+# get the list of AWS file URLs in Google bucket from SeriesInstanceUID
+aws_file_urls = idc_client.get_series_file_URLs(seriesInstanceUID="1.3.6.1.4.1.14519.5.2.1.131619305319442714547556255525285829796",
+                                   source_bucket_location="aws")
+```
+
 ##### From Google Cloud Storage blobs
 
 The [official Python SDK for Google Cloud Storage][1] (installable from pip and PyPI as `google-cloud-storage`) provides a "file-like" interface allowing other Python libraries, such as Pydicom, to work with blobs as if they were "normal" files on the local filesystem.
@@ -15,15 +34,29 @@ To read from a GCS blob with Pydicom, first create a storage client and blob obj
 from pydicom import dcmread
 from google.cloud import storage
 
+from pydicom.datadict import keyword_dict
+
+from idc_index import IDCClient
+
+# create IDCClient() for looking up bucket URLs
+idc_client = IDCClient()
 
 # Create a client and bucket object representing the IDC public data bucket
 client = storage.Client.create_anonymous_client()
-bucket = client.bucket("idc-open-data")
+
+# get the list of file URLs in Google bucket from SeriesInstanceUID
+file_urls = idc_client.get_series_file_URLs(seriesInstanceUID="1.3.6.1.4.1.14519.5.2.1.131619305319442714547556255525285829796",
+                                   source_bucket_location="gcs")
+
+# URLs will look like this:
+# s3://idc-open-data/668029cf-41bf-4644-b68a-46b8fa99c3bc/f4fe9671-0a99-4b6d-9641-d441f13620d4.dcm
+(_,_,bucket_name,folder_name,file_name) = file_urls[0].split("/")
+blob_key = f"{folder_name}/{file_name}"
+
+bucket = client.bucket(bucket_name)
 
 # This is the path (within the above bucket) to a CT image in the IDC
-blob = bucket.blob(
-    "f44633af-5e76-4e01-a7fe-63764fc7e8c2/e36b336b-3550-48c9-8457-c853eab14e25.dcm"
-)
+blob = bucket.blob(blob_key)
 
 # Read the whole file directly from the blob
 with blob.open("rb") as reader:
@@ -36,7 +69,9 @@ with blob.open("rb") as reader:
 # Read only specific attributes, identified by their tag
 # (here the Manufacturer and ManufacturerModelName attributes)
 with blob.open("rb") as reader:
-    dcm = dcmread(reader, specific_tags=[0x0008_0070, 0x0008_1090])
+    dcm = dcmread(reader, specific_tags=[keyword_dict['Manufacturer'],
+                                         keyword_dict['ManufacturerModelName']])
+    print(dcm)
 ```
 
 Reading only metadata or only specific attributes will reduce the amount of data that needs to be pulled down under some circumstances and therefore make the loading process faster. This depends on the size of the attributes being retrieved, the `chunk_size` (a parameter of the `open()` method that controls how much data is pulled in each HTTP request to the server), and the position of the requested element within the file (since it is necessary to seek through the file until the requested attributes are found, but any data after the requested attributes need not be pulled).
@@ -48,17 +83,28 @@ This works because running the [open][4] method on a Blob object returns a [Blob
 The `boto3` package provides a Python API for accessing S3 blobs. It can be installed with `pip install boto3`. In order to access open IDC data without providing AWS credentials, it is necessary to configure your own client object such that it does not require signing. This is demonstrated in the following example, which repeats the above example using the counterpart of the same blob on AWS S3. If you want to read an entire file, we recommend using a temporary buffer like this:
 
 ```python
-
 from io import BytesIO
 from pydicom import dcmread
 
 import boto3
 from botocore import UNSIGNED
 from botocore.config import Config
 
+from idc_index import IDCClient
+
+# create IDCClient() for looking up bucket URLs
+idc_client = IDCClient()
+
+# get the list of file URLs in AWS bucket from SeriesInstanceUID
+file_urls = idc_client.get_series_file_URLs(seriesInstanceUID="1.3.6.1.4.1.14519.5.2.1.131619305319442714547556255525285829796",
+                                   source_bucket_location="aws")
+
+# URLs will look like this:
+# s3://idc-open-data/668029cf-41bf-4644-b68a-46b8fa99c3bc/f4fe9671-0a99-4b6d-9641-d441f13620d4.dcm
+(_,_,bucket_name,folder_name,file_name) = file_urls[0].split("/")
+blob_key = f"{folder_name}/{file_name}"
 
-# Path (within the idc-open-data bucket) to an IDC CT image on AWS S3
-blob_key = "f44633af-5e76-4e01-a7fe-63764fc7e8c2/e36b336b-3550-48c9-8457-c853eab14e25.dcm"
+bucket = client.bucket(bucket_name)
 
 # Configure a client to avoid the need for AWS credentials
 s3_client = boto3.client('s3', config=Config(signature_version=UNSIGNED))
@@ -70,7 +116,6 @@ with BytesIO() as buf:
     # Use pydicom to read from the in-memory buffer
     buf.seek(0)
     dcm = dcmread(buf)
-
 ```
 
 Unlike `google-cloud-storage`, `boto3` does not provide a file-like interface to access data in blobs. Instead, the `smart_open` [package][15] is a third-party package that wraps an S3 client to expose a "file-like" interface. It can be installed with `pip install 'smart_open[s3]'`. However, we have found that the buffering behavior of this package (which is intended for streaming) is not well matched to the use case of reading DICOM metadata, resulting in many unnecassary requests while reading the metadata of DICOM files (see [this](https://github.com/piskvorky/smart_open/issues/712) issue). Therefore while the following will work, we recommend using the approach in the above example (downloading the whole file) in most cases even if you only want to read the metadata as it will likely be much faster. The exception to this is when reading only the metadata of very large images where the total amount of pixel data dwarfs the amount of metadata (or using frame-level access to such images, see below).
@@ -83,12 +128,25 @@ from botocore import UNSIGNED
 from botocore.config import Config
 import smart_open
 
+from idc_index import IDCClient
+
+# create IDCClient() for looking up bucket URLs
+idc_client = IDCClient()
+
+# get the list of file URLs in AWS bucket from SeriesInstanceUID
+file_urls = idc_client.get_series_file_URLs(seriesInstanceUID="1.3.6.1.4.1.14519.5.2.1.131619305319442714547556255525285829796",
+                                   source_bucket_location="aws")
+
+# URL to an IDC CT image on AWS S3
+url = file_urls[0]
 
 # Configure a client to avoid the need for AWS credentials
 s3_client = boto3.client('s3', config=Config(signature_version=UNSIGNED))
 
-# URL to an IDC CT image on AWS S3
-url = 's3://idc-open-data/f44633af-5e76-4e01-a7fe-63764fc7e8c2/e36b336b-3550-48c9-8457-c853eab14e25.dcm'
+# Read the whole file directly from the blob
+dcm = dcmread(
+    smart_open.open(url, mode="rb", transport_params=dict(client=s3_client)),
+)
 
 # Read the whole file directly from the blob
 with smart_open.open(url, mode="rb", transport_params=dict(client=s3_client)) as reader:
@@ -116,20 +174,46 @@ import numpy as np
 import highdicom as hd
 import matplotlib.pyplot as plt
 from google.cloud import storage
+from pydicom import dcmread
+
+from pydicom.datadict import keyword_dict
 
+from idc_index import IDCClient
+
+# create IDCClient() for looking up bucket URLs
+idc_client = IDCClient()
+
+# get the list of file URLs in AWS bucket from SeriesInstanceUID
+# in this case we are using a series from the IDC CCDI-MCI collection
+file_urls = idc_client.get_series_file_URLs(seriesInstanceUID="1.3.6.1.4.1.5962.99.1.1900325859.924065538.1719887277027.4.0",
+                                   source_bucket_location="gcs")
+
+(_,_,bucket_name,folder_name,file_name) = file_urls[0].split("/")
 
 # Create a storage client and use it to access the IDC's public data package
 client = storage.Client.create_anonymous_client()
-bucket = client.bucket("idc-open-data")
+bucket = client.bucket(bucket_name)
+
+# go over series instances to find the base (largest matrix) layer
+# based on TotalPixelMatrixColumns value 
+largest_dimension = 0
+base_layer_blob = None
+for instance_file_url in file_urls:
+  (_,_,_,folder_name,file_name) = instance_file_url.split("/")
+  blob_name = f"{folder_name}/{file_name}"
+  
+  blob = bucket.blob(blob_name)
+
+  with blob.open("rb") as reader:
+    dcm = dcmread(reader, specific_tags=[keyword_dict['TotalPixelMatrixColumns']])
+    total_columns = dcm.TotalPixelMatrixColumns
+    if total_columns>largest_dimension:
+      largest_dimension = total_columns
+      base_layer_blob = blob
 
-# This is the path (within the above bucket) to a whole slide image from the
-# IDC collection called "CCDI MCI"
-blob = bucket.blob(
-    "763fe058-7d25-4ba7-9b29-fd3d6c41dc4b/210f0529-c767-4795-9acf-bad2f4877427.dcm"
-)
 
 # Read directly from the blob object using lazy frame retrieval
-with blob.open(mode="rb") as reader:
+with base_layer_blob.open(mode="rb") as reader:
     im = hd.imread(reader, lazy_frame_retrieval=True)
 
     # Grab an arbitrary region of tile full pixel matrix
@@ -159,15 +243,25 @@ As a further example, we use lazy frame retrieval to load only a specific set of
 import highdicom as hd
 from google.cloud import storage
 
+# create IDCClient() for looking up bucket URLs
+from idc_index import IDCClient
+idc_client = IDCClient()
+
+# Get the file URL corresponding to the segmentation of a CT series
+# containing a large number of different organs - the same one as used in the 
+# IDC Portal front page
+file_urls = idc_client.get_series_file_URLs(seriesInstanceUID="1.2.276.0.7230010.3.1.3.313263360.15787.1706310178.804490",
+                                   source_bucket_location="gcs")
+
+(_,_,bucket_name,folder_name,file_name) = file_urls[0].split("/")
 
 # Create a storage client and use it to access the IDC's public data package
 client = storage.Client.create_anonymous_client()
-bucket = client.bucket("idc-open-data")
+bucket = client.bucket(bucket_name)
 
-# This is the path (within the above bucket) to a segmentation of a CT series
-# containing a large number of different organs
+blob_name = f"{folder_name}/{file_name}"
 blob = bucket.blob(
-    "3f38511f-fd09-4e2f-89ba-bc0845fe0005/c8ea3be0-15d7-4a04-842d-00b183f53b56.dcm"
+    blob_name
 )
 
 # Open the blob with "segread" using the "lazy frame retrieval" option
@@ -182,13 +276,16 @@ with blob.open(mode="rb") as reader:
         segment_numbers=selected_segment_numbers,
         combine_segments=True,
     )
+
+# print dimensions of the liver segment volume
+print(volume.shape)
 ```
 
 See [this][11] page for more information on highdicom's `Image` class, and [this][12] page for the `Segmentation` class.
 
-### The importance of offset tables
+### The importance of offset tables for SM modality
 
-Achieving good performance for these frame-level retrievals requires the presence of a "Basic Offset Table" or "Extended Offset Table" in the file. These tables specify the starting positions of each frame within the file's byte stream. Without an offset table being present, libraries such as highdicom have to parse through the pixel data to find markers that tell it where frame boundaries are, which involves pulling down significantly more data and is therefore very slow. This mostly eliminates the potential speed benefits of frame-level retrieval. Unfortunately there is no simple way to know whether a file has an offset table without downloading the pixel data and checking it. If you find that an image takes a long time to load initially, it is probably because highdicom is constucting the offset table itself because it wasn't included in the file.
+Achieving good performance for the Slide Microscopy frame-level retrievals requires the presence of a "Basic Offset Table" or "Extended Offset Table" in the file. These tables specify the starting positions of each frame within the file's byte stream. Without an offset table being present, libraries such as highdicom have to parse through the pixel data to find markers that tell it where frame boundaries are, which involves pulling down significantly more data and is therefore very slow. This mostly eliminates the potential speed benefits of frame-level retrieval. Unfortunately there is no simple way to know whether a file has an offset table without downloading the pixel data and checking it. If you find that an image takes a long time to load initially, it is probably because highdicom is constucting the offset table itself because it wasn't included in the file.
 
 Most IDC images do include an offset table, but some of the older pathology slide images do not. [This page][14] contains some notes about whether individual collections include offset tables.