Data Ingestion
There are several ways to upload images into the Repository. Depending on the desired workflow and volume of data to be ingested, use one of the following methods:
- Upload files manually in a browser
- Upload in bulk from a PC using the Upload Client software
- Configure the image indexing service by contacting support
- Upload through the API for customized workflows
Manual Uploads
Files can be manually uploaded into a folder using drag and drop or through a file selection dialog. Multiple images can be selected for upload using either method. The status of uploaded files will be indicated through a progress notification at the top of the screen as well as a progress bar on each new item in the Folder Listing.
To upload file(s) using drag and drop
Navigate to any folder in the Repository (click on the Repository button from the Navigation Menu then navigate to a folder)
Drag one or more files onto the folder listing area (until the “Drop files to upload” notification appears) then drop the files
To check upload status, click on the Details button from the upload progress notification at the top of the screen
To upload file(s) using a file selection dialog
Navigate to any folder in the Repository (click on the Repository button from the Navigation Menu then navigate to a folder)
Click on the button at the top right of the item list
In the popup menu, click Upload
Click in the blue box to activate the file selection dialog or simply drag files into the blue box; repeat as required for other files
Click on the Start Upload button to start upload
Click on the X button to minimize the File Uploads dialog
To check upload status, click on the Details button from the upload progress notification at the top of the screen
To check the status of recently uploaded files
Look for the upload progress notification at the top of the screen
Click on the Details button on the notification
To edit metadata for recently uploaded files
Look for the upload progress notification at the top of the screen and, if it has not been dismissed, click on the Details button. If the notification has been dismissed, this method is not available and file metadata can be edited individually through the Repository listing
Click on the Edit Metadata button for any of the files listed
In the Add fields box look for the fields to be added or type a few letters to find the desired field; repeat as required for other fields
Edit the value in the provided field
Click on the Save Changes button to commit the field(s), or click on the Cancel button to discard changes
Click on the X button in the top right-hand corner of the File Uploads dialog
To locate recently uploaded files
Look for the upload progress notification at the top of the screen and, if it has not been dismissed, click on the Details button
Click on the button below any of the recently uploaded files
Click on the X button in the top right-hand corner of the File Uploads dialog
To clear the completed list of recently uploaded files
Look for the upload progress notification at the top of the screen and, if it has not been dismissed, click on the Details button
Click on the Clear Completed button
To clear the upload progress history and remove it from view
Look for the upload progress notification at the top of the screen
Click on the Dismiss button on the notification
Multi-File Images
To upload multi-file images such as MRXS, NDPIS, AFI and VSI via the browser, the set of files and folders related to the image must be uploaded as a single archive file (e.g., a zip file). Do not include additional files or folders in the archive file or change the relative path of files in the archive (i.e., ensure the main file is at the root of the archive and there should be one image per archive).
The following archive formats are supported:
- tar: Standard tarball archive format
- tar.gz: gzip compressed tarball
- ZIP: Created using the DEFLATE or DEFLATE64 algorithm. Microsoft Windows and macOS create these types of archives when a user sends files to a "compressed folder"
The following multi-file image formats are supported, with notes on how to properly organize the archive file contents.
3DHISTECH MRXS
An MRXS image consists of a base .mrxs image, and a correspondingly named folder which contains:
- Slidedat.ini: Metadata for the entire image
- Index.dat: Index describing how the data files are combined into a WSI
- DataXXXX.dat: One or more image data files
Example: An archive file named "slide_001A.zip" containing:
slide_001A.mrxsslide_001A/Data0000.datData0001.dat...Data0021.datData0022.datIndex.datSlidedat.ini
Aperio AFI
An AFI image consists of:
- IMAGE_NAME.afi: An XML manifest of the files representing each channel
- *.svs: One or more images, each representing a channel
Example: An archive file named "273481.zip" containing:
273481_Alexa Fluor 488.svs273481_Alexa Fluor 594.svs273481_DAPI.svs273481.afi
Hamamatsu NDPIS
An NDPIS image consists of:
- IMAGE_NAME.ndpis: An INI-like manifest of the files representing each channel
- *.ndpi: One or more images, each representing a channel
Example: An archive file named "Ex_changer.tar.gz" containing:
Ex_changer-DAPI.ndpiEx_changer-FITC.ndpiEx_changer-TxRed.ndpiEx_changer.ndpis
Lunaphore OME-TIFF
An image in the Lunaphore variant of OME-TIFF can have an auxiliary XML file which defines the minimum value, maximum value, gamma value, and pseudocolor for each channel. This XML file needs to match the filename of the OME-TIFF file with ".xml" appended.
Example: An archive file named "Lunaphore 10plex (5 cycles)_Slide 7_Annotation 1.tar" containing:
Lunaphore 10plex (5 cycles)_Slide 7_Annotation 1.ome.tiffLunaphore 10plex (5 cycles)_Slide 7_Annotation 1.ome.tiff.xml
Olympus VSI
A VSI image consists of a base .vsi image, and a folder named _IMAGE_NAME_ which can contain a variety of subfolders and files.
Example: An archive file named "Her2.tar" containing:
Her2.vsi_Her2_/stack1/frame_t.etsstack10001/frame_t.etsstack10004/frame_t.ets
Upload Client
Where the volume of images to be uploaded is large, it can be more efficient to upload images with a dedicated tool. The Upload Client is a Windows application that has been designed for automating image uploads. It can be configured to upload images that appear in a “watch” folder. Once the images are uploaded, they are available for review in the Unsorted Uploads section of the application, and from there they can be moved to the appropriate folder in the Repository.
The Upload Client can be configured to detect and upload images that are generated by a whole slide image scanner or it can be used to batch upload images on demand.
Image Indexing Services
[add-on] [on-prem]
The indexing service has been designed for on-premise deployments. The service will find all the images in an attached storage system and reproduce the storage hierarchy in the Repository. The service provides a very efficient mechanism for managing storage in an on-premise deployment.
A side effect of the indexing service is the possibility in some circumstances for duplicate records to be created in the Repository. Images that are moved or copied on the storage system after they have been found by the indexing service will result in duplicate records, as the duplication will be detected by the service as a new file. This is not a huge problem, because both records will function normally (i.e., they will be viewable, no broken links, etc). However, the 2nd record won't have any of the annotations or metadata that may have been associated with the first record.
To reorganize the images in the Repository, we recommend using PathcoreFlow's built-in copy or move functions, which handle image metadata. Note that copy operations in the Repository are shallow, so the image data will not be duplicated on disk.
Custom Built Uploader
Users can develop custom upload workflows using the API, which provides methods for adding files and metadata to the Repository. Contact Pathcore Support to learn more.