Deploying your website on Firebase Hosting

This is another of a series of mini-posts, mostly cheatsheets for my own use. Maybe of some value for others.

Software Development is Magic

Developers are magicians
Asking sand to do their bidding

You speak aloud your desire (or type, or click) and it is made manifest
The right spell and you house lights up when it’s dark
The right spell and you house cools down when the outside world is hot
The correct spell and money moves accounts and (hopefully) makes you even more money

These are the spells that magicians have crafted so that non magicians can indirectly ask Sand to grant their desires

The magicians themselves know that you have to be very careful what you wish for
The Sand is a fickle genie
A mis-cast spell can have consequences

We call objects into being by calling them by a name
Give them inheritance, properties and abilities

But like the Sorcerer’s Apprentice, we can create magic beyond our ability to control.

A little humility
And an understanding that our spells can have consequences
Is what separates a true wizard from an acolyte

Remember to be careful what you wish for

Imports

Import libraries and write settings here.

# File handling
import os
import glob

# Data manipulation
import pandas as pd

import numpy as np

# Data visualization
import matplotlib
from matplotlib import pyplot as plt
from matplotlib import image as mpimage

import seaborn as sns
from tqdm import tqdm

# Date manipulation
import datetime

# DICOM format reader
import pydicom

def show_dcm_info(ds):
    print()
    print("Filename.........: ", file_path)
    print("Storage type.....: ", ds.SOPClassUID)
    print()
    
    pat_name = ds.PatientName
    display_name = pat_name.family_name + ", " + pat_name.given_name
    print("Patient name.....: ", display_name)
    print("Patient ID.......: ", ds.PatientID)
    print("Patient Age......: ", ds.PatientAge)
    print("Patient Sex......: ", ds.PatientSex)
    print("Modality.........: ", ds.Modality)
    print("Body Part........: ", ds.BodyPartExamined)
    print("View Position....: ", ds.ViewPosition)
    
    
    if 'PixelData' in ds:
        rows = int(ds.Rows)
        cols = int(ds.Columns)
        print("Image size.......:  {rows:d} x {cols:d}, {size:d} bytes".format(
            rows=rows, cols=cols, size=len(ds.PixelData)))
        if 'PixelSpacing' in ds:
            print("Pixel spacing....: ", ds.PixelSpacing)

def plot_pixel_array(ds, figsize=(10, 10)):
    plt.figure(figsize=figsize)
    plt.imshow(ds.pixel_array, cmap=plt.cm.bone)
    plt.show()

i = 1
num_to_plot = 5
path = '../dicom/train_images/'
for file_name in os.listdir(path):
    file_path = os.path.join(path, file_name)
    ds = pydicom.dcmread(file_path)
    show_dcm_info(ds)
    plot_pixel_array(ds)
    
    if i >= num_to_plot:
        break
        
    i += 1

Filename.........:  ../dicom/train_images/1.2.276.0.7230010.3.1.4.8323329.300.1517875162.258081.dcm
Storage type.....:  1.2.840.10008.5.1.4.1.1.7

Patient name.....:  88c14312-3265-4a3f-b7bb-41818107d607, 
Patient ID.......:  88c14312-3265-4a3f-b7bb-41818107d607
Patient Age......:  58
Patient Sex......:  F
Modality.........:  CR
Body Part........:  CHEST
View Position....:  AP
Image size.......:  1024 x 1024, 154050 bytes
Pixel spacing....:  [0.139, 0.139]

Filename.........:  ../dicom/train_images/1.2.276.0.7230010.3.1.4.8323329.304.1517875162.301989.dcm
Storage type.....:  1.2.840.10008.5.1.4.1.1.7

Patient name.....:  fa43083b-0d94-4849-a5c4-40120c380164, 
Patient ID.......:  fa43083b-0d94-4849-a5c4-40120c380164
Patient Age......:  60
Patient Sex......:  M
Modality.........:  CR
Body Part........:  CHEST
View Position....:  PA
Image size.......:  1024 x 1024, 150238 bytes
Pixel spacing....:  [0.14300000000000002, 0.14300000000000002]

Filename.........:  ../dicom/train_images/1.2.276.0.7230010.3.1.4.8323329.301.1517875162.280319.dcm
Storage type.....:  1.2.840.10008.5.1.4.1.1.7

Patient name.....:  97d32841-8836-4630-873c-be0b4d2e5478, 
Patient ID.......:  97d32841-8836-4630-873c-be0b4d2e5478
Patient Age......:  50
Patient Sex......:  F
Modality.........:  CR
Body Part........:  CHEST
View Position....:  PA
Image size.......:  1024 x 1024, 159756 bytes
Pixel spacing....:  [0.14300000000000002, 0.14300000000000002]

Filename.........:  ../dicom/train_images/1.2.276.0.7230010.3.1.4.8323329.303.1517875162.295039.dcm
Storage type.....:  1.2.840.10008.5.1.4.1.1.7

Patient name.....:  019a6d6a-4eac-4372-a1d4-fe9193826333, 
Patient ID.......:  019a6d6a-4eac-4372-a1d4-fe9193826333
Patient Age......:  44
Patient Sex......:  M
Modality.........:  CR
Body Part........:  CHEST
View Position....:  PA
Image size.......:  1024 x 1024, 130602 bytes
Pixel spacing....:  [0.171, 0.171]

Filename.........:  ../dicom/train_images/1.2.276.0.7230010.3.1.4.8323329.302.1517875162.286330.dcm
Storage type.....:  1.2.840.10008.5.1.4.1.1.7

Patient name.....:  e6f57005-8262-46ac-92ab-7c858e4ae126, 
Patient ID.......:  e6f57005-8262-46ac-92ab-7c858e4ae126
Patient Age......:  74
Patient Sex......:  M
Modality.........:  CR
Body Part........:  CHEST
View Position....:  PA
Image size.......:  1024 x 1024, 136118 bytes
Pixel spacing....:  [0.171, 0.171]

Data in DICOM header

ds

Dataset.file_meta -------------------------------
(0002, 0000) File Meta Information Group Length  UL: 200
(0002, 0001) File Meta Information Version       OB: b'\x00\x01'
(0002, 0002) Media Storage SOP Class UID         UI: Secondary Capture Image Storage
(0002, 0003) Media Storage SOP Instance UID      UI: 1.2.276.0.7230010.3.1.4.8323329.302.1517875162.286330
(0002, 0010) Transfer Syntax UID                 UI: JPEG Baseline (Process 1)
(0002, 0012) Implementation Class UID            UI: 1.2.276.0.7230010.3.0.3.6.0
(0002, 0013) Implementation Version Name         SH: 'OFFIS_DCMTK_360'
-------------------------------------------------
(0008, 0005) Specific Character Set              CS: 'ISO_IR 100'
(0008, 0016) SOP Class UID                       UI: Secondary Capture Image Storage
(0008, 0018) SOP Instance UID                    UI: 1.2.276.0.7230010.3.1.4.8323329.302.1517875162.286330
(0008, 0020) Study Date                          DA: '19010101'
(0008, 0030) Study Time                          TM: '000000.00'
(0008, 0050) Accession Number                    SH: ''
(0008, 0060) Modality                            CS: 'CR'
(0008, 0064) Conversion Type                     CS: 'WSD'
(0008, 0090) Referring Physician's Name          PN: ''
(0008, 103e) Series Description                  LO: 'view: PA'
(0010, 0010) Patient's Name                      PN: 'e6f57005-8262-46ac-92ab-7c858e4ae126'
(0010, 0020) Patient ID                          LO: 'e6f57005-8262-46ac-92ab-7c858e4ae126'
(0010, 0030) Patient's Birth Date                DA: ''
(0010, 0040) Patient's Sex                       CS: 'M'
(0010, 1010) Patient's Age                       AS: '74'
(0018, 0015) Body Part Examined                  CS: 'CHEST'
(0018, 5101) View Position                       CS: 'PA'
(0020, 000d) Study Instance UID                  UI: 1.2.276.0.7230010.3.1.2.8323329.302.1517875162.286329
(0020, 000e) Series Instance UID                 UI: 1.2.276.0.7230010.3.1.3.8323329.302.1517875162.286328
(0020, 0010) Study ID                            SH: ''
(0020, 0011) Series Number                       IS: '1'
(0020, 0013) Instance Number                     IS: '1'
(0020, 0020) Patient Orientation                 CS: ''
(0028, 0002) Samples per Pixel                   US: 1
(0028, 0004) Photometric Interpretation          CS: 'MONOCHROME2'
(0028, 0010) Rows                                US: 1024
(0028, 0011) Columns                             US: 1024
(0028, 0030) Pixel Spacing                       DS: [0.171, 0.171]
(0028, 0100) Bits Allocated                      US: 8
(0028, 0101) Bits Stored                         US: 8
(0028, 0102) High Bit                            US: 7
(0028, 0103) Pixel Representation                US: 0
(0028, 2110) Lossy Image Compression             CS: '01'
(0028, 2114) Lossy Image Compression Method      CS: 'ISO_10918_1'
(7fe0, 0010) Pixel Data                          OB: Array of 136118 elements

Info about undelying pixel data

im = ds.pixel_array
print(type(im))
print(im.dtype)
print(im.shape)

<class 'numpy.ndarray'>
uint8
(1024, 1024)

Convert DICOM to JPG/PNG

import cv2

def dicom_to_jpg():
    # Change to True to make PNG format
    PNG = False
    # DICOM folder path
    folder_path = '../dicom/train_images/'
    # Output path
    output_path = '../dicom/converted/'
    images_path = os.listdir(folder_path)
    for n, image in enumerate(images_path):
        ds = pydicom.dcmread(os.path.join(folder_path, image))
        pixel_array_numpy = ds.pixel_array
        if PNG == False:
            image = image.replace('.dcm', '.jpg')
        else:
            image = image.replace('.dcm', '.png')
        cv2.imwrite(os.path.join(output_path, image), pixel_array_numpy)
        if n % 50 == 0:
            print('{} image converted'.format(n))

dicom_to_jpg()

0 image converted

Patient Demographics

# Load functions from scikit-image library

from skimage import exposure
import skimage.morphology as morp
from skimage.filters import rank

# For date functions
from datetime import datetime

ds.dir('patient')

['PatientAge',
 'PatientBirthDate',
 'PatientID',
 'PatientName',
 'PatientOrientation',
 'PatientSex']

ds.PatientName

'e6f57005-8262-46ac-92ab-7c858e4ae126'

# Change Patient Name

AKA = "Blaine, Richard"
ds.PatientName = AKA
ds.PatientName

'Blaine, Richard'

# Change DOB

epoch = '19700101'
ds.PatientBirthDate = epoch

ds.PatientBirthDate

'19700101'

! python --version

Python 3.9.7

# Calculate patient age at time of scan
dob = datetime.strptime(ds.PatientBirthDate, '%Y%m%d')
study_date = datetime.strptime(ds.StudyDate, '%Y%m%d')
age = (study_date - dob).days/365

print ("Age = ", age)

Age =  -69.04657534246576

# Calculate current age
dob = datetime.strptime(ds.PatientBirthDate, '%Y%m%d')
now = datetime.now()
current_age = (now - dob).days/365

print("Current age = ", current_age)

Current age =  53.25205479452055

# Change study date

fictional = '20210504'
ds.StudyDate = fictional

ds.StudyDate

'20210504'

Anonymizing a folder of DICOM images

for filename in glob.iglob('cleft/*.DCM', recursive=True):
    ds = pydicom.read_file(filename, force=True)
    
    # Obtain the name of the folder containing the DICOM files
    foldername=os.path.basename(os.path.dirname(os.path.dirname(filename)))
    
    # Delete patient name, patientID, accession number and patient birthdate
    del ds.PatientName
    del ds.PatientID
    del ds.AccessionNumber
    del ds.PatientBirthDate
    
    pydicom.write_file(filename,ds)
    print(filename)

iPython Widgets

from ipywidgets import interactive, interact, widgets, Layout, Button, Box, Dropdown, IntSlider
from IPython.display import display

# Turn off warnings
import warnings
warnings.filterwarnings('ignore')

def contrast_adjust(image_name, percentile_lo, percentile_hi):
    
    # save optimized image array to global vaiable so that other functions can use
    global img_rescale_interactive, image_name_global
    
    image_name_global = image_name
    p_lo, p_hi = np.percentile(eval(image_name), (percentile_lo, percentile_hi))
    
    img_rescale = exposure.rescale_intensity(eval(image_name), in_range=(p_lo, p_hi))
    
    
    img_rescale_interactive = img_rescale
    
    plt.figure(figsize = (6,6), dpi=100)
    plt.imshow(img_rescale, cmap=plt.cm.gray)
    
    plt.show()

# Widget to save image in differnt formats

button_jpg = widgets.Button(description = "Save .jpg version")
button_png = widgets.Button(description = "Save .png version")
button_tiff = widgets.Button(description = "Save .tiff version")

items = [
    button_jpg,
    button_png,
    button_tiff
]

box_layout = Layout(display = 'flex',
                   flex_flow = 'row',
                   align_items = 'stretch'
                   )

box = Box(children = items, layout = box_layout)

def jpg_button_clicked(b):
    plt.imsave(image_name_global + '.jpg', img_rescale_interactive, cmap=plt.cm.gray)
    
def png_button_clicked(b):
    plt.imsave(image_name_global + '.png', img_rescale_interactive, cmap=plt.cm.gray)
    
def tiff_button_clicked(b):
    plt.imsave(image_name_global + '.tiff', img_rescale_interactive, cmap=plt.cm.gray)
    
button_jpg.on_click(jpg_button_clicked)
button_png.on_click(png_button_clicked)
button_tiff.on_click(tiff_button_clicked)

w = interactive(contrast_adjust, image_name='ds.pixel_array', percentile_lo=(1,100,.5), percentile_hi=(1,100,.5))

display(w)

box

image_name_global

'ds.pixel_array'

Invert Image

plt.figure(figsize = (6, 6), dpi=100)
figure = plt.imshow(ds.pixel_array, cmap=plt.cm.gist_yarg)

Mirror Image R > L

plt.imshow(np.fliplr(ds.pixel_array), cmap=plt.cm.gray)

Flip Image

plt.imshow(np.flipud(ds.pixel_array), cmap=plt.cm.gray)

Histogram

plt.hist(ds.pixel_array.flatten(), bins=64)
plt.show()

print("pixel array = ", ds.pixel_array.shape)
print("minimum value = ", np.amin(ds.pixel_array)) # Find minimum value in pixel array
print("maximum value = ", np.amax(ds.pixel_array)) # Find maximum value in pixel array

pixel array =  (1024, 1024)
minimum value =  0
maximum value =  248

by Eric M. Baumel

JupyterLab is the latest iteration of the popular interactive Python notebook environment, inheriting from iPython and Jupyter Notebooks. See Project Jupyter.
JupyterLab provides the interactive notebook format of Jupyter Notebooks, with an integrated file borwser, console and more.

You install JupyterLab either from Anaconda or via pip with: pip install jupyterlab.

You launch the JupyterLab server from the command line with jupyter lab.

If your browser doesn’t launch by itself, you go to http://localhost:8888/lab

A iPython notebook file .ipynb can be opened with either Jupyter Notebok or JupyterLab.

JupyterLab Interface Shortcuts

• Toggle the sidebar with Cmd + B
• Save the notebook with Cmd + S
• Save As with Shift + Cmd + S
• Close a tab with Option + W or Alt + W

Running a Cell

• To execute a cell and move tho the next cell in command mode click the run button ▸ in the toolbar or use Shift + Enter on the keyboard

• To execute the cell and stay on that cell use Ctl + Enter

• Interupt a Running Cell with II

Toggle Between Command and Edit Modes

• Change to Command mode with the ESC key
• Change to the Edit mode with the Enter key

Code Completion

The Tab key gives you a list of possible class or function choices.

Shift + Tab shows a tooltip of the fucion or class the cursor is on.

Markdown Basics

Adapted from: Markdown Guide

Heading 1: Document Title

# Imports
import pandas
import numpy

Python

For a single line code, Python ignores everything after the hash mark ‘#’ and up to the end of the line.

Inline comments occur on the same line of a statement, following the code itself. They are separated by two spaces from the statement. These begin with a hash mark and a single whitespace character.

While PEP 8 https://www.python.org/dev/peps/pep-0008/#maximum-line-length advises keeping code at 79 characters or fewer per line, it suggests a max of 72 characters for inline comments and docstrings. If your comment is approaching or exceeding that length, then you’ll want to spread it out over multiple lines.

You can comment out a block of code by selecting the lines and pressing Ctrl+/ on PC, or Cmd+/ on Mac.

Docstring convention https://www.python.org/dev/peps/pep-0257/

Docstrings are documentation comments which provide documentations for functions, classes and modules. These are defined by a pair of triple quotes (“”” ), just below the function or class declaration.

This string will become the .__doc__ attribute of your function and will officially be associated with that specific method.

There are tools (e.g., pdoc https://pdoc3.github.io/pdoc and pydoc https://docs.python.org/3/library/pydoc.html) that allow you to generate documentation from the multiline docstrings in your code.

Imports (Not really about comments, but this is also good code hygiene.) Imports should be grouped from most generic to least generic: Python standard library imports third-party module or package imports Code repository sub-package imports

Place a blank line between import sections. Alphabetize within each group.

Python filenames must have a .py extension and must not contain dashes (-). You can use underscores instead.

Swift

Code Comments

Code after double slashes, //, or in between slash stars, /* */ are code comments and are ignored by the compiler.

Documentation Comments

Option+Cmd+/, tells Xcode to insert a documentation comment placeholder for us. When your text cursor is above a function it will automatically add placeholders for parameters and return type.

C and C++

C++ uses // for single line and inline comments and the C style /* for multi line / comments. Some authors suggest ’/’
and ’/**/’ for multi line commenting out of code, so you can then toggle the comments by deleting the initial comment characters to uncomment the lines.

“C-style” or “multi-line” comments are /* Comment */.

“C++-style” or “single-line” comments are // Comment.

HTML

HTML comments are for both single and multi line.

CSS

CSS comments are the C-style /* */

JavaScript

JavaScript uses the same comment style as Swift: // for single lines and: /* For multi lines */

JSDoc comments are used to for documentation comments, as in Swift, immediately above the function call. These comments begin with /** and use tags starting with the “at” symbol, such as @constructor and @param .

YAML

YAML only supports single line comments, starting with the ‘#’ sign.

JSON

Nope. There are no comments in json.

Markdown

<!--- Using 3 hyphens —->

Alternative methods:

[//]: # “Comment goes here.” [//]: # (Comment goes here.)

[comment]: # (Yet another comment style.)

Jekyll

{% comment %}

These lines are commented out.

{% endcomment %}

Ruby

A single line comment starts with # character and they extend from # to the end of the line

You comment multiple lines using =begin and =end syntax. These tokens should start at the beginning of the line and be the only thing on that line.

SQL

SQL comments use /* For both inline comments and multi lines. */ Single line comments start with ‘—’, a double dash.

Now go forth and be excellent to each other (and youself).

To get started:

• Install Python on computer, if this isn’t already installed.

• Check instillation of Python on a clean Terminal window with python —version (that is a double dash).

• Check instillation of pip with pip —version.

• Install virtualenv with pip install virtualenv.

• Confirm the instillation with virtualenv —version.

• In your development directory of choice (ex.: the Desktop, /Dev, /Documents, etc.), create a new directory with mkdir <Project_Name>.

• Navigate to the new project directory cd <Project_Name>.

• Create a new env environment with python -m venv venv.

• Activate with . venv/bin/activate (don’t forget the ‘dot space’). You should see (venv) before the terminal prompt, confirming that you are in the active virtual environment.

• Add new packages with pip install SomePackage.

• Upgrade a package with pip install --upgrade SomePackage.

• After installing dependencies, use pip freeze > requirements.txt to list the requirements and version numbers for that project. For a clean install of all of these dependencies into another fresh environment, use pip install -r requirements.txt

You can view the installed dependencies with cat requirements.txt.

• Deactivate the virtual environment with deactivate venv.

Putting your project under source control with GitHub

This is the first of a series of mini-posts of various computer housekeeping tips, initially for my own use. Hopefully these may also be some some benefit for others.

Source control with git serves several functions: - It provides a method of keeping track of changes made to your code, so you can roll back to a previous save if needed. - If used with a remote hosting service such as GitHub, gives a convenient back up of your code in the cloud. - Allows easy collaboration with other developers. - Can be used by many providers for managing deployment.

Open Source Imaging Datasets (CT and X-rays)

COVID CT Scans

http://covidctscans.org

Open source repository of COVID-19 CT scans, segmented 3D models, simulation models and more

(16 cases (includes the 10 from Coronacases.org))

Kaggle : COVID-19 CT scans

https://www.kaggle.com/andrewmvd/covid19-ct-scans

(20 cases – confirmed)

COVID-CT-Dataset: A CT Scan Dataset about COVID-19

https://github.com/UCSD-AI4H/COVID-CT

https://arxiv.org/pdf/2003.13865.pdf

COVID-19 Radiology Dataset (chest XRay & CT) for Annotation & Collaboration

https://towardsdatascience.com/covid-19-imaging-dataset-chest-xray-ct-for-annotation-collaboration-5f6e076f5f22

Coronacases.org

https://coronacases.org/forum/coronacases-org-helping-radiologists-to-help-people-in-more-than-100-countries-1

(10 cases – confirmed)

COVID-19 image data collection

https://github.com/ieee8023/covid-chestxray-dataset

(Chest Xrays)

COVID-19 CT segmentation dataset

http://medicalsegmentation.com/covid19/

COVID-19 CT Lung and Infection Segmentation Dataset

https://zenodo.org/record/3757476#.XqivsC2z3yw

(20 labeled CT scans taken from cornacases and Radiopedia)

General COVID-19 Radiology Information

Radiopedia

https://radiopaedia.org/articles/covid-19-3

Excellent overview of the disease and radiologic manifestations

RSNA Special Focus: COVID-19

https://pubs.rsna.org/2019-ncov

WHO General Health Information

World Health Organization: Global research on coronavirus disease (COVID-19)

https://www.who.int/emergencies/diseases/novel-coronavirus-2019/global-research-on-novel-coronavirus-2019-ncov

Epidemiologic Datasets

Johns Hopkins Coronavirus Resource Center

https://coronavirus.jhu.edu/us-map

Johns Hopkins Github Data Repository

https://github.com/CSSEGISandData/COVID-19

Tableau COVID-19 Data Hub

https://www.tableau.com/covid-19-coronavirus-data-resources

Nextstrain

https://github.com/nextstrain/ncov

Genomic epidemiology of novel coronavirus - Global subsampling

https://nextstrain.org/ncov/global

Kaggle COVID-19 Open Research Dataset Challenge (CORD-19)

https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge

Research papers dataset

CORD-19 COVID-19 Open Research Dataset

https://www.semanticscholar.org/cord19