These notes are from the Google AI Blog post found here

Convolutional neural networks (CNNs) are usually created with a baseline model, then upgraded by scaling up certain dimensions of that model. This can include scaling up the width (adding more nodes to each layer), depth (adding more layers), or resolution (using a larger resolution of input images for training and evaluation), all for increased efficiency and accuracy of the CNN. Conventional methods scale each dimension independently and arbitrarily, and this requires a tedious manual tuning step in which training parameters are redefined to accommodate the new network size. The resulting upscaled CNN is often less accurate and less efficient than it might have been under more optimized scaling conditions.

In this paper, the authors propose a simple and effective scaling method that uniformly scales each dimension (depth, width, resolution of CNN) with a fixed set of scaling coefficients, depending on the computational capacity that can be afforded. A parameter sweep determines how to optimally scale depth, width, and input resolution to take advantage of available resources. These scaling coefficients are used to transform the base model. The resulting scaled model is called an EfficientNet. The models share a simple base image classification architecture that is scaled to generate different instances of the EfficientNet.

The EfficientNet models are capable of achieving state-of-the-art accuracy with many fewer parameters than competing networks. EfficientNet documentation and source code are available on GitHub (X).

The paper focuses on illumination angle self-calibration for Fourier ptychographic microscopy (FPM).  However, there appears to be errors in FPM due to shifts or rotations of the LED array, source instabilities, nonplanar illuminator arrangements, or sample-induced aberrations. The paper proposes  a two-pronged-angle self-calibration method that uses both preprocessing (brightfield calibration) and iterative joint estimation (spectral correlation calibration) that is quicker and more robust to system changes than state- of-the-art calibration methods.

For the Brightfield calibration, the following algorithm is used to perform the task.

For Spectral Correlation Calibration, we can see from the figure below that cost function is lower when both BC and SC are implemented.

 

References: 

R. Eckert, Z.F. Phillips,L.Waller.”Efficient illumination angle self-calibration in Fourier ptychography”, 2018 Optical Society of America

 

 

Implementation (X)

References:

 

Blog by Victor Zhou (X)

There is a heavy need for low-cost, portable microscopes for medical purposes and disease prevention in communities with limited resources. Attempts at microscopy suited for these communities have previously resulted in images with a poor space bandwidth product (SBP). Unfortunately, a high SBP is essential for accurate identification of diseases or targeted cells in medical diagnosis. SBP can be improved with expensive equipment such as automated stages for mechanical scanning and a high quality lens, but these costly materials are not feasible for resource-limited communities.

This paper by T. Aidukas, R. Eckert, A. Harvey, L. Waller, and P. Konda proposes a low-cost, portable microscope with a high SBP using a 3D printed Fourier Ptychographic microscope (FPM) equipped with an LED array and a Raspberry Pi computer board and camera. Using this setup,  multiple LR images are taken by the Raspberry Pi camera lens using  oblique illumination angles from the LED array. Using the Raspberry Pi computer, a reconstruction algorithm can then stitch these multiple LR images into one HR image with a high SBP. The reconstruction algorithm can recover both amplitude and phase images, both of which are useful for interpretation of data, especially for transparent samples like living cells or tissues. The algorithm is also able to computationally calibrate illumination angles of the LED array, which corrects misalignment errors in the array that are inevitable with imperfect, 3D printed materials. 

Previous attempts with 3D printed FPMs have used a mobile phone camera for imaging, but this also required an expensive monochrome color sensor and still resulted in a poor SBP.  In this paper, the authors instead use a Raspberry Pi camera, which includes a low-cost Bayer color sensor and produces images with a much better SBP. In addition, the fact that the algorithm as well as control of the LED illuminations and camera is done by a Raspberry Pi computer is great for compactness and portability. Together, these materials allow for an inexpensive setup with high SBP, ideal for medical purposes in isolated, resource-limited communities.

References: 

T. Aidukas, R. Eckert, A. Harvey, L. Waller, and P. Konda. “Low-Cost, Sub-Micron Resolution, Wide-Field Computational Microscopy using Opensource Hardware.” Scientific Reports 9. (2019)

 

 

 

 

In biological microscopy, a high space-bandwidth product (SBP) is desired, meaning a high resolution and high field of view. Having a high SBP often comes at the cost of temporal resolution, a common problem among medical and biological studies.  In addition, high quality imaging generally requires expensive lenses and costly equipment such as spacial light modulators for precise illumination of the sample. In this paper, L. Yeh, S. Chowdhury, and L. Waller propose a microscopy technique that is cheap, efficient, and allows for the multi-modal imaging of both fluorescence and phase.

In conventional microscopy techniques, acquiring a high SPB required the use of mechanical scanning, a time-consuming process in which an expensive automated translation stage scans the sample laterally, taking many images which are later stitched together into one image. Efforts to eliminate the need for mechanical scanning have been developed, such as with Fourier Ptychographic microscopy (FPM). As a faster and cheaper replacement to mechanical scanning, FPM uses an LED array and low numerical aperture (cheaper lens) with a high field of view to take many LR images. A computer algorithm then combines these LR images into one HR image, somewhat speeding up the imaging process while maintaining a high SBP.

This paper discusses a similar technique to FPM, but instead uses structured illumination microscopy (SIM) to capture both incoherent florescence images and coherent phase images. Most conventional types of microscopy are able to only produce either fluorescence images or phase imaging, but not both. This can be disadvantageous, since both intensity and phase images contain different types of useful information about the sample. In this paper, the authors describe a technique of SIM that allows for the acquisition of both types, all while maintaining a high SBP and having a cost-effective setup.

SIM uses the idea of Moiré ‘s Law to aid in super resolution.  Moiré‘s Law can be seen on certain images or on television, producing a wavy pattern. This  Moiré pattern occurs when an object that is being photographed contains some sort of pattern, like stripes or dots, that exceed the resolution of the sensor. This is the same idea as “beats” in sound, and happens as a result of the interference of waves. 

SIM technology takes advantage of the Moiré effect in order to extract higher resolution information from the image. In SIM, a striped or dotted illumination pattern is placed just before the sample. This pattern excites the sample, causing it to fluoresce. The interference of the fluorescence emission and excitation waves creates a Moiré pattern, which can be used and interpreted by a computer algorithm to find more spatial information about the image. This in turn allows for the enhancement of the resolution of the image. 

In this paper, Scotch tape is placed just before the sample and mounted on a  stage. This Scotch tape contains unknown illumination patterns (a type of SIM known as blind SIM) that can be used to excite the sample and produce a Moiré pattern. A computer algorithm then reconstructs the sample’s super-resolution fluorescent and phase images and corrects any aberrations or out of focus blur. The process is cheap, requiring a low NA and therefor a cheaper lens, yet still manages to have a resolution gain of 4× the original resolution of the objective. 

Overall, this particular SIM setup is low-cost, able to capture both incoherent fluorescence images and coherent phase images, and  still achieves a high SBP. The authors of this paper claim to be the first to accomplish all three of these at once. Some drawbacks, which could be improved in future work, include the still relatively slow data-acquisition time, which could potentially be sped up via a deep learning framework, and the fact that SIM does not work as well with thick samples.

References: 

L. Yeh, S. Chowdhury, and L. Waller. “Computational Structured Illumination for High-Content Fluorescence and Phase Imaging.”Department of Electrical Engineering and Computer Sciences, University of California, Berkeley. Biomedical Optical Express 10. 1978-1998 (2019).

A. Lavdas “Structured Illumination Microscopy–An Introduction.” Bitesizebio.com

N. Mansurov. “What is Moiré?” PhotographyLife.com (2012). 

L. Yeh, L. Tian, and L. Waller. “Structured Illumination Microscopy with Unknown Patterns and a Statistcal Prior.”Department of Electrical Engineering and Computer Sciences, University of California, Berkeley. Biomedical Optical Express 8. 695-711 (2017).

 

 

As was addressed in the last paper about FPM, it takes a long time and usually costly to scan a specimen and produce a high quality picture. This can be solved by reconstructing 69 photos taken under different LED pattern and using deep learning, the microscope can be trained to be able to produce a high-quality picture by taking one best photo, which shortens the time by a factor of 69.

However, such methods only satisfy situations where the sample given is thin enough so that the camera only needs to focus once. When encountering with circumstances where the sample is thick, the picture taken can be blurred due to the different distance each layer is from the camera. To improve the microscope’s functionality, digital refocusing is required during the process of reconstructing and training.

With assumptions of geometric optics, the 3-dimensional is assumed to modulate only the intensity instead of the phase. Instead of adding all the photos taken from which the obtained image would simply be that when all LED lights are on, the LED stack was sheared and resulting images were then added together. The shift-add algorithm would produce an example as shown below.

After completing the shift-add algorithm, the microscope is trained by using deep learning to achieve the goal of producing high-resolution picture in one shot of the photo, increasing the speed of scanning. Deep learning is divided up into training and evaluation and the algorithm is shown below.

Using this type of training and fine-tuning, the experiment is able to produce a result as shown in the graph.

References: 

A common challenge among 3D imaging techniques is accomplishing both high resolution and high temporal resolution.  Confocal imaging, for instance, is popular for its ability to achieve high resolution 3D images, but has poor temporal resolution. This is due to the fact that confocal imaging utilizes a point scanning system in which the light beam is focused to a small point, so analyzing the whole specimen would require multiple images as the laser beam analyzes each optical section. This paper by L. Tian and L. Waller proposes an alternative technique which improves data acquisition time using Fourier ptychography and a commercial microscope equipped with an LED array.

Phase imaging is a common method for imaging  label-free transparent specimens such as living cells or thin tissue slices, allowing you to distinguish samples from the background. The drawback of this is it requires taking multiple images at different illumination angles, thus rendering it time-intensive. Instead, this paper suggests using an LED array, which allows you to take a single intensity image per illumination angle, as opposed to multiple images per illumination angle.  By using an LED array you can also acquire both 2D phase imaging of thin samples and 3D recovery of thick samples, since data from multiple illumination angles provides both 3D information (intensity) and phase contrast.

The idea of using an LED array is an effective method for collecting 4D light field measurements. Light fields use space and angle to represent light rays in 3D. By lighting up LEDs individually in the 2D LED array, you can create a 4D data set consisting of two spatial and two angular variables, since you are two-dimensionally changing the position of light and illumination angle each time. This 4D data set can be treated as a light field measurement, which further enhances resolution of 3D images.

This method also utilizes dark-field illumination with angles of illumination that exceed the numerical aperture (NA) of the objective.  Dark-field illumination, like phase imaging,  is also commonly used for unstained and transparent specimens. Dark-field illumination involves blocking direct sources of light that would be illuminating samples along the optical axis of the microscope and instead using a light source from oblique rays with large illumination angles. In an LED array, dark-field illumination would be achieved by using only LEDs on the borders of the array, which would allow for large illumination angles.

Using dark-field illumination with angles of illumination that exceed the numerical aperture, combined with Fourier pytchography, allows for a larger total NA. Having a larger NA increases lateral resolution. Lateral resolution is a subcategory of spatial resolution, and describes the ability to distinguish two points located perpendicular to the direction of the incoming beam of light. Lateral resolution is dependent on depth of field, with shallower depths having improved lateral resolution. This method also requires a low-magnification objective with a large field of view, helping to increase axial resolution, or the ability to distinguish two points parallel to the incoming beam of light. Together, the improved axial resolution and lateral resolution (comparison shown in the image below) contributes to high-resolution 3D intensity and phase images.

In order to recover 3D intensity and phase images from the captured data and remove aberrations that would cause blurred or distorted images, L. Tian and L. Waller use an algorithm called multislicing, which models the sample as a series of thin slices. The model  also uses an iterative reconstruction routine that uses the light field  measurements as an initial guess, and through a series of computations is able to eliminates out of focus blur and recover the images.

Overall, this paper proposes an effective alternative to confocal microscopy and other forms of microscopy for 3D imaging by improving temporal resolution while maintaining high resolution. This alternative uses a programmable LED array with dark-field illumination that can quickly and efficiently scan through a series of illumination angles, producing light field measurements that can be incorporated into an iterative reconstruction model for the fast and high quality production of 3D intensity and phase images.

References: 

L. Tian and L. Waller . “3D Intensity and Phase Imaging from Light Field Measurements in an LED Array Microscope.” Department of Electrical Engineering and Computer Sciences, University of California, Berkeley. Optica 2. 104-111(2015).

D. Murphy, R. Oldfield, S. Schwartz, and M. Davidson. “Introduction to Phase Contrast Microscopy.” MicroscopyU.

W. Chambers, T. Fellers, and M. Davidson. “Darkfield Illumination.” MicrocopyU.

https://sites.google.com/site/ektasphysicseportfolio/axial-lateral-resolution

 

 

 

 

 

LED Dome

No response has yet been received from Spectral Coded Illumination, Inc and the inquiry e-form is questionably functional.

DiffuserCam

DiffuserCam consists of a raspberry pi camera and tapes (can be obtained from stores as Target) where the quote for the camera Isi the following:

https://www.ebay.com/itm/Official-Raspberry-Pi-PiNoir-Camera-Module-V2-1-8MP-Raspberry-Pi-Zero-Zero-W/232707958580?hash=item362e79a734%3Ag%3A59EAAOSwXRxatTYS&LH_ItemCondition=3

Fluorescence Microscope

For Florescence Microscopes, they have different functions available and below is a table from a buyers’ guide :

Company	Instrument	Confocal	Confocal laser-scanning	Cell-imaging system
BioTek Instruments	Lionheart FX Automated Live Cell Imager	No	No	Yes
Bio-Rad	ZOE Fluorescent Cell Imager	No	No	Yes
Carl Zeiss Microscopy	Axio Observer Research Inverted Microscope	No	No	No
Carl Zeiss Microscopy	Cell Observer SD Spinning Disk Confocal Microscope	Yes	No	No
Carl Zeiss Microscopy	Lightsheet Z.1	No	No	No
Carl Zeiss Microscopy	LSM 880 Confocal Laser Scanning Microscope	Yes	Yes	No
Etaluma	Lumascope 720 Automated Fluorescence Microscope	No	No	Yes
GE Healthcare Life Sciences	DeltaVision Elite	No	No	Yes
KEYENCE Corporation	BZ-X700 All-in-one Fluorescence Microscope	No	No	Yes
KEYENCE Corporation	VK-X250 3D Laser Scanning Microscope	Yes	Yes	No
Leica Microsystems	DM IL LED Tissue Culture Microscope	No	No	No
Leica Microsystems	MZ10 F	No	No	No
Leica Microsystems	TCS series	Yes	Yes	No
Leica Microsystems	Personal Confocal Imaging System	Yes	Yes	No
Logos Biosystems	iRiS Digital Cell Imaging System	No	No	Yes
Molecular Devices	ImageXpress Micro XLS Widefield High Content Screening System	No	No	Yes
Molecular Devices	ImageXpress Ultra High-Throughput Imaging System	Yes	Yes	Yes
Neutec Group Inc.	Multispectral Imaging / VideometerLab Benchtop Lab Analyzer	No	No	Yes
Nikon Instruments, Inc.	A1 Series Confocal Microscopes	Yes	Yes	No
Nikon Instruments, Inc.	BioStation IM-Q Time-Lapse Imaging System	No	No	Yes
Nikon Instruments, Inc.	C2+ Confocal Microscope	Yes	Yes	No
Nikon Instruments, Inc.	Eclipse Series	No	No	No
Olympus	BX Series	No	No	No
Olympus	IX Series	No	No	No
Olympus	Spinning Disk Confocal	Yes	No	No
Olympus	VivaView FL Incubator Fluorescence Microscope	No	No	Yes
Photometrics	DC2 Two-Channel Imaging System	No	No	No
Photometrics	DV2 Two-Channel Simultaneous-Imaging System	No	No	No
Photometrics	QV2 Multi-Channel Imaging System	No	No	No
Thermo Fisher Scientific	ArrayScan Series	Yes	No	Yes
Thermo Fisher Scientific	EVOS FL Auto Cell Imaging System	No	No	Yes
Thermo Fisher Scientific	EVOS FL Cell Imaging System	No	No	Yes

From the table, several types that have multiple functions are looked up and the prices are below:

• Leica TCS SP8 Series Laser Scanning Confocal Microscope
  Link: https://www.leica-microsystems.com/products/confocal-microscopes/p/leica-tcs-sp8-dls/
• Nikon C2+ Confocal Microscope
  Link: https://www.microscope.healthcare.nikon.com/products/confocal-microscopes/c2
• Nikon A1 HD25/A1R HD25 Confocal Microscope
  Link: https://www.microscope.healthcare.nikon.com/products/confocal-microscopes/a1hd25-a1rhd25
• ZEISS LSM 980 Confocal Laser Scanning Microscope
  Link: https://www.zeiss.com/microscopy/int/products/confocal-microscopes.html
  
• Keyence VK-X250 3D Laser Scanning MicroscopeLink: https://www.keyence.com/landing/microscope/lp_vk250_micro.jsp

All prices for the microscopes are under inquiry and if the prices exceeds our expectation, we will look on Ebay for alternative.

Thorlabs movable stage in x,y,z

Thorlabs movable stage in x,y,z can be obtained by the following link: https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=998
https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=2163
From Thorlabs, the pricing is listed below, no quote is needed since it’s for sale:
for 12 mm travel range: $2,703.75
for 25 mm travel range: $2,653.82

Spatial light modulators

Reference

Buyers’ link for Florescence Microscope: https://www.biocompare.com/333817-2017-Fluorescence-Microscopy-Buyers-Guide/

 

The goal of this paper is to introduce the Deep Recursive Residual Network (DRRN), a type of CNN with the goal of providing a deep and effective model that uses many fewer parameters and has a higher performance than competing single image super-resolution (SISR) models.

The authors are motivated by recent models that achieve improved performance in super resolution reconstruction by adding many more layers (as many as 30) than previous designs (usually no more than 5). A pitfall of these deep models are their very large parameters. The described DRRN has significantly fewer parameters (2-14x less) while maintaining depth, with most DRRN models having as many as 52 layers.

Residual networks use an identity branch to retain the input to a residual block.  Then, the block only optimizes the residual mapping between the input and output instead of optimizing the mapping from input to output directly.

In the picture above (from the open source textbook ‘Dive into Deep Learning’), the difference between a regular (on the left) and residual block (right) is shown. The mapping output generated by the regular block, f(x), contains x more information than the residual block, and thus requires more computations. Residual blocks simplify the amount of input required and thus reduce computations.

The output of a SR model is anticipated to be highly similar to the input, so DRRN uses both residual learning. It employs both global residual learning, which connects the input and output images, and local residual learning, which ties the input and output of every few layers.

DRRN uses a recursive block of residual units that share weights to keep the model size small even though it is deep. Adding more depth (and consequently improving accuracy) can be done without adding more weight parameters. It uses a multi-path structure in the recursive block to avoid the vanishing/exploding gradients problem that appears in very deep models.

The figure above, taken from the paper by Tai et al., compares the basic structure of DRRN with other popular deep SR frameworks.

By combining global and local residual learning, the authors achieve visibly finer resolution than competing networks. Weight sharing in the recursive model allows them to deepen their network without requiring many parameters. Implementations of DRRN are available in tensorflow and pytorch in addition to the original caffe implementation (X).

The DRRN architecture may be very useful for our lab. Accurately and efficiently mapping low resolution images to corresponding high resolution images is a priority of our microscopy research. When comparing to a previous post describing the UNet, a different type of CNN, we found that while both are fast, highly effective networks, their purposes diverge. As stated earlier, DRRN models are designed for effective single-shot super-resolution, a central focus of our research, while Unet models are designed for image segmentation and multi-categorization. While this is a potential extension of our research, it appears that the DRRN model better fits the needs of our current research.

 

References

Y. Tai, J. Yang and X. Liu, “Image Super-Resolution via Deep Recursive Residual Network,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 2790-2798.
doi: 10.1109/CVPR.2017.298

A. Zhang, Z.C. Lipton, M. Li and A.J. Smola, Dive into Deep Learning (X)

Tensorflow documentation offers useful introductory lessons on developing machine learning models for many uses.

First, we trained a basic classification network using the Fashion MNIST dataset. Like the canonical MNIST dataset, the Fashion MNIST consists of many thousands of 28×28 pixel images and associated image labels. This dataset pairs pictures of clothing items with one of nine class labels.

Below, a sample of images and associated labels is displayed. This plot is generated during the data inspection and preprocessing portion of the tutorial. To preprocess the data, pixel intensity values are scaled between 0 and 1 from the original 0-255.

Before the model can be trained, it must be built with an appropriate configuration of layers. Overcomplicated models are more sensitive to overfitting, so it is important to chose a sufficiently simple design.

## Set up the Model ##
model = keras.Sequential([

# Flatten the 28x28 input image into a 1d array of 784 pix
keras.layers.Flatten(input_shape=(28,28)),

# Create a dense/fully-connected layer with 128 nodes
keras.layers.Dense(128, activation=tf.nn.relu),

# Create a dense/fully-connected 'softmax' layer that returns 10 probability
# scores (one for each class) that sum to 1
keras.layers.Dense(10, activation=tf.nn.softmax)
])

The optimizer, loss function, and metrics of the model are chosen during the compile step. The optimizer (Adam, Stochastic gradient descent, etc.) defines the way that the model is updated .The loss function prioritizes minimization of different types of error. Metrics evaluate the successfulness of a step.

## Compile the model 

# Chose a loss function, optimizer, and metric(s)
model.compile(optimizer='adam',
             loss='sparse_categorical_crossentropy',
             metrics=['accuracy'])

Next, the model was trained on the training data using fit command.

model.fit(train_images, train_labels, epochs=5)

After training, the test data were used to evaluate the accuracy of the model.

test_loss, test_acc = model.evaluate(test_images, 
test_labels)

Finally, the model was used to predict classifications of the test dataset. Below is a plot of the output of my implementation of the model. Each image is displayed with the model’s prediction and confidence alongside the correct label. A bar plot of probabilities for each label is displayed next to the image.

It is notable that these results are not identical to those in the tensorflow documentation, but are very similar. The model builds its own association between the training data and corresponding tags, and may learn slightly different relationships each time it is built and trained.

Our commented implementation of the tensorflow walkthrough may be viewed here.

 

We also completed an example of sentiment classification in text (positive or negative) and a deep convolutional general adversarial network (DCGAN)designed to generate ‘written’ numbers in the style of the classic MNIST dataset, both using walkthroughs in the tensorflow documentation pages.

In the case of the DCGAN, we were able to observe improvement from patchy white noise to moderately resolved shapes. However, our personal computers lack the computational power to quickly train a model of this type, and we have not yet completed the prescribed 50 epochs.

 

The same Fashion MNIST example is available for other deep learning frameworks, including pytorch. Progress on these implementations has temporarily stalled because documentation is more limited than for tensorflow and because Google colab does not easily support all of the libraries used in pytorch examples.