In the spring, we received two computers from the computer science department. Now that we are setting up the new lab in Singer Hall, we transferred control of the LED array microscope to one of the computers. Previously, data collection was run out of a very old, very slow laptop. The new computer should make work faster and more efficient.

First, we downloaded the python 2.7 distribution of Anaconda. Our lab prefers to run python programs in Spyder, a platform included in the Anaconda distribution. We use the built-in console and variable explorer to debug and monitor our progress. We also downloaded the Arduino IDE to control the LED array.

Our data collection program interfaces with both the Arduino and the PCO Edge camera. In order to run the program, we needed to install the camera software and move the files to a location where the python code could interact with them.

First, we tried to install the camera software using a USB that was given to us when the software was installed on the old laptop. This installation was unsuccessful. Instead, we downloaded camera software (pco.camware 4.07, 64 bit) from the PCO website (X).

Next, we downloaded micromanager. We added our PCO Edge to the micromanager environment using PCO’s micromanager guide (X). To test that micromanager was working correctly with the camera, we launched micromanager and watched a few seconds of the live camera feed.

Then we attempted to run python code that interfaces with the camera through micromanager. It did not work because the python code was not able to access the camera. We changed the python file configuration so that the code executed in the same folder as the mmgr_dal_PCO_Camera.dll file, which lets python and the camera interact through micromanager.

This solution is not ideal because our code executes and saves files in a folder very crowded with micromanager software. However, it does work. We are able to use our faster new computer to run all of the programs that were previously run on the laptop.

 

To assemble the Fourier Ptychographic mircosope (FPM), we first rewired the Arduino where it had become disconnected during storage and restored the input and power connections to the array before mounting it over the microscope objective. Fortunately, the wiring for the Arduino connected to the LED array was mostly already set up, saving a large portion of time.

The pco camera was refitted with our c mount adapter and attached to the microscope. Power and usb cables were used to connect the camera to our laptop and the outlet. The 10x, 20x, and 40x objectives were also attached into place just under the stage.

We first tested the functionality of the LED array using our LEDPlayground Arduino file, which illuminates a small section of the LED array. After successfully illuminating the array, we fully tested our imaging procedure on the reassembled microscope.

The test run was performed using a calibration slide, which consists of grids and numbers for points of reference when focusing. We chose a diameter of 9 LEDs and collected a total of 69 images of the calibration slide. The resulting image stack appeared as expected, with small variations present between images as the illumination angle shifted slightly.

The successful test run confirms that  the microscope has been reassembled correctly and is ready for use collecting data sets of unstained, fixed hydra samples.

 

These notes are from the Google AI Blog post found here

Convolutional neural networks (CNNs) are usually created with a baseline model, then upgraded by scaling up certain dimensions of that model. This can include scaling up the width (adding more nodes to each layer), depth (adding more layers), or resolution (using a larger resolution of input images for training and evaluation), all for increased efficiency and accuracy of the CNN. Conventional methods scale each dimension independently and arbitrarily, and this requires a tedious manual tuning step in which training parameters are redefined to accommodate the new network size. The resulting upscaled CNN is often less accurate and less efficient than it might have been under more optimized scaling conditions.

In this paper, the authors propose a simple and effective scaling method that uniformly scales each dimension (depth, width, resolution of CNN) with a fixed set of scaling coefficients, depending on the computational capacity that can be afforded. A parameter sweep determines how to optimally scale depth, width, and input resolution to take advantage of available resources. These scaling coefficients are used to transform the base model. The resulting scaled model is called an EfficientNet. The models share a simple base image classification architecture that is scaled to generate different instances of the EfficientNet.

The EfficientNet models are capable of achieving state-of-the-art accuracy with many fewer parameters than competing networks. EfficientNet documentation and source code are available on GitHub (X).

The paper focuses on illumination angle self-calibration for Fourier ptychographic microscopy (FPM).  However, there appears to be errors in FPM due to shifts or rotations of the LED array, source instabilities, nonplanar illuminator arrangements, or sample-induced aberrations. The paper proposes  a two-pronged-angle self-calibration method that uses both preprocessing (brightfield calibration) and iterative joint estimation (spectral correlation calibration) that is quicker and more robust to system changes than state- of-the-art calibration methods.

For the Brightfield calibration, the following algorithm is used to perform the task.

For Spectral Correlation Calibration, we can see from the figure below that cost function is lower when both BC and SC are implemented.

 

References: 

R. Eckert, Z.F. Phillips,L.Waller.”Efficient illumination angle self-calibration in Fourier ptychography”, 2018 Optical Society of America

 

 

Implementation (X)

References:

 

Blog by Victor Zhou (X)

There is a heavy need for low-cost, portable microscopes for medical purposes and disease prevention in communities with limited resources. Attempts at microscopy suited for these communities have previously resulted in images with a poor space bandwidth product (SBP). Unfortunately, a high SBP is essential for accurate identification of diseases or targeted cells in medical diagnosis. SBP can be improved with expensive equipment such as automated stages for mechanical scanning and a high quality lens, but these costly materials are not feasible for resource-limited communities.

This paper by T. Aidukas, R. Eckert, A. Harvey, L. Waller, and P. Konda proposes a low-cost, portable microscope with a high SBP using a 3D printed Fourier Ptychographic microscope (FPM) equipped with an LED array and a Raspberry Pi computer board and camera. Using this setup,  multiple LR images are taken by the Raspberry Pi camera lens using  oblique illumination angles from the LED array. Using the Raspberry Pi computer, a reconstruction algorithm can then stitch these multiple LR images into one HR image with a high SBP. The reconstruction algorithm can recover both amplitude and phase images, both of which are useful for interpretation of data, especially for transparent samples like living cells or tissues. The algorithm is also able to computationally calibrate illumination angles of the LED array, which corrects misalignment errors in the array that are inevitable with imperfect, 3D printed materials. 

Previous attempts with 3D printed FPMs have used a mobile phone camera for imaging, but this also required an expensive monochrome color sensor and still resulted in a poor SBP.  In this paper, the authors instead use a Raspberry Pi camera, which includes a low-cost Bayer color sensor and produces images with a much better SBP. In addition, the fact that the algorithm as well as control of the LED illuminations and camera is done by a Raspberry Pi computer is great for compactness and portability. Together, these materials allow for an inexpensive setup with high SBP, ideal for medical purposes in isolated, resource-limited communities.

References: 

T. Aidukas, R. Eckert, A. Harvey, L. Waller, and P. Konda. “Low-Cost, Sub-Micron Resolution, Wide-Field Computational Microscopy using Opensource Hardware.” Scientific Reports 9. (2019)

 

 

 

 

In biological microscopy, a high space-bandwidth product (SBP) is desired, meaning a high resolution and high field of view. Having a high SBP often comes at the cost of temporal resolution, a common problem among medical and biological studies.  In addition, high quality imaging generally requires expensive lenses and costly equipment such as spacial light modulators for precise illumination of the sample. In this paper, L. Yeh, S. Chowdhury, and L. Waller propose a microscopy technique that is cheap, efficient, and allows for the multi-modal imaging of both fluorescence and phase.

In conventional microscopy techniques, acquiring a high SPB required the use of mechanical scanning, a time-consuming process in which an expensive automated translation stage scans the sample laterally, taking many images which are later stitched together into one image. Efforts to eliminate the need for mechanical scanning have been developed, such as with Fourier Ptychographic microscopy (FPM). As a faster and cheaper replacement to mechanical scanning, FPM uses an LED array and low numerical aperture (cheaper lens) with a high field of view to take many LR images. A computer algorithm then combines these LR images into one HR image, somewhat speeding up the imaging process while maintaining a high SBP.

This paper discusses a similar technique to FPM, but instead uses structured illumination microscopy (SIM) to capture both incoherent florescence images and coherent phase images. Most conventional types of microscopy are able to only produce either fluorescence images or phase imaging, but not both. This can be disadvantageous, since both intensity and phase images contain different types of useful information about the sample. In this paper, the authors describe a technique of SIM that allows for the acquisition of both types, all while maintaining a high SBP and having a cost-effective setup.

SIM uses the idea of Moiré ‘s Law to aid in super resolution.  Moiré‘s Law can be seen on certain images or on television, producing a wavy pattern. This  Moiré pattern occurs when an object that is being photographed contains some sort of pattern, like stripes or dots, that exceed the resolution of the sensor. This is the same idea as “beats” in sound, and happens as a result of the interference of waves. 

SIM technology takes advantage of the Moiré effect in order to extract higher resolution information from the image. In SIM, a striped or dotted illumination pattern is placed just before the sample. This pattern excites the sample, causing it to fluoresce. The interference of the fluorescence emission and excitation waves creates a Moiré pattern, which can be used and interpreted by a computer algorithm to find more spatial information about the image. This in turn allows for the enhancement of the resolution of the image. 

In this paper, Scotch tape is placed just before the sample and mounted on a  stage. This Scotch tape contains unknown illumination patterns (a type of SIM known as blind SIM) that can be used to excite the sample and produce a Moiré pattern. A computer algorithm then reconstructs the sample’s super-resolution fluorescent and phase images and corrects any aberrations or out of focus blur. The process is cheap, requiring a low NA and therefor a cheaper lens, yet still manages to have a resolution gain of 4× the original resolution of the objective. 

Overall, this particular SIM setup is low-cost, able to capture both incoherent fluorescence images and coherent phase images, and  still achieves a high SBP. The authors of this paper claim to be the first to accomplish all three of these at once. Some drawbacks, which could be improved in future work, include the still relatively slow data-acquisition time, which could potentially be sped up via a deep learning framework, and the fact that SIM does not work as well with thick samples.

References: 

L. Yeh, S. Chowdhury, and L. Waller. “Computational Structured Illumination for High-Content Fluorescence and Phase Imaging.”Department of Electrical Engineering and Computer Sciences, University of California, Berkeley. Biomedical Optical Express 10. 1978-1998 (2019).

A. Lavdas “Structured Illumination Microscopy–An Introduction.” Bitesizebio.com

N. Mansurov. “What is Moiré?” PhotographyLife.com (2012). 

L. Yeh, L. Tian, and L. Waller. “Structured Illumination Microscopy with Unknown Patterns and a Statistcal Prior.”Department of Electrical Engineering and Computer Sciences, University of California, Berkeley. Biomedical Optical Express 8. 695-711 (2017).

 

 

As was addressed in the last paper about FPM, it takes a long time and usually costly to scan a specimen and produce a high quality picture. This can be solved by reconstructing 69 photos taken under different LED pattern and using deep learning, the microscope can be trained to be able to produce a high-quality picture by taking one best photo, which shortens the time by a factor of 69.

However, such methods only satisfy situations where the sample given is thin enough so that the camera only needs to focus once. When encountering with circumstances where the sample is thick, the picture taken can be blurred due to the different distance each layer is from the camera. To improve the microscope’s functionality, digital refocusing is required during the process of reconstructing and training.

With assumptions of geometric optics, the 3-dimensional is assumed to modulate only the intensity instead of the phase. Instead of adding all the photos taken from which the obtained image would simply be that when all LED lights are on, the LED stack was sheared and resulting images were then added together. The shift-add algorithm would produce an example as shown below.

After completing the shift-add algorithm, the microscope is trained by using deep learning to achieve the goal of producing high-resolution picture in one shot of the photo, increasing the speed of scanning. Deep learning is divided up into training and evaluation and the algorithm is shown below.

Using this type of training and fine-tuning, the experiment is able to produce a result as shown in the graph.

References: 

A common challenge among 3D imaging techniques is accomplishing both high resolution and high temporal resolution.  Confocal imaging, for instance, is popular for its ability to achieve high resolution 3D images, but has poor temporal resolution. This is due to the fact that confocal imaging utilizes a point scanning system in which the light beam is focused to a small point, so analyzing the whole specimen would require multiple images as the laser beam analyzes each optical section. This paper by L. Tian and L. Waller proposes an alternative technique which improves data acquisition time using Fourier ptychography and a commercial microscope equipped with an LED array.

Phase imaging is a common method for imaging  label-free transparent specimens such as living cells or thin tissue slices, allowing you to distinguish samples from the background. The drawback of this is it requires taking multiple images at different illumination angles, thus rendering it time-intensive. Instead, this paper suggests using an LED array, which allows you to take a single intensity image per illumination angle, as opposed to multiple images per illumination angle.  By using an LED array you can also acquire both 2D phase imaging of thin samples and 3D recovery of thick samples, since data from multiple illumination angles provides both 3D information (intensity) and phase contrast.

The idea of using an LED array is an effective method for collecting 4D light field measurements. Light fields use space and angle to represent light rays in 3D. By lighting up LEDs individually in the 2D LED array, you can create a 4D data set consisting of two spatial and two angular variables, since you are two-dimensionally changing the position of light and illumination angle each time. This 4D data set can be treated as a light field measurement, which further enhances resolution of 3D images.

This method also utilizes dark-field illumination with angles of illumination that exceed the numerical aperture (NA) of the objective.  Dark-field illumination, like phase imaging,  is also commonly used for unstained and transparent specimens. Dark-field illumination involves blocking direct sources of light that would be illuminating samples along the optical axis of the microscope and instead using a light source from oblique rays with large illumination angles. In an LED array, dark-field illumination would be achieved by using only LEDs on the borders of the array, which would allow for large illumination angles.

Using dark-field illumination with angles of illumination that exceed the numerical aperture, combined with Fourier pytchography, allows for a larger total NA. Having a larger NA increases lateral resolution. Lateral resolution is a subcategory of spatial resolution, and describes the ability to distinguish two points located perpendicular to the direction of the incoming beam of light. Lateral resolution is dependent on depth of field, with shallower depths having improved lateral resolution. This method also requires a low-magnification objective with a large field of view, helping to increase axial resolution, or the ability to distinguish two points parallel to the incoming beam of light. Together, the improved axial resolution and lateral resolution (comparison shown in the image below) contributes to high-resolution 3D intensity and phase images.

In order to recover 3D intensity and phase images from the captured data and remove aberrations that would cause blurred or distorted images, L. Tian and L. Waller use an algorithm called multislicing, which models the sample as a series of thin slices. The model  also uses an iterative reconstruction routine that uses the light field  measurements as an initial guess, and through a series of computations is able to eliminates out of focus blur and recover the images.

Overall, this paper proposes an effective alternative to confocal microscopy and other forms of microscopy for 3D imaging by improving temporal resolution while maintaining high resolution. This alternative uses a programmable LED array with dark-field illumination that can quickly and efficiently scan through a series of illumination angles, producing light field measurements that can be incorporated into an iterative reconstruction model for the fast and high quality production of 3D intensity and phase images.

References: 

L. Tian and L. Waller . “3D Intensity and Phase Imaging from Light Field Measurements in an LED Array Microscope.” Department of Electrical Engineering and Computer Sciences, University of California, Berkeley. Optica 2. 104-111(2015).

D. Murphy, R. Oldfield, S. Schwartz, and M. Davidson. “Introduction to Phase Contrast Microscopy.” MicroscopyU.

W. Chambers, T. Fellers, and M. Davidson. “Darkfield Illumination.” MicrocopyU.

https://sites.google.com/site/ektasphysicseportfolio/axial-lateral-resolution

 

 

 

 

 

LED Dome

No response has yet been received from Spectral Coded Illumination, Inc and the inquiry e-form is questionably functional.

DiffuserCam

DiffuserCam consists of a raspberry pi camera and tapes (can be obtained from stores as Target) where the quote for the camera Isi the following:

https://www.ebay.com/itm/Official-Raspberry-Pi-PiNoir-Camera-Module-V2-1-8MP-Raspberry-Pi-Zero-Zero-W/232707958580?hash=item362e79a734%3Ag%3A59EAAOSwXRxatTYS&LH_ItemCondition=3

Fluorescence Microscope

For Florescence Microscopes, they have different functions available and below is a table from a buyers’ guide :

Company	Instrument	Confocal	Confocal laser-scanning	Cell-imaging system
BioTek Instruments	Lionheart FX Automated Live Cell Imager	No	No	Yes
Bio-Rad	ZOE Fluorescent Cell Imager	No	No	Yes
Carl Zeiss Microscopy	Axio Observer Research Inverted Microscope	No	No	No
Carl Zeiss Microscopy	Cell Observer SD Spinning Disk Confocal Microscope	Yes	No	No
Carl Zeiss Microscopy	Lightsheet Z.1	No	No	No
Carl Zeiss Microscopy	LSM 880 Confocal Laser Scanning Microscope	Yes	Yes	No
Etaluma	Lumascope 720 Automated Fluorescence Microscope	No	No	Yes
GE Healthcare Life Sciences	DeltaVision Elite	No	No	Yes
KEYENCE Corporation	BZ-X700 All-in-one Fluorescence Microscope	No	No	Yes
KEYENCE Corporation	VK-X250 3D Laser Scanning Microscope	Yes	Yes	No
Leica Microsystems	DM IL LED Tissue Culture Microscope	No	No	No
Leica Microsystems	MZ10 F	No	No	No
Leica Microsystems	TCS series	Yes	Yes	No
Leica Microsystems	Personal Confocal Imaging System	Yes	Yes	No
Logos Biosystems	iRiS Digital Cell Imaging System	No	No	Yes
Molecular Devices	ImageXpress Micro XLS Widefield High Content Screening System	No	No	Yes
Molecular Devices	ImageXpress Ultra High-Throughput Imaging System	Yes	Yes	Yes
Neutec Group Inc.	Multispectral Imaging / VideometerLab Benchtop Lab Analyzer	No	No	Yes
Nikon Instruments, Inc.	A1 Series Confocal Microscopes	Yes	Yes	No
Nikon Instruments, Inc.	BioStation IM-Q Time-Lapse Imaging System	No	No	Yes
Nikon Instruments, Inc.	C2+ Confocal Microscope	Yes	Yes	No
Nikon Instruments, Inc.	Eclipse Series	No	No	No
Olympus	BX Series	No	No	No
Olympus	IX Series	No	No	No
Olympus	Spinning Disk Confocal	Yes	No	No
Olympus	VivaView FL Incubator Fluorescence Microscope	No	No	Yes
Photometrics	DC2 Two-Channel Imaging System	No	No	No
Photometrics	DV2 Two-Channel Simultaneous-Imaging System	No	No	No
Photometrics	QV2 Multi-Channel Imaging System	No	No	No
Thermo Fisher Scientific	ArrayScan Series	Yes	No	Yes
Thermo Fisher Scientific	EVOS FL Auto Cell Imaging System	No	No	Yes
Thermo Fisher Scientific	EVOS FL Cell Imaging System	No	No	Yes

From the table, several types that have multiple functions are looked up and the prices are below:

• Leica TCS SP8 Series Laser Scanning Confocal Microscope
  Link: https://www.leica-microsystems.com/products/confocal-microscopes/p/leica-tcs-sp8-dls/
• Nikon C2+ Confocal Microscope
  Link: https://www.microscope.healthcare.nikon.com/products/confocal-microscopes/c2
• Nikon A1 HD25/A1R HD25 Confocal Microscope
  Link: https://www.microscope.healthcare.nikon.com/products/confocal-microscopes/a1hd25-a1rhd25
• ZEISS LSM 980 Confocal Laser Scanning Microscope
  Link: https://www.zeiss.com/microscopy/int/products/confocal-microscopes.html
  
• Keyence VK-X250 3D Laser Scanning MicroscopeLink: https://www.keyence.com/landing/microscope/lp_vk250_micro.jsp

All prices for the microscopes are under inquiry and if the prices exceeds our expectation, we will look on Ebay for alternative.

Thorlabs movable stage in x,y,z

Thorlabs movable stage in x,y,z can be obtained by the following link: https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=998
https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=2163
From Thorlabs, the pricing is listed below, no quote is needed since it’s for sale:
for 12 mm travel range: $2,703.75
for 25 mm travel range: $2,653.82

Spatial light modulators

Reference

Buyers’ link for Florescence Microscope: https://www.biocompare.com/333817-2017-Fluorescence-Microscopy-Buyers-Guide/