Realop Corporation

Realop Corporation
evort
blog

Learn from the real world and create technologies to picture the real world!

Technical Information

Optical Learning®

"Optical Learning (optical learning type aberration-free restoration)" removes all aberrations and blurring of the lens by machine learning. (Janapese Patent No. 6164564)

Optical Learning Principles

An optical simulation is performed from the lens design data, and the result is machine-learned to generate a dictionary corresponding to the position of the image. Use the dictionary to restore aberration-free.

Optical simulation            <Optical simulation>

Learning               < Learning >

Restore               < Restore >

Conventional aberration correction technology

As an aberration correction technology that appeals to high image quality, it is incorporated as a function in offline editing machines. In addition, for surveillance cameras, etc. are on sale as those that can realize real time. However, there is no technology to correct all aberrations with high accuracy from lens design data.

As a problem of conventional aberration correction technology,
・Since the filter is used to shift the pixels, the central part with high resolution is also blurred.
・There are many methods using approximate expressions, and it cannot be corrected. Distortion remains. The line is jagged. There are problems such as.
・The stretched part of the surrounding area is blurred.
・Lens aberrations other than distortion cannot be corrected.
・High image quality cannot be corrected in real time.
Etc.

 In other words, conventional aberration correction technology is correction of distortion, not aberration restoration that improves lens performance.

Effect of Optical Learning

  • Realization of reality of no aberration...It restores all aberrations such as wide-angle lenses, fisheye lenses, and endoscopes, as well as distortion and blur caused by dimming and MTF. In other words, any camera can be made aberration-free.
  • Achieves image quality (SN standard about 40 dB) comparable to professional Codec.
  • Improves inexpensive lenses to a higher level of performance. (Example: HD lens → 4K lens)

before <before> GoPro Hero4

after <after> GoPro Hero4 + Optical Learning

before <before> GoPro Hero4

after <after> GoPro Hero4 + Optical Learning

before <before> GoPro Hero4

after <after> GoPro Hero4 + Optical Learning

before <before> GoPro Hero4

after <after> GoPro Hero4 + Optical Learning

before <before> GoPro Hero4

after <after> GoPro Hero4 + Optical Learning

before <before> GoPro Hero4

after <after> GoPro Hero4 + Optical Learning

* This is an example of our Optical Learning GoPro Hero4. The lens was analyzed by Fit.

Application of Optical Learning

  • It can be applied to all types of cameras and photography equipment such as single-lens cameras, action cameras, drones, surveillance cameras, endoscopes, automobiles, and smartphones.
  • It is effective for optical systems such as endoscopes where there are many restrictions on lens design.
  • By reducing the number of lenses and improving the yield, it will lead to cost reduction and compactness.
  • It is also effective for stitching, CG, VR, image recognition, measurement, robots, etc.

Endoscope             <Endoscope>

Monitoring camera          <Monitoring camera>

Drone               <Drone>

Action camera            <Action camera>

Automobile             <Automobile>

Smartphone             <Smartphone>

Single-lens reflex camera       <Single-lens reflex camera>

In fact, most cameras are distorted.
There may come a time when only distortion-free cameras can be sold, just as ordinary CRT TVs can no longer be sold when flat CRT TVs are released.

AIRD™(Artificial Intelligence Retina Development)

"AIRD (Artificial Intelligence Retina Development)" is a RAW development (demosaic) technology that reproduces the function of the human retina using machine learning. Eliminates zipper noise and artifacts, and eliminates sensor noise, optical LPF and diffraction blur. (Japanese Patent No. 6435560)

AIRDのコンセプト

AIRD concept

 Instead of using R and B pixels, only G pixels and super-resolution technology using machine learning are used to shift twice the G pixels by half a pixel to generate them. By not using R and B pixels to generate G pixels, zipper noise and artifacts are not generated in principle.

 In addition, since the image is generated only from the highly sensitive G image, the actual sensitivity is high. It goes well with RCCB and RYYB sequences.

DLMMSE pixel <Pixel position generated by conventional demosaic (DLMMSE)>

AIRD pixel <Pixel position generated by AIRD>

CONV block <Conventional(DLMMSE) block diagram>

AIRD block <AIRD block diagram>

Conventional demosaic technology

 Artifacts such as zipper noise and false colors are almost always present. One of the causes is that it may be caused by the influence of lens aberration.

 As a problem of conventional demosaic technology,
・Artifacts such as zipper noise occur.
・False color.
・Reduced resolution.
・Degraded resolution of red and blue images.
Etc.

AIRD effect

  • Completely suppresses artifacts such as zipper noise.

DLMMSE image        <DLMMSE(conventional) development>

DLMMSE zipper noise        <DLMMSE(conventional) development zipper noise>

AIRD image        <AIRD development>

  • False colors due to zipper noise no longer occur.
  • It improves the resolution of red, blue, green, and monochromatic colors with super-resolution, and suppresses jaggies.

DLMMSE image        <DLMMSE(conventional) development>

AIRD image        <AIRD development>

  • By machine learning the optical LPF and diffraction phenomenon, blurring due to the optical LPF and diffraction is removed.

f22 DLMMSE image        <DLMMSE(conventional) development(Aperture F22)>

f22 AIRD image        <AIRD Optical LPF/diffraction learning development(Aperture F22)>

  • Zipper noise due to the influence of lens aberration can be suppressed.
  • Since it is generated only from the highly sensitive green image, the actual sensitivity is high, and sensor noise can also be suppressed by machine learning the sensor noise.

dark DLMMSE image        <DLMMSE(conventional) development(Shooting in the dark)>

dark AIRD noise removal image        <AIRD noise learning development(Shooting in the dark)>

  • When used in combination with Optical Learning or super-resolution, it is possible to suppress the emphasis on image quality deterioration due to demosaic.
  • Defect correction by demosaic processing is also possible.

Keep Resolution Mapping™

 "Keep Resolution Mapping(Super-resolution geometric transformation)" is a unique technology that uses machine learning to perform geometric transformation without reducing the resolution.

Super Resolution Stepless Zoom™

 "Super Resolution Stepless Zoom" is a stepless electronic zoom that uses machine learning.

Principle of Keep Resolution Mapping / Super Resolution Stepless Zoom

 By combining phase shift using machine learning and aperture correction using machine learning, geometric transformation (keystone correction) and stepless zoom can be restored without resolution deterioration.

KRM concept

 Keep Resolution Mapping and Super Resolution Stepless Zoom are actually exactly the same algorithms.

principle          <principle>

4 pixel generation    <Half pixel shift to generate 4 pixels>

Blurred image        <4x dense blurred image>

1/4 aperture correction        <1/4 aperture correction>

Effect of Keep Resolution Mapping / Super Resolution Stepless Zoom

 Geometric transformation (keystone correction) and stepless zoom can be freely restored without deterioration of resolution. (Conventional learning-type super-resolution and deep learning could only restore to the specified teacher resolution.)

mitchell          <3x zoom with Mitchell Filter>

srsz    <3x zoom with Super Resolution Stepless Zoom>

Visual Sharpness™

 "Visual Sharpness" is a sharpness that changes only the tilt without adding edges based on the visual model.

Traditional Sharpness

 This is an adjustment function to supplement the viewing environment such as screen size and distance, and the viewer's eyesight.

VS concept 1

 On YouTube etc., there are many people who edit with smartphones, and I see many images with borders that are over-sharpened.

Visual Sharpness Principle

 Visual Sharpness that changes only the tilt without adding edges based on the visual model.

VS concept 1

Effect of Visual Sharpness

Original                      <Original>

Traditional sharpness                  <Traditional sharpness>

Traditional sharpness                  <Traditional sharpness>

Visual Sharpness                  <Visual Sharpness>

Original                      <Original>

Traditional sharpness                  <Traditional sharpness>

Traditional sharpness                  <Traditional sharpness>

Visual Sharpness                  <Visual Sharpness>

Visual AI(Japanese trademark registration pending)

 "Visual AI" projects far and near like the human eye. Depth of field control technology using machine learning realizes pan focus by AI. (Japanese Patent No. 6694626)

Visual AI Principles

 By combining optical learning with a distance measuring means such as a stereo camera, you can create an image that can be projected as far or near as the human eye.

principle                      <principle>

Real world                    <Real world>

Single-lens reflex camera                    <Single-lens reflex camera>

deep focus                    <deep focus>

focus                    <focus>

Out of focus                    <Out of focus>

Machine learning                    <Machine learning>

Distance measurement                    <Distance measurement>

復元                    <Restore>

Visual_AI                    <Visual AI>

 Can a person see far or near by measuring the distance with both eyes and changing the resolution with the brain? I made a hypothesis. "Visual AI" has realized this mechanism by machine learning.

Effects of Visual AI

・In order to increase the depth of field, the image sensor cannot be enlarged due to optical issues.

・The aperture ratio of the image sensor is back-illuminated and has reached its limit, making it difficult to increase the sensitivity.

・Visual AI is the only way to design an easy-to-use camera with deep depth of field and high sensitivity.

・Visual AI achieves ideal deep focus with a lot of information.