“Light”, a former camera company announced a new depth sensor it could be a game-changer, disrupting LIDAR-based depth measurement and computer vision, producing a combination of RGB image and depth map with ranges of up to 1,000 meters. Assuming it works as promised and can be delivered on a large scale, that’s amazing news.
Some of you know Light as a company that made a computer camera with 16 lenses at 3 different focal lengths. Combining the images allows them to produce a high resolution image with adjustable depth of field, hoping to achieve SLR quality in a flat body. I tested their first product, and while interesting, it wasn’t ready for prime time. They later received money from Softbank. My presumption was that they would try to use computer photography to estimate the distance to pixels, but they did it beyond all my expectations, as a range of 1000m did not seem possible.
My first thought was that Light could use his history in multiple focal length photography to create this sensor, but instead he uses 3 cameras with an 88cm baseline to make a “multi-view” depth estimate (that’s i.e. stereo plus) with calculation. It also uses a fairly long basic stereo.
Although they haven’t announced a price, such a system shouldn’t be expensive. Sensors and lenses are inexpensive and solid-state. Computing gets cheaper over time, and custom silicon makes it even cheaper. This differs from most LIDARS which use more expensive components and often have moving parts or parts that need to be calibrated with extreme precision, making it difficult to align them in the high vibration environment of a car. Although considerable efforts have been made to resolve these issues with LIDARs, work continues. The Clarity claims to offer everything LIDAR does (with a few exceptions) and does it over a much larger range. Range accuracy values are not as good as LIDAR, especially at a distance, but this can be a problem that can be resolved.
The “big exception” is that LIDAR uses the light emitted, so it works the same regardless of lighting, day or night. The LIDAR sees an object in the dark of the night, the Clarity will need it to be illuminated by headlights or other sources, in the same way as human eyes and cameras. However, being as good as a human is a pretty reasonable performance.
Another exception is that some LIDARs provide the speed of a target, like radar does. It remains precious. On the other hand, with higher frame rates and enough precision and consistency, you can measure the speed of objects quite well.
The other competition is the use of computer vision, sometimes called pseudo LIDAR. It’s analogous to how humans determine the depth of things, using our brains and our knowledge of the world. This and other computer vision techniques not only suffer from the need for lighting, but also have to deal with changes in lighting and shadows that alter the appearance of things. This shouldn’t affect the clarity technique much.
The Clarity generates a distance point cloud, much the same as LIDARs do. It should be usable in some cases as a replacement for directional LIDAR. (Some LIDARs offer a 360-degree view, which Clarity doesn’t unless you install more than one.) Light claims a speed of 30 frames per second, which is better than most LIDARs.
In addition, the Clarity offers the ordinary camera a perfectly calibrated photographic image with its depth map. This is very useful – everyone wants to match their LIDAR point clouds with the images from their camera to merge the sensors, but this technique is pre-merged. Neural networks can now classify objects from a full 3D photograph of them, which should improve their performance.
A range of a thousand meters is amazing. This is for more than most automobile radars. Most 905nm LIDARs can only see reliably at around 120m while they can see some things a little further away. This has led many companies to work with 1550nm LIDARs that can reliably see up to 250m. You have to see 250m to drive fast on the highway, especially in wet weather, if you want to stop in time. It’s always good to see better.
1000m is actually more than what you typically need, although it can be useful on the freeway to spot things like a traffic jam or an accident well ahead of the road. Normally your stopping distance (up to 300m in a fast truck) is your primary detection target.)
The radar sees quite a distance and of course works day and night. It also sees through fog and dust, and even detects objects you can’t see while bouncing on the road, allowing you to track the speed of vehicles blocked by the truck you are following. It will remain in the portfolio of sensors. It also gives you speed.
famous “LIDAR is a crutch”
As a replacement for LIDAR, the Clarity, if it delivers on its promises, would kill much of the LIDAR industry but validate anyone who designed it to use it. Elon Musk does not use LIDAR, believing that all computer vision problems can and should be solved. It’s pretty hard to imagine why you wouldn’t want to use it, because you can combine the best of both worlds. Luckily for Tesla, this could be a pretty minimal modification to even existing Tesla vehicles, but it will be more of a software upgrade. A superhuman vision that also sees distance would not be a crutch, it would be an improvement in LIDAR and vision approaches.
What’s not to like?
There are very few things not to like, although there are some things we don’t know. I contacted Light to get answers to some of these questions.
- What will it end up costing? Right now anything under $ 1,000 would be a no-brainer for most teams, but it looks like it could be a lot less. It has to be lower if you want 360 degree vision. Answer: The cameras are standard automobile cameras, the chips will be reasonably cost.
- When will it be available in bulk for deployment? Answer: Probably 2022-2023.
- What is the precision? In their demonstration video, they identify points that are only 4 cm apart. Many LIDARS can do better than this, but in many cases it is not strictly necessary. One key app – spotting potholes, bumps and debris in the road – likes good accuracy, but not identifying a pedestrian. Answer: The accuracy varies with the difference and is better than the accuracy, but it is actually a good thing.
- Will it work well at night under the headlights? Light claims it’s doing well and a new version using near infrared light can do even better.
- What is the resolution of the depth maps? The images shown suggest that the depth map has a much lower resolution than the RGB image, but still better than LIDAR. Answer: 4 pixels per point of depth in the closest ranges.
- How accurate are the results? Low precision will also inherently reduce precision, but otherwise some error can be tolerated. Answer: The accuracy is around 1% up to 100m, around 2% at 250m and 10% at 1000m.
- This seems to produce errors with objects moving in front of other objects. We will need to know the scope of these errors and whether they affect the key detection algorithms.
- Will it be as sturdy as I suggest? With 3 cameras on a rack there will be a lag – does it stay calibrated? Answer: It is constantly recalibrating and therefore works even when mounted on a roof rack without shock absorbers.