Searching the World Wide World

Today, we’re unveiling a technology demonstration of machine learning at global scale, which we call GeoVisual Search. This release makes the global composite images we released last month a bit smarter.

Click anywhere on the map and we’ll scan the US, China, or the entire globe to find things that look similar. Click on a baseball diamond, a wind turbine, or even a house with a moat and GeoVisual Search will find thousands of other matching objects.

Though GeoVisual Search is just a demo, not a product, we wanted to release it because it will hopefully inspire you to think about what’s possible when computers can recognize patterns in satellite imagery. Plus, it’s really fun to play with.

Go ahead, try it. We’ll be here when you come back to tell you more.

Finding every haystack on the planet

Visual search attempts to find similar things in a set of images. An example of this is when Facebook automatically identifies who is in your pictures. You don’t look the same in every picture, but there are visually similar elements that make up your likeness. Humans are particularly skilled at recognizing patterns in the world. Thanks to recent improvements in technology, computers are getting better at this, too.

Last year, a team at Carnegie Mellon University applied the principles of visual search to seven cities around the world in a demo called Terrapattern. We were impressed with their work and wondered: could we do this not just for a few cities, but for the entire globe?

With a lot work, ingenuity, and pixels, we were able to build GeoVisual Search for three maps using three different imagery sources:

• The continental United States — This uses aerial imagery at 1m per pixel from the National Agriculture Imagery Program (NAIP) in the US. Because the imagery is high-resolution, you’ll be able to find solar farms and orchards.
• China — This map uses satellite imagery at 4m resolution from our friends at Planet. Though the resolution isn’t as high as the NAIP map over the US, Planet’s satellites will soon be providing daily pictures of the globe. In this map, you’ll be able to find solar farms and stadiums.
• The entire world — This uses a composite that Descartes Labs released a few weeks ago uses Landsat 8 and is at 15m resolution. Though much coarser than our other maps, you’ll still be able to find lots of interesting larger scale phenomena, like pivot irrigation and suburbs.

In building GeoVisual Search, the first problem to tackle is training the computer to figure out what “similarity” means. Since the human brain is really good at this, that’s a good place to start. We can simulate a simple human brain in the computer using what we understand about the physiology of the brain and the structure of neurons. We build a “neural network” in the computer and teach it hundreds of “features” like shadows, colors, and edges (this is commonly called deep learning). We then use those features to look at the entire earth and look for other images that share similar features. The more the images have in common, the higher they go on the list. Effectively, we’ve taught the computer how to recognize patterns. Sometimes the results are surprising (and sometimes they’re just plain wrong), but they’re almost always interesting.

The second problem is not nearly as sexy — there’s no simulation of neurons involved — but in many ways much harder. The challenge is that the entire Earth is many times larger than even the largest city: 150,000,000 square kilometers of landmass in total. And, the search needs to be done in “real time” — we need to be able to scan the entire globe as quickly as possible to return you results. This requires a sophisticated infrastructure.

When you click on something in GeoVisual Search, notice how it only takes a few seconds? In this time, we’ve searched through all of the potential matching images, filtered them down to only the best, and returned the results back to you. Broad area searches like this would simply be impossible with humans and would take too many people too much time. This kind of performance is only possible with a lot of computers and a considerable amount of ingenuity.

What’s next?

That’s a high-level overview of how GeoVisual Search works, meant to give you a flavor of how challenging this problem is. If you want to geek out on the details, we’ve written a technical blog post on our new Descartes Tech Blog.

GeoVisual Search is a demo, not quite a product yet. But, it’s an important step toward better understanding our planet through satellite imagery. Ultimately, we want to be able to search the globe for any object you define and track how that object changes over time.

For example, imagine that you were trying to better understand renewable energy to make a policy recommendation to the United Nations. In addition to other data sources you could gather, you might want to pinpoint the location of every wind turbine and solar panel on the planet and watch how the number has grown and at what rate over the last ten years. The underlying infrastructure we’ve built for GeoVisual Search lays the foundation for exactly this type of technical analysis.

We’ve got a lot of projects in the works, and more exciting announcements coming soon, so follow Descartes Labs and keep track of how we are changing the way we look at our world. (Twitter, Facebook, Medium, LinkedIn).

But for now, just have fun exploring the Earth, the US, or China with GeoVisual Search.

(Oh, and if you’re a developer and looking for a new challenge, drop us a line: we’re hiring!)