Facebook’s data center in Prineville, Oregon.
Image: Facebook

PRINEVILLE, Oregon Tucked deep in a Facebook data center, amid hundreds of humming server racks, nearly 2,000 smartphones are running some some version of a Facebook app.

Some are scrolling through the News Feed, Facebook’s notoriously algorithmic and somehow always-controversial stream of updates from its users. Other devices are launching Messenger, the spin-off messaging app Facebook made a requirement for users about two years ago. Some of the phones are rebooting. At least one is scrolling through Lady Gaga’s official Facebook page.

The phones are part of Facebook’s Mobile Device Lab, a new system the company is using to test the performance of Facebook, Messenger, Instagram and the company’s other apps. The goal? To ensure updates and features Facebook is testing at its Melo Park headquarters won’t break the app when they get pushed out to the company’s 1.6 billion users including ones using four-year-old devices half a world away.

In total, Prineville has about 60 racks, each containing 32 smartphones, mounted on boards behind layers of insulation. Most of the phones themselves are a at least a year old, and some are significantly older. Some represent cutting-edge smartphone tech from half a decade ago: the iPhone 4S, Samsung Galaxy Nexus and Nexus 5 are all there. In each rack, above the tangle of cables and phones, sits a camera, recording every onscreen movement in case a developer needs to review specific hiccup.

Facebook’s Mobile Device Lab at its Prinveille, Oregon, data center.

Image: Facebook

“When a developer makes a change to one of the mobile applications, we take that change, we build the app with the change, and then we install it on one of the devices that are here and we run the app while collecting metrics,” explains Facebook production engineer Antoine Reversat.

If the update negatively impacts a device’s memory usage, battery life or the performance of the app, like slowing down News Feed scrolling, then it’s sent back to the developer for a fix before the update can be pushed live.

The whole point is to go fast, we have to get better quickly

The setup is an important one for Facebook, which is aggressively pursuing users in developing countries many of whom are using older devices and operating systems as it looks to get its next billion users online. At stake is much more than a few app crashes or bug reports. The elaborate testing setup helps Facebook push performance-enhancing updates faster, which could be the difference between whether or not a frustrated user deletes the Facebook app from their phone after noticing it’s a battery hog.

“The whole point is to go fast; we have to get better quickly,” says Ken Patchett, Facebook’s Director of Western Data Center Operations

Right now the Mobile Device Lab is only looking at how updates affect each app’s performance while on Wi-Fi networks. But this eventually could grow to include how they perform while on reduced network speeds an issue that is particularly important to the company as it looks to grow its presence un developing markets.

“That’s sort of the next step,” Reversat says.

Facebook’s Mobile Device Lab at its Prineville, Oregon, data center. Each rack has 32 phones in it.

Image: Mashable/Karissa bell

Besides device testing, Facebook’s sprawling Prineville facility is also home to one of the social network’s biggest artificial intelligence projects: the hardware Facebook has designed specifically for machine learning, also known as Big Sur.

At first glance, Big Sur doesn’t look much different than what’s inside other racks in the data center. But inside are eight powerful graphical processing units (GPUs) that allow Facebook to train its artificial intelligence at previously unprecedented speeds.

Big Sur is used to train neural networks a type of AI that learns from data much the same way as the human brain does. Previously, the development of a single neural network could take weeks or even months, depending on the hardware setup. With Big Sur, Facebook has cut that time down to less than a day.

Facebook’s Big Sur at its Prineville data center.

Image: Facebook

If you’re a Facebook user, chances are you are already experiencing the benefits of this. Big Sur helps power the social network’s real-time translations and photo-recognition abilities where Facebook actually describes what’s in a photo instead of relying solely on tagging and user-written captions, which are often notoriously unspecific. Visually impaired users experience this through the app’s ability to recognize photos and read what’s in them.

In the future, Big Sur could be used for even more ambitious AI projects. The field is increasingly moving toward something called unsupervised learning a type of AI that’s able to learn from data on its own which could have even bigger implications for how Facebook uses artificial intelligence in its products.

Have something to add to this story? Share it in the comments.

Read more: