Facebook is using your Instagram hashtags to teach its AI image recognition

During the opening F8 2018 keynote, Facebook CEO Mark Zuckerberg showed off the company's latest Instagram updates: Spotify integration, AI-based anti-bullying comment filters, AR camera effects and four-way video chat.

During the Day 2 keynote, Facebook revealed how your daily Instagram updates are giving its AI technology a deep-learning crash course in image recognition—one that’s apparently made its AI even smarter than Google’s at categorizing objects in photos.

Facebook pulled this off, amazingly enough, by instructing its AI to read photo hashtags and interpret photos’ subject matter.

Using this strategy, called “weakly supervised training”, Facebook's AI achieved a record 85.4% accuracy rating on an industry-wide test of image recognition, beating out Google’s previous record.

Oculus Go vs Oculus Rift: should you make the switch?

A Facebook Engineering blog post went into detail on the methods. Facebook gave the AI 3.5 billion Instagram images to learn from, labeled with 17,000 hashtags for categorization.

Facebook then trained the AI to push aside nonvisual or vague tags and focus on ones that provide the most specific categorization.

The team was able to train their AI using this method in just a few weeks by spreading the process across 336 GPUs at once. Compared to the usual method of teaching AIs to categorize photos using visual clues, this method was much faster and less labor-intensive.

A “supervised learning process often yields the best performance results,” the post reads, “but hand-labeled data sets are already nearing their functional limits in terms of size. Scaling up to billions of training images is unfeasible when all supervision is supplied by hand.”

Thus, while most object recognition software can figure out that a subject in a photo is a bird, Facebook’s AI will use hashtags or captions to figure out that it’s actually, say, a Meadowlark.

Facebook’s AI potential

Some important things to note, in light of Facebook’s recent privacy scandal: Facebook’s engineers only used public photos to train its AI—nothing from private accounts.

Plus, the AI is focused on object recognition, not facial recognition. So, it won’t be using hashtags to figure out which of your Facebook friends is your #bff.

Instead, it’ll be using this information to improve automated audio captions for the visually impaired. And the engineers also foresee “using AI to better understand video footage or to change how an image is ranked in Facebook feeds”, or to “improve the way we resurface Memories on Facebook”.

Overall, Facebook’s AI technology has been a major focus of F8 2018. On Tuesday, Facebook revealed that its Messenger AI assistant could now automatically translate messages in other languages.

And today, Mike Schroepfer, Facebook’s Chief Technology Officer, said at F8 that the company uses AI to automatically remove spam, fake accounts and propagandist content from Facebook, including “two million pieces of ISIS and al-Qaeda content”.