For anyone who has ever dreamt of being able to click on any image and buy it, whether it’s Jimmy Fallon’s suit on Late Night or the chair he’s sitting on, Facebook has announced it’s stepping up its e-commerce game with a brand new shopping experience powered by AI.
In a Facebook Live on May 19, Mark Zuckerberg revealed a snap-and-shop feature that will allow users to buy, sell and discover items seamlessly across all of its Facebook platforms. An auto-tagging feature in Marketplace (coming soon to Facebook Pages) now autofills product descriptions upon uploading, making it easier to rank in search and communicate across languages. Facebook is aiming to make all photos shoppable across its app, and in time, videos too.
Although Facebook’s new shopping AI doesn’t have a consumer-facing name like Alexa, its co-creator, Sean Bell named it GrokNet to represent the scale in which it is capable of analyzing data. Originally developed at GrokStyle, a startup co-founded by Bell and acquired by Facebook in 2019, GrokNet is a universal AI model that leverages computer vision to identify fine-grained product attributes from billions of images.
In a video on the Facebook AI blog, Bell described how GrokNet’s product recognition system was built to simultaneously analyze dozens of disparate verticals like clothing, cars and home goods. “We built, trained, and deployed a model with 83 loss functions across seven data sets to combine multiple verticals into a single embedding space. For some of the tasks that are really difficult, you need to apply a very large weight on the loss function and you need to apply many images for every batch when training. For easier data sets, you can use a small weight, and just a very small number of images per batch. We then scaled this up across all different data sets and then figured out a way of how to combine everything at once, without compromising on any one of the tasks. We use weekly supervised learning, where a model is arranged in a feedback loop to automatically create additional training data.”
Although snap-and-shop features have been available for years on e-commerce apps like Amazon, Shopify and Stitch Fix (which acquired Brooklyn Decker’s Finery app in 2019), Facebook’s AI team has developed advanced segmentation, detection and classification capabilities that enable it to do the seemingly impossible, that is identify hidden objects. This means the AI can recognize a sweater under a jacket, a scarf obscured by hair, and even a skirt seated under a desk. To achieve this, the model uses a new method called Instance Mask Projection which can parse clothing and segment street scenes at a granular level. To ensure accuracy of the predictions and account for diversity of photos, the team trained its AI on varying poses, ages, skin tones, body types, locations, and socioeconomic classes.
“We’re building a system that can look at a picture of your outfit, segment each piece of clothing that you’re wearing, and organize them into a digital closet, allowing you to shop anything in the context of what you own,” said Tamara Berg, Facebook AI researcher. In future iterations, Facebook hopes to release an AI lifestyle assistant which provides wardrobe management, personalized recommendations of items to buy, and AR try-outs to share with friends. Facebook Ads has been testing augmented reality with brands like NYX, NARS and Ray-Ban that lets users virtually try on lipsticks and sunglasses.
Additional features for Marketplace include 3D image uploads with rotating product views and a tool called Facebook Shops that helps small businesses set up shop seamlessly across Facebook, Instagram and Whatsapp. Facebook is opening its ecosystem to allow third parties to integrate supply chain solutions. Launch partners include Shopify, BigCommerce, Cafe24, CEDCommerce, ChannelAdvisor, Feedonomics, Tiendanube, and WooCommerce.
For a Coronavirus world that needs to live virtually for awhile, Zuckerberg shared his heartfelt sentiment that Facebook Shops is meant to provide a way for those in need to generate income online, whether they’re a physical store that can’t yet fully reopen or simply someone seeking to declutter Marie Kondo style. (Heads up Poshmark and LetGo)
“Many store owners have a mission and values to make your life better with their products. We want to enable them to tell their stories,” Zuckerberg said. “We’re hoping to play a role in helping small businesses get through this difficult time and thrive long term.” Since Facebook is an ad-supported business, Zuckerberg emphasized Facebook Shops will be free to users.
originally posted on forbes.com by Martine Paris
