The Future of Robotics Training: Inside Encord’s Jenga Game
The frontier of physical AI is a Jenga game in a warehouse in San Leandro, California.
In a bustling warehouse belonging to Encord, innovative strides in artificial intelligence are being made. Encord is a pioneering company focused on developing data tooling essential for training AI models. Among the team is Andrew Ceja, affectionately referred to as a pilot — a term Encord uses for its robotic trainers. Ceja is engaged in a unique task: carefully pulling wooden blocks from a precariously balanced Jenga tower while wearing a specialized headset. This isn’t just any headset; it includes state-of-the-art sensors that monitor brain activity as he meticulously disassembles the tower.
The Data Generation Challenge
Encord stands out in a landscape where many startups are recognizing that the true limitation of humanoid and warehouse robotics lies not in their model architecture, but in the scarcity of real-world physical training data. Instead of merely assisting robotics companies with the data they have, Encord is committed to creating the data they lack.
The brain wave headset Ceja is utilizing has been developed by Zander Labs, a German neuroscience startup. Their innovative approach seeks to measure brain activity to discern mental states such as error, intent, and surprise. This initiative aims to enhance the datasets available for training robotics models. At present, Encord’s partnership with Zander is in a trial phase, focused on generating an initial brain wave-tagged dataset, which will be evaluated in customer robotics models to assess its impact on performance before a larger rollout is considered.
Insights from Neuroscience
According to Lucas Gehrke, a neuroscientist at Zander overseeing the project, the insights gained from monitoring brain activity during various tasks can provide valuable guidance for model builders. This can help them determine when to deploy their most advanced models as they strive for more effective training.
Vineeth Velmurugan, Encord’s head of robot learning and a veteran of OpenAI’s robotics lab, describes this initiative as the “bleeding edge” of solving the bottleneck in robotic training data. Encord was initially founded to assist companies in annotating data and evaluating models, but as customers began to explore end-to-end learning for robotic tasks, they recognized a critical mismatch: “The data simply does not exist,” Velmurugan explains.
Scaling Challenges in Data Collection
The ambition that generative AI can revolutionize robotics in the same way it transformed chatbots continues to face substantial hurdles. For instance, self-driving car companies often rely on collecting physical-world data, which is increasingly challenging to scale. While some success can be found in training from video sources, they lack the fidelity of real-world experiences. Velmurugan estimates that breaking through this barrier will require a dataset as vast as five times that of YouTube’s video library, underscoring why data generation has evolved into a business necessity rather than merely a research problem.
Innovative Data Sources
To address these challenges, robotics companies are turning to two main channels: “egocentric” video gathered from workers donning cameras, often supplemented by additional angles and metrics, and data from remotely operated robots. Encord has embraced both avenues, extracting egocentric data from multiple factories worldwide. The San Leandro facility is also a testing ground for new modalities, including brain wave data and specific skill sets needed for fine-tuning robotic performance.
During a recent visit to TechCrunch, diverse tasks were being engaged with by the pilots. For instance, robotic arms were used in a leader-follower configuration where one arm mimics the motions of a human-controlled arm. This approach helps generate data for various tasks, such as pouring coffee and stacking poker chips. Velmurugan emphasizes that requests for these datasets are coming from nearly every humanoid robotics company.
Real-World Application and Future Insights
Within the Encord warehouse, inventory included an eclectic mix of items—from fake flowers in vases to various household items, each of which serves as training props for robots tasked with performing everyday activities.
Sofia Infante, another pilot at Encord, skillfully operates robotic arms to connect and disconnect ethernet cables from a server. This task mirrors the real-world demands of data center operations where precise robotic manipulation is still a work in progress. As she noted, while robotic pincers currently lack the dexterity of human hands, projects like these pave the way for future advancements.
Enhanced Understanding Through Data Annotation
Encord is also developing methods to deepen their data insights via forearm sensors that detect electrical signals in muscles. This data, combined with video footage, promises a comprehensive understanding of hand movements during manipulation tasks—a crucial aspect for training models effectively.
The datasets produced by Encord benefit from extensive annotations, describing in detail the actions taking place in each video frame. These annotations, such as “right hand tightens bolt,” bolster the efficacy of models, potentially providing 100 times more value than generic ego data. While this thorough annotation process comes at a cost—20 times greater than simple data collection—it is touted as a valuable trade-off.
However, the significant expense associated with generating physical training data poses a considerable challenge. Unlike simply scraping text from the internet, producing this type of data requires dedicated resources and finances, complicating comparisons between physical AI methodologies and those employed in large language models (LLMs).
Looking Ahead in Robotics Technology
Despite the challenges, Velmurugan observes that Encord is making sustained progress in developing innovative solutions. With visibility into the robotics landscape, the company can identify emerging data techniques being adopted across the industry, offering insights that no single customer could attain.
As the operations in Encord’s facility continue, the momentum is palpable. With a growing team of pilots like Ceja and Infante—who both transitioned from AI data annotation roles—the demand for meticulously crafted training datasets is clear. Ceja reflects on the rewarding nature of this work, stating, “It’s something new every day!”
For further insights into the advancements in physical AI, you can read more here.
Image Credit: techcrunch.com
