- We present the first method capable of photorealistically reconstructing a non-rigidly - deforming scene using photos/videos captured casually from mobile phones. + This project introduces TickleBot, a low-cost open-source robotic research platform that lowers the entry barrier to Physical AI research for students and researchers.
- Our approach augments neural radiance fields - (NeRF) by optimizing an - additional continuous volumetric deformation field that warps each observed point into a - canonical 5D NeRF. - We observe that these NeRF-like deformation fields are prone to local minima, and - propose a coarse-to-fine optimization method for coordinate-based models that allows for - more robust optimization. - By adapting principles from geometry processing and physical simulation to NeRF-like - models, we propose an elastic regularization of the deformation field that further - improves robustness. + Recent advances in Physical AI have highlighted the importance of foundation models for robotics, + yet the absence of standardized, accessible hardware platforms and the high cost of commercial robotic arms + continue to hinder progress and scalability. In this work, we present TickleBot, a low-cost, + open-source robotic research platform designed to lower the entry barrier for undergraduate students and researchers. + Built at approximately USD 725, TickleBot integrates the SO-100 open-source robotic arm with a ROS2-based middleware, enabling precise manipulation and mobility without requiring a physical twin. By incorporating the NoMaD model for mobile base navigation and SmolVLA for upper-body control, the platform supports AI-driven teleoperation and task execution using only RGB camera inputs. + Owing to the aforementioned characteristics, Ticklebot can attract a large pool of researchers, supporting high-quality teleoperation data collection and robot model evaluation, thereby contributing to the scaling of robot foundation models.
- We show that Nerfies can turn casually captured selfie - photos/videos into deformable NeRF - models that allow for photorealistic renderings of the subject from arbitrary - viewpoints, which we dub "nerfies". We evaluate our method by collecting data - using a - rig with two mobile phones that take time-synchronized photos, yielding train/validation - images of the same pose at different viewpoints. We show that our method faithfully - reconstructs non-rigidly deforming scenes and reproduces unseen views with high - fidelity. + TickleBot is designed not only as a research tool + but also as an educational resource. Its affordability and modularity make it suitable for + integration into university curricula, providing students with hands-on experience in robotics and AI integration. + Furthermore, as an open-source platform, TickleBot encourages community-driven development and + rapid dissemination of robotic technologies. Beyond education and research, its extensibility allows potential applications in manufacturing, + logistics, healthcare, and assistive services. Through TickleBot, we aim to democratize access to humanoid robotics and foster innovation by + bridging the gap between advanced AI models and practical, low-cost robotic systems.
+
- Using nerfies you can create fun visual effects. This Dolly zoom effect - would be impossible without nerfies since it would require going through a wall. + The robot arm recognizes a T-shirt placed on the table using its vision system, grasps it with the gripper, + and executes a folding sequence. After folding, the arm neatly places the garment in an organized stack.
+
+
+
- As a byproduct of our method, we can also solve the matting problem by ignoring - samples that fall outside of a bounding box during rendering. + Using object recognition, the robot identifies different waste items such as cans and paper + cups. It then classifies the objects by type and places each one into the corresponding recycling bin using + precise arm movements.
+
+
+
- We can also animate the scene by interpolating the deformation latent codes of two input - frames. Use the slider here to linearly interpolate between the left frame and the right - frame. -
+ Students assemble 3D-printed components with the low-cost SO-100 + robotic arm and power on the system in a ROS2 environment. + 여기에는 모바일 이동 과정 넣으면 좋을 듯 함 +
+
Start Frame
End Frame
@@ -385,14 +448,18 @@
+ - Using Nerfies, you can re-render a video from a novel - viewpoint such as a stabilized camera by playing back the training deformations. + On Jetson Orin nano(Jetpack 6.2.1), ROS2 nodes and the robot action model such as SmolVLA, ACT policy are executed, + and the robotic arm performs basic motion tests to verify integration.
+ + Students directly input commands or add new datasets, via teleoperation system using joystick and robopilot handles. + enabling the robotic arm to grasp and stack blocks as a practical learning exercise. +
+
+ + The NoMaD-based mobile base navigates to a new workspace(based on user input image) where the robotic arm resumes manipulation tasks, + demonstrating scalability for advanced research. +
+- Progressive Encoding for Neural Optimization introduces an idea similar to our windowed position encoding for coarse-to-fine optimization. + NoMaD: Navigation with Multimodal Diffusion A vision-based navigation model that enables robots to reach target locations using only RGB inputs.
- D-NeRF and NR-NeRF - both use deformation fields to model non-rigid scenes. + LeRobot An open-source library for robot learning that provides datasets, models, and training utilities for scalable robotic research.
- Some works model videos with a NeRF by directly modulating the density, such as Video-NeRF, NSFF, and DyNeRF + SmolVLA A vision-based navigation model that enables robots to reach target locations using only RGB inputs.
-- There are probably many more by the time you are reading this. Check out Frank Dellart's survey on recent NeRF papers, and Yen-Chen Lin's curated list of NeRF papers. +
+ Mobile ALOHA A low-cost mobile humanoid platform combining dual arms with a wheeled base, demonstrating household task automation.