Sketch2Anim is towards Transferring Sketch Storyboards into 3D Animation.
What is Sketch2Anim?
Storyboarding is a common practice in 3D animation, where animators use 2D sketches as references to manually create motions—a process that is both time-consuming and requires significant expertise. To automate this workflow, we introduce Sketch2Anim, a novel framework that translates 2D storyboard sketches into 3D animations through conditional motion synthesis. Sketch2Anim features a multi-conditional motion generator, leveraging 3D keyposes, joint trajectories, and action words for precise motion control. To bridge the 2D-3D domain gap, we also propose a neural mapper that aligns 2D sketches with 3D motion constraints in a shared embedding space, enabling direct 2D-to-3D animation control. Our method produces high-quality 3D motions and naturally supports interactive editing. Extensive experiments and user studies confirm the effectiveness of our approach.
Sketch2Anim Function
Motion Retrieval: We adapt the recent state-of-the-art method TMR, which retrieves the closest motion clip based on text input. We extend it to additionally support 2D keyposes and joint trajectories.
Lift-and-Control: We leverage an additional network to lift 2D keyposes and trajectories into 3D representations for conditional motion generation.
Direct 2D-to-Motion: We train a multi-conditional motion diffusion model directly on 2D keyposes, joint trajectories, and action.