Hugging Face publishes LoRA fine-tuning guide and scripts for open 3D model TRELLIS.2

Hugging Face has published a comprehensive guide and accompanying repository for fine-tuning TRELLIS.2, an open-source 3D object generation model, using Low-Rank Adaptation (LoRA) techniques. The release includes Docker containers and scripts that can run both on local GPU hardware and through Hugging Face Jobs cloud service, lowering the barrier for developers to adapt the model to specific object categories. The guide demonstrates fine-tuning on a subset of Google Scanned Objects (GSO) containing 188 footwear items, targeting improved geometric accuracy in generated shoe soles.

The technical walkthrough explains TRELLIS.2's five-stage pipeline and identifies which components to adapt versus freeze during training. Rather than fine-tuning the entire model, the approach adds LoRA adapters only to the sparse-structure flow diffusion transformer and the shape flow diffusion transformer, which control coarse and detailed geometry respectively. The texture flow DiT remains frozen since the demonstration dataset lacks the separate material maps required for that stage. This selective adaptation strategy, implemented through Hugging Face's PEFT library, reduces memory requirements and training compute while preserving the base model's existing material generation capabilities.

Pipeline architecture and selective fine-tuning strategy

• Unified latent representation: TRELLIS.2 uses an internal O-Voxel format, a sparse voxel grid retaining only surface voxels with local geometry and material attributes, enabling CPU-to-GPU conversion in seconds and efficient sparse tensor computation using approximately 10% of grid voxels.
• LoRA adapter placement: Adapters are applied exclusively to the sparse-structure flow DiT (16³ × 8 latent to 64³ occupancy grid) and the shape flow DiT (512³ O-Voxel geometry generation), leaving the texture flow DiT and VAE decoders frozen.
• Sparse vs. dense transformers: The sparse-structure flow DiT operates as a dense transformer, while both the shape flow and texture flow DiTs are sparse transformers that avoid computation on empty voxels.
• Input preprocessing pipeline: BiRefNet removes image backgrounds before DINOv3-L encodes images into feature tokens that guide all three generation stages; DINOv3 access requires gated model approval through Hugging Face.
• Containerized deployment: The provided Docker containers support both Hugging Face Jobs cloud execution and local GPU training, with dataset meshes expected to be hosted on the Hugging Face Hub.

Why it matters

This release provides a practical reference implementation for parameter-efficient fine-tuning of 3D generative models, an area where open-source tooling has lagged behind 2D image and text model ecosystems. By documenting which pipeline stages to freeze and why, Hugging Face enables independent developers and researchers to adapt TRELLIS.2 for robotics simulation, game asset generation, or specialized manufacturing workflows without requiring full model retraining or proprietary infrastructure. The containerized approach and compatibility with consumer GPU setups through local Docker execution reduces dependency on cloud compute providers, though the gated DINOv3 requirement creates a partial access friction point.