Create README.md

README.md

@@ -0,0 +1,67 @@
+---
+base_model:
+- Wan-AI/Wan2.1-T2V-1.3B
+- Wan-AI/Wan2.1-T2V-1.3B-Diffusers
+library_name: diffusers
+pipeline_tag: text-to-video
+---
+
+<p align="center">
+  <img src="https://github.com/mkturkcan/suturingmodels/blob/main/static/images/title.svg?raw=true"  />
+</p>
+
+# Towards Suturing World Models (Wan, t2v)
+
+<p align="center">
+  <img src="https://github.com/mkturkcan/suturingmodels/blob/main/static/images/lora_sample.jpg?raw=true"  />
+</p>
+
+
+This repository hosts the fine-tuned Wan2.1-T2V-1.3B text-to-video (t2v) diffusion model specialized for generating realistic robotic surgical suturing videos, capturing fine-grained sub-stitch actions including needle positioning, targeting, driving, and withdrawal. The model can differentiate between ideal and non-ideal surgical techniques, making it suitable for applications in surgical training, skill evaluation, and autonomous surgical system development.
+
+## Model Details
+
+- **Base Model**: Wan2.1-T2V-1.3B
+- **Resolution**: 768×512 pixels (Adjustable)
+- **Frame Length**: 49 frames per generated video (Adjustable)
+- **Fine-tuning Method**: Low-Rank Adaptation (LoRA)
+- **Data Source**: Annotated laparoscopic surgery exercise videos (∼2,000 clips)
+
+## Usage Example
+
+```python
+import torch
+from diffsynth import ModelManager, WanVideoPipeline, save_video, VideoData
+
+
+model_manager = ModelManager(torch_dtype=torch.bfloat16, device="cpu")
+model_manager.load_models([
+    "../Wan2.1-T2V-1.3B/diffusion_pytorch_model.safetensors",
+    "../Wan2.1-T2V-1.3B/models_t5_umt5-xxl-enc-bf16.pth",
+    "../Wan2.1-T2V-1.3B/Wan2.1_VAE.pth",
+])
+model_manager.load_lora("mehmetkeremturkcan/Suturing-WAN-T2V", lora_alpha=1.0)
+pipe = WanVideoPipeline.from_model_manager(model_manager, device="cuda")
+pipe.enable_vram_management(num_persistent_param_in_dit=None)
+
+video = pipe(
+    prompt="A needledrivingnonideal clip, generated from a backhand task.",
+    num_inference_steps=50,
+    tiled=True
+)
+save_video(video, "video.mp4", fps=30, quality=5)
+```
+
+## Applications
+- **Surgical Training**: Generate demonstrations of both ideal and non-ideal surgical techniques for training purposes.
+- **Skill Evaluation**: Assess surgical skills by comparing actual procedures against model-generated standards.
+- **Robotic Automation**: Inform autonomous surgical robotic systems for real-time guidance and procedure automation.
+
+## Quantitative Performance
+| Metric                  | Performance |
+|-------------------------|---------------|
+| L2 Reconstruction Loss | 0.0667   |
+| Inference Time         | ~360 seconds per video |
+
+## Future Directions
+Further improvements will focus on increasing model robustness, expanding the dataset diversity, and enhancing real-time applicability to robotic surgical scenarios.