A proposal for addition of FSDP + LoRA with different dtypes for different modules?

[tolgacangoz]

Hi @a-r-r-o-w, @sayakpaul, and community! Lately, I have been learning about parallelism strategies. And, I came across this note in the repo:

- Using FSDP1 requires that all FSDP flat parameters are of the same dtype. For LoRA training, we default lora parameters to fp32 and transformer parameters to dtype chosen by user. There seems to be no easy workaround than performing lora training in same type.

This problem attracted my attention: 1) It would directly support the democratization purposes of the repo. 2) This problem seemed quite interesting to me.
Can I research and work on the addition of this feature?