Falcon H1 Model Support

[sbhavani]

Add support for Falcon H1 models, following the implementation in Megatron-LM PR #1785

The upstream Megatron-LM implementation provides the foundation, but Megatron Bridge needs additional integration work to support:

1. Model Provider Implementation: Create FalconH1ModelProvider similar to existing Llama provider
2. Hugging Face Bridge Integration: Implement bidirectional conversion between HuggingFace Falcon H1 checkpoints and Megatron format
3. Configuration Management: Add Falcon H1-specific configuration handling and parameter mappings
4. Training Recipes: Provide pre-configured training recipes compatible for Falcon H1

[dhiaEddineRhaiem]

Hi @sbhavani and all,

I understand you'd like the Falcon H1 integration in Megatron-Bridge rather than core. However, there's a dependency consideration:

Current Architecture:

• Megatron-Bridge's MambaProvider imports MambaModel from Megatron-LM core here
• My PR extends core's MambaModel to support the parallel_hybrid layer needed for Falcon H1
• Bridge's Falcon H1 implementation will require this enhanced MambaModel as its foundation

Proposed Approach:

1. Foundation: Merge PR in core to provide parallel_hybrid layer support in MambaModel
2. Bridge Integration: I can then implement the Bridge-specific components you outlined:
   • FalconH1ModelProvider
   • HuggingFace conversion utilities
   • Configuration management
   • Training recipes

This approach will maintains clean separation in which core provides the architectural foundation, while Bridge handles the model-specific integration and conversion logic.

Would this approach work?

[sbhavani]

CC @ntajbakhsh