[WIP][AQUA] GPU Shape Recommendation

[elizjo]

Wrote an additional POST API and aqua command for recommending GPU shapes for a particular model

ads aqua recommend shapes --model_ocid 'ocid1.datasciencemodel.oc1.<ocid>' "" 

Returns

{
  "model_ocid" : "ocid1.datasciencemodel.oc1.<ocid>"
}

Returns

{                                                                                                                                                                                                                                            
    "display_name": "Almawave/Velvet-14B",
    "recommendations": [
        {
            "shape_details": {
                "available": false,
                "core_count": null,
                "memory_in_gbs": null,
                "name": "BM.GPU.MI300X.8",
                "shape_series": "GPU",
                "gpu_specs": {
                    "gpu_memory_in_gbs": 1536,
                    "gpu_count": 8,
                    "gpu_type": "MI300X",
                    "quantization": [
                        "fp8",
                        "gguf"
                    ],
                    "ranking": {
                        "cost": 90,
                        "performance": 90
                    }
                }
            },
            "configurations": [
                {
                    "model_details": {
                        "model_size_gb": 28.16,
                        "kv_cache_size_gb": 26.84,
                        "total_model_gb": 55.0
                    },
                    "deployment_params": {
                        "quantization": "bfloat16",
                        "max_model_len": 131072,
                        "params": ""
                    },
                    "recommendation": "No override PARAMS needed. \n\nModel fits well within the allowed compute shape (55.0GB used / 1536.0GB allowed)."
                }
            ]
        }
    ],
     "troubleshoot": ""
}

Status: business logic works, API works, unit tests finished, rich diff CLI table finished

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[mrDzurb]

Let's add FP8 for the A10 shapes as well.

[mrDzurb]

Why do we need this?

[mrDzurb]

Maybe we can name it as __shape_handler?

[mrDzurb]

It would be better to move this into constants.

[mrDzurb]

Let's move "float32" to constants

[mrDzurb]

Let's move the numbers like 16384 to constants and add some description there

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[mrDzurb]

Would it make sense to move this handler to deployment_handler.py under the AquaDeploymentHandler class?

We already have a list_shape method there, so I suggest adding a new method called list_recommended_shape.

The endpoint path could be /aqua/deployments/recommended_shapes.

The main reason for this change is that we’ll be implementing similar logic for service models as well. Having a unified handler will allow us to return the recommended list of shapes for both service and custom models in a consistent way.

[mrDzurb]

Looks like this const variable is not used?

[mrDzurb]

If we are not planning to use this method outside of the class, it would be better to make it as private.
def _validate_model_ocid. Check for the others.

[mrDzurb]

NIT: Maybe we can name it as AquaShapeRecommendApp?

[mrDzurb]

Technically, the shape recommendation feature falls under the broader Model Deployment functionality. Instead of creating a new App class, I suggest we implement the business logic in a regular helper class and then integrate it into the existing AquaDeploymentApp.

Since AquaDeploymentApp already includes a list_shapes method, we can add a new method called list_recommended_shapes to maintain backward compatibility.

Looking ahead, if we need similar logic for other modules like fine-tuning or evaluation, we can follow the same pattern and add the corresponding methods in those apps as well.

As for naming, we could rename the current AquaRecommendApp to AquaShapeRecommend. I don't think we need to inherit from AquaApp in this case, since it's primarily a utility class focused on recommendation logic.

[mrDzurb]

Maybe to make it a bit more generic - which_shape?

[mrDzurb]

I think we can make it private?

[mrDzurb]

Technically, the shape recommendation feature falls under the broader Model Deployment functionality. Instead of creating a new App class, I suggest we implement the business logic in a regular helper class and then integrate it into the existing AquaDeploymentApp.

Since AquaDeploymentApp already includes a list_shapes method, we can add a new method called list_recommended_shapes to maintain backward compatibility.

Looking ahead, if we need similar logic for other modules like fine-tuning or evaluation, we can follow the same pattern and add the corresponding methods in those apps as well.

As for naming, we could rename the current AquaRecommendApp to AquaShapeRecommend. I don't think we need to inherit from AquaApp in this case, since it's primarily a utility class focused on recommendation logic.

[mrDzurb]

To avoid calling AquaDeploymentApp().list_shapes, I think it would be cleaner to add a shapes method directly to the OCIDataScienceModelDeployment class.

For example:

class OCIDataScienceModelDeployment(
    OCIDataScienceMixin,
    oci.data_science.models.ModelDeployment,
):

  @classmethod
  def shapes(
      cls,
      compartment_id: str = None,
      **kwargs,
  ) -> List[oci.data_science.models.ModelDeploymentShapeSummary]:
      return oci.pagination.list_call_get_all_results(
          cls().client.list_model_deployment_shapes,
          compartment_id or COMPARTMENT_ID,
          **kwargs
      ).data

This makes the logic more self-contained within the deployment model and avoids relying on external app instances just to retrieve shape info.

The usage could be:

user_shapes = OCIDataScienceModelDeployment.shapes(compartment_id=compartment_id)

[mrDzurb]

The oci.data_science.models.ModelDeploymentShapeSummary already contains the def shape_series(self, shape_series):. Maybe we can use it?