Integration: Bytez Chat Model Provider #1175
Conversation
…Params in the context's requestOptions.
|
Description🔄 What ChangedSummary By MatterAI
This Pull Request focuses on integrating the Bytez Chat Model Provider and enhancing existing plugin and provider configurations. Key changes include:
🔍 Impact of the ChangeThese changes enable the integration of the Bytez Chat Model Provider, expanding the platform's capabilities. The refactoring in the WalledAI plugin improves code readability and maintainability by using named constants for default lists and activating necessary authentication. The parameter renaming in Azure AI Inference enhances configuration clarity. Overall, the PR contributes to a more robust and extensible system. 📁 Total Files Changed
🧪 Test Added
🔒Security VulnerabilitiesNo direct security vulnerabilities were detected in this Pull Request. The activation of the API key for WalledAI is a functional requirement for authentication and is handled via standard bearer token practices. MotivationThis PR aims to integrate the Bytez Chat Model Provider into the platform, expanding the range of available LLM providers. It also includes improvements to the WalledAI guardrails plugin for better default handling and authentication, and refines parameter naming for Azure AI Inference to enhance consistency and clarity. Type of Change
How Has This Been Tested?
Screenshots (if applicable)N/A Checklist
Related IssuesN/A Tip Quality Recommendations
Tanka Poem ♫
Sequence DiagramsequenceDiagram
participant App as Application
participant PH as PluginHandler (WalledAI)
participant WAAPI as WalledAI API
App->>PH: Call handler(context, parameters, 'beforeRequestHook')
PH->>PH: Prepare request body with parameters.text, parameters.text_type, etc.
PH->>PH: Use parameters.greetings_list || DEFAULT_GREETINGS_LIST
PH->>PH: Use parameters.pii_list || DEFAULT_PII_LIST
PH->>PH: Use parameters.compliance_list || []
PH->>PH: Set Authorization header with Bearer token from parameters.credentials.apiKey
PH->>WAAPI: POST /v1/guardrail/moderate (text, text_type, generic_safety_check, greetings_list, pii_list, compliance_list)
WAAPI-->>PH: Response (verdict, data, error)
PH-->>App: Return GuardrailCheckResult
|
![matter-code-review[bot]](https://avatars.githubusercontent.com/in/474549?s=60&v=4){kind=small}
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
We do not want to bloat the code base with packages that aren't needed. It should be sufficient. It already does this where necessary, otherwise whatever is passed from the server is passed to the client Overkill, either fetch is going to fail, or there is an upstream error, either way it will get reported to the client. Perhaps in the future... We currently stream character by character, we do not return JSON chunks. In the future we may update this. YAGNI and overkill, bytez manages its own integration code. |
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
@narengogi @VisargD Requested changes have been made, and I dropped the LRU. Pls lmk if you need any further changes. |
…rm to openai compliant spec.
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
@narengogi @VisargD Pinging for visibility, we should be ready to rock 😄 |
|
@narengogi @VisargD Hey again folks! We've created the docs that correspond to this pr on your docs repos, very much appreciate you taking the time to look this all over 😄 |
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
Description
Hi all! 👋
We're Bytez, the largest model provider on the internet! We may also be one of the cheapest if not the cheapest.
We'd love to integrate with PortKey. Please see the changed files and let me know if anything needs to change.
I'd like to point out that we do a check against our api to see if a model is a "chat" model. This is stored in a simple cache that is just an object. If that's going to be a problem due to having an unbounded ceiling in terms of memory utilization pls lmk, and I will convert it to a LRU with 100 entries.
Our API's input signature is a bit more bespoke than other model providers, please lmk if the custom requestHandler I have is sufficient, or if there's an easier way to do what I've done.
Bonus feedback: Ya'll need an integration guide! Would be immensely useful 😄
Motivation
We'd love to be integrated in PortKey!
Type of Change
How Has This Been Tested?
Screenshots (if applicable)
Checklist
Related Issues