feat: add new custom resolver to fetch the reference from private repo #1810

AayushSaini101 · 2025-06-21T12:35:01Z

Resolves: #1796

Create a custom resolver to fetch the reference from the private repo
Add new command auth add for storing the repo name and repo token.
Tests and documents need to be updated

…and add new command auth add

github-actions · 2025-06-21T12:36:48Z

Changeset has been generated for this PR as part of auto-changeset workflow.

Please review the changeset before merging the PR.

---
'@asyncapi/cli': minor
---

feat:  add new custom resolver to fetch the reference from private repo

- 0213926: Add new custom resolver to fetch the reference from private template and add new command auth add
- 26037eb: Merge remote-tracking branch 'remote/master' into 1796-b
- 6473326: chore: separate cli and api

Signed-off-by: Shurtu-gal <[email protected]>
- d484f2f: update
- ab58365: remove invalid changes
- 932c09c: update package-lock.json
- ec5eb66: update package-lock.json
- 7e4b523: fix github jobs
- c7501a1: Fix elint issue
- 925b90a: remove uncessary files
- 5d651f7: remove invalid file
- 5d76066: update-changeset
- 5d89ce7: update-doc
- 3461288: update-suggestions
- 4f63af9: add new testcase
- 5d83ac7: add new testcases
- f1a4d39: add new test case for the validationService
- af11ce4: update test case
- 824bdb9: update changeset
- cf4d070: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 150865c: update validation service for the github
- 4c5be56: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- fcc864d: Update src/apps/api/middlewares/validation.middleware.ts

Co-authored-by: Fran Méndez <[email protected]>
- 2f69485: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 5328d62: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 407bacc: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 3b7159b: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- f601cf7: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>

If you are a maintainer or the author of the PR, you can change the changeset by clicking here

Tip

If you don't want auto-changeset to run on this PR, you can add the label skip-changeset to the PR or remove the changeset and change PR title to something other than fix: or feat:.

changeset-bot · 2025-06-21T12:36:51Z

🦋 Changeset detected

Latest commit: 25860cc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@asyncapi/cli	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

AayushSaini101 · 2025-06-21T12:44:54Z

Tests and documentation need to be updated, i will update after getting ack from this approach : )

Spec containing private repo:

Trying to parse using the existing version of the asyncapi cli:

New feature:

Add auth value for the github token:

Repo URL format: https://github.com/AayushSaini101/AayushSaini101/tree/main

Trying to fetch the private ref:

cc: @fmvilas @Souvikns @Shurtu-gal

fmvilas · 2025-06-21T13:30:16Z

src/core/parser.ts

+  schema: 'https',
+  order: 1,
+
+  canRead: (uri: URI) => uri.hostname() === 'raw.githubusercontent.com',


Why checking for GitHub here? 🤔 References can be in many places, not just GitHub, and even in such a case the GitHub deployment might be in a private domain, which is pretty common in medium to large companies.

fmvilas · 2025-06-21T13:33:53Z

src/core/parser.ts

+
+  read: async (uri: URI) => {
+    const url = uri.toString();
+    const normalized = normalizeRawGitUrl(uri);


What if the URL is not pointing to a GitHub repo? This could perfectly be a Schema Registry URL, an S3 bucket, some random URL, or even a GitLab or BitBucket URL.

@fmvilas, we can limit this PR for the GitHub repo only because we know proper mapping for this for example, when the user provide the URL in this format 'https://github.com/AayushSaini101/AayushSaini101/blob/main/user-signedup.yaml#/payload', we know we have to modify the URL in this format n https://api.github.com/repos/${owner}/${repo}/contents/${path}?ref=${branch} to obtain the data from the private template, but we don't about the Gitlabm. or other BitBucket and other services modification, we can start with the github private url approach, then we can start updating bit by bit according to the requirements from the users thanks

mmm 🤔 this is being applied to all URLs and normalizeRawGitUrl assumes it's a GitHub URL. I'm fine with only doing the mapping for GitHub for now but other URLs shouldn't be modified. You know what I mean?

yes, at this time we can consider with GithubURL only, in future we can update according the requirements, like for Gitlab and other services based on the requirements cc: @fmvilas

Just to be sure we're understanding each other. If I'm understanding it correctly, right now we're assuming all URLs are GitHub URLs, which is wrong because that would make parser stop working with custom URLs. In the normalizeRawGitUrl function we should be checking if it's actually a GitHub URL, otherwise, leave it intact. Mappings for GitLab and Bitbucket can wait but breaking all URLs except the GitHub ones is definitely not acceptable.

yes, i need to change the functionnaming properly i will make it generic so that we can normalise other url as well in the future based on the requirements

src/core/parser.ts

AayushSaini101 · 2025-07-02T06:33:32Z

/u

sonarqubecloud · 2025-07-02T06:34:26Z

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

AayushSaini101 · 2025-07-05T16:18:15Z

@fmvilas For the private Github URL should be in this format: like https://api.github.com/repos/AayushSaini101/AayushSaini101/contents/user-signedup.yaml, not direct URL like we simple copy the path of the URL from the private repo, this is the github url, not sure about the private repo, in short we need to compare each and every url and create special check to find whether it is matching with the stored pattern or not, we can start with the github and gitlab pattern then we can slowly slowly add the comparison format as we grow thanks

aeworxet · 2025-07-21T10:32:29Z

@asyncapi/bounty_team

AayushSaini101 · 2025-08-04T06:13:01Z

/u

fmvilas · 2025-08-04T19:39:51Z

@fmvilas For the private Github URL should be in this format: like https://api.github.com/repos/AayushSaini101/AayushSaini101/contents/user-signedup.yaml, not direct URL like we simple copy the path of the URL from the private repo, this is the github url, not sure about the private repo, in short we need to compare each and every url and create special check to find whether it is matching with the stored pattern or not, we can start with the github and gitlab pattern then we can slowly slowly add the comparison format as we grow thanks

@AayushSaini101 I'm not sure I follow. I think you can translate from one to another, right? For instance, if I give you https://github.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml (which is the URL I can copy from the browser), you can transform it to https://api.github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml or whatever format you need it to be, right? The idea is to make it really easy for the user.

Likewise, if my company has a Github installation behind the https://source.mycompany.com URL, then a URL like https://source.mycompany.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml should also be possible to resolve.

In short, picture this, you're working at a company and have an AsyncAPI or JSON/YAML file in a private repo. You want to reference it using $ref and it should just work out of the box. Sure, the user has to provide the necessary details to resolve these URLs but it should be fairly easy once it's configured.

AayushSaini101 · 2025-08-05T07:10:40Z

github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml

yes, i can tranform/configured the URL for the Github because i know the format for the accepted URL in the case of the Github, but for the case of another services like registry or any private registry what should be the correct URL that we need to consider for the custom resolver, that's why for this issue we can enable feature Github then in future according to the requirements we can add bit by bit @fmvilas

AayushSaini101 · 2025-08-11T12:57:52Z

/u

AayushSaini101 · 2025-09-19T10:23:52Z

@AayushSaini101 let me know if you need guidance.

@fmvilas tried to added some test cases thanks: : )

Shurtu-gal

The code looks fine to me. Just wanted to know the use cases for this (is it just limited to github private repo).

Also @fmvilas what should be the proposed way of testing the auth?

fmvilas

Left comment on the tests.

IMHO, you should mock the parser functionality that's trying to make an HTTP request. When you're testing the private URL, it should fake a 401 Unauthorized HTTP response. Perhaps a 404 Not found response too, since some servers will give you that a 404 instead of a 401.

The way tests are right now, they're not testing the new feature. Let me know if you need guidance for testing.

fmvilas · 2025-09-22T09:48:24Z

test/unit/services/config.service.test.ts

@@ -0,0 +1,88 @@
+import { expect } from 'chai';


This file is definitely not needed. It's not testing any part of our code.

fmvilas · 2025-09-22T09:52:44Z

test/unit/services/validation.service.test.ts

+      }
+    });
+
+    it('should validate document with private GitHub reference', async () => {


This is testing that the document fails to validate because the $ref is in a private URL and we haven't configured how to resolve it. IMHO, the name of the test is not reflecting what this is testing.

I suggest something like the following:

Suggested change

it('should validate document with private GitHub reference', async () => {

it('should fail to validate document with private GitHub reference when not properly configured', async () => {

fmvilas · 2025-09-22T09:55:18Z

test/unit/services/validation.service.test.ts

+      const result = await validationService.validateDocument(specFile, options);
+
+      // The validation succeeds but the document is invalid due to unresolved ref
+      expect(result.success).to.equal(true);


If this test is passing we're doing something wrong. result.success shouldn't be true because a key part of the document couldn't even be resolved.

For the case of the githubURL private one here is the logs getting success and page not found one @fmvilas

Don't you find it weird that success === true but the document is invalid?

fmvilas · 2025-09-22T09:55:51Z

test/unit/services/validation.service.test.ts

+      expect(result.success).to.equal(true);
+      if (result.success) {
+        expect(result.data).to.have.property('status');
+        expect(result.data?.status).to.equal('invalid');


Having result.success === true and result.data?.status === 'invalid' is definitely weird. Don't you think so?

fmvilas

More comments as before I was focusing on the tests but now I'm realizing the implementation is not there yet.

fmvilas · 2025-09-22T10:05:06Z

src/domains/services/validation.service.ts

+
+    const authInfo = await ConfigService.getAuthForUrl(url);
+
+    if (url.includes('github.com') && url.includes('/blob/')) {


The following url would pass this check and it shouldn't: https://mycompany.com/something?url=github.com/org/repo/blob/whatever.

You should deconstruct the URL and check for:

host === 'github.com'

path starts with (or contains) '/blob/'

Make sense thanks : )

Added thanks

fmvilas · 2025-09-22T10:10:43Z

src/domains/services/validation.service.ts

+        throw new Error(`Failed to fetch GitHub URL: ${url} - ${res.statusText}`);
+      }
+      return await res.text();
+    } else {


What if the URL is not from GitHub but it's still private? That should be covered too. Actually, many big companies don't have their GitHub installation on github.com but on their own domains. Or maybe they want to use it with a schema registry like the one from Confluent.

Just to clarify, what I meant by "limiting to GitHub in this PR" is the ability to translate URLs from user-facing style to API URLs, as we're doing on line 102.

But if a URL is added to config.auth, we should attach the Authorization header to the request.

src/domains/services/validation.service.ts

fmvilas · 2025-09-22T10:16:59Z

The code looks fine to me. Just wanted to know the use cases for this (is it just limited to github private repo).

To clarify, this is not just for GitHub private repos. The URL translation feature is just for GitHub though.

Also @fmvilas what should be the proposed way of testing the auth?

We should mock the resolver and fake the status code and response we get. We should be testing that unless we pass the Authorization header with a specific value (a random value), the resolver will respond with 401 or 404. Basically, imitate what a real server would do without making the actual HTTP call.

Co-authored-by: Fran Méndez <[email protected]>

AayushSaini101 · 2025-09-22T11:27:04Z

The code looks fine to me. Just wanted to know the use cases for this (is it just limited to github private repo).

To clarify, this is not just for GitHub private repos. The URL translation feature is just for GitHub though.

Also @fmvilas what should be the proposed way of testing the auth?

We should mock the resolver and fake the status code and response we get. We should be testing that unless we pass the Authorization header with a specific value (a random value), the resolver will respond with 401 or 404. Basically, imitate what a real server would do without making the actual HTTP call.

Thanks for the clarification @fmvilas I thought we just needed to add for the GitHub support only, it can extend other private URL as well:

} else {
      if (authInfo) {
        headers['Authorization'] = `${authInfo.authType} ${authInfo.token}`;
        Object.assign(headers, authInfo.headers); // merge custom headers
      }
      const res = await fetch(url, { headers });
      if (!res.ok) {
        throw new Error(`Failed to fetch URL: ${url} - ${res.statusText}`);
      }
      return await res.text();
    }

fmvilas

Another round of review. If I were you, I'd start with the tests. Write them first. Even if they don't pass. Write the tests as you'd like them to pass. Then adjust the logic to make the tests pass. It becomes way easier this way.

src/domains/services/validation.service.ts

src/apps/api/middlewares/validation.middleware.ts

src/domains/services/validation.service.ts

fmvilas · 2025-09-24T09:49:26Z

src/domains/services/validation.service.ts

+};
+
+/**
+ * Custom GitHub URL resolver for private repositories


Suggested change

* Custom GitHub URL resolver for private repositories

* Custom URL resolver for private repositories

src/domains/services/validation.service.ts

fmvilas · 2025-09-24T10:05:33Z

src/domains/services/validation.service.ts

+      url = createHttpWithAuthResolver().convertGitHubWebUrl(url);
+    }
+
+    // Only require authentication for GitHub URLs


This comment is not true. We should not require authentication only for GitHub URLs.

Make sense thanks

src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>

sonarqubecloud · 2025-09-24T10:44:47Z

Quality Gate passed

Issues
22 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Add new custom resolver to fetch the reference from private template …

0213926

…and add new command auth add

AayushSaini101 requested review from Souvikns, Amzani, Shurtu-gal and asyncapi-bot-eve as code owners June 21, 2025 12:35

github-project-automation bot added this to CLI - Kanban Jun 21, 2025

github-project-automation bot moved this to To Triage in CLI - Kanban Jun 21, 2025

AayushSaini101 changed the title ~~Add new custom resolver to fetch the reference from private repo~~ feaat: add new custom resolver to fetch the reference from private repo Jun 21, 2025

AayushSaini101 changed the title ~~feaat: add new custom resolver to fetch the reference from private repo~~ feat: add new custom resolver to fetch the reference from private repo Jun 21, 2025

chore: add changeset for PR asyncapi#1810

c685ac2

fmvilas requested changes Jun 21, 2025

View reviewed changes

github-project-automation bot moved this from To Triage to Backlog in CLI - Kanban Jun 21, 2025

Merge branch 'master' into 1796-b

181dbb9

asyncapi-bot added the bounty AsyncAPI Bounty program related label label Jul 21, 2025

aeworxet added this to Bounty Program Jul 21, 2025

aeworxet moved this to In Progress in Bounty Program Jul 21, 2025

Merge branch 'master' into 1796-b

82ac26b

aeworxet mentioned this pull request Aug 6, 2025

[FEATURE] Pass authentication/authorization to $ref HTTP resolvers #1796

Open

2 tasks

Merge branch 'master' into 1796-b

0a4c2ed

AayushSaini101 and others added 2 commits September 19, 2025 15:50

add new test case for the validationService

f1a4d39

chore: add changeset for PR asyncapi#1810

f729636

update test case

af11ce4

asyncapi-bot pushed a commit to AayushSaini101/cli that referenced this pull request Sep 19, 2025

chore: add changeset for PR asyncapi#1810

a3230f9

update changeset

824bdb9

AayushSaini101 force-pushed the 1796-b branch from a3230f9 to 824bdb9 Compare September 19, 2025 10:37

chore: add changeset for PR asyncapi#1810

0211fed

AayushSaini101 requested a review from fmvilas September 19, 2025 11:50

Shurtu-gal previously approved these changes Sep 19, 2025

View reviewed changes

fmvilas requested changes Sep 22, 2025

View reviewed changes

Update src/domains/services/validation.service.ts

cf4d070

Co-authored-by: Fran Méndez <[email protected]>

AayushSaini101 dismissed Shurtu-gal’s stale review via cf4d070 September 22, 2025 10:32

chore: add changeset for PR asyncapi#1810

95df8d3

AayushSaini101 and others added 2 commits September 22, 2025 16:59

update validation service for the github

150865c

chore: add changeset for PR asyncapi#1810

e04600f

fmvilas requested changes Sep 24, 2025

View reviewed changes

AayushSaini101 and others added 9 commits September 24, 2025 16:03

Update src/domains/services/validation.service.ts

4c5be56

Co-authored-by: Fran Méndez <[email protected]>

Update src/apps/api/middlewares/validation.middleware.ts

fcc864d

Co-authored-by: Fran Méndez <[email protected]>

chore: add changeset for PR asyncapi#1810

6c66199

Update src/domains/services/validation.service.ts

2f69485

Co-authored-by: Fran Méndez <[email protected]>

Update src/domains/services/validation.service.ts

5328d62

Co-authored-by: Fran Méndez <[email protected]>

Update src/domains/services/validation.service.ts

407bacc

Co-authored-by: Fran Méndez <[email protected]>

Update src/domains/services/validation.service.ts

3b7159b

Co-authored-by: Fran Méndez <[email protected]>

Update src/domains/services/validation.service.ts

f601cf7

Co-authored-by: Fran Méndez <[email protected]>

chore: add changeset for PR asyncapi#1810

25860cc

	it('should validate document with private GitHub reference', async () => {
	it('should fail to validate document with private GitHub reference when not properly configured', async () => {


		const authInfo = await ConfigService.getAuthForUrl(url);

		if (url.includes('github.com') && url.includes('/blob/')) {

	* Custom GitHub URL resolver for private repositories
	* Custom URL resolver for private repositories

Uh oh!

feat: add new custom resolver to fetch the reference from private repo #1810

Are you sure you want to change the base?

feat: add new custom resolver to fetch the reference from private repo #1810

Uh oh!

Conversation

AayushSaini101 commented Jun 21, 2025

Uh oh!

github-actions bot commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changeset has been generated for this PR as part of auto-changeset workflow.

Uh oh!

changeset-bot bot commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

AayushSaini101 commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New feature:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AayushSaini101 Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AayushSaini101 commented Jul 2, 2025

Uh oh!

sonarqubecloud bot commented Jul 2, 2025

Quality Gate passed

Uh oh!

AayushSaini101 commented Jul 5, 2025

Uh oh!

aeworxet commented Jul 21, 2025

Uh oh!

AayushSaini101 commented Aug 4, 2025

Uh oh!

fmvilas commented Aug 4, 2025

Uh oh!

AayushSaini101 commented Aug 5, 2025

Uh oh!

AayushSaini101 commented Aug 11, 2025

Uh oh!

AayushSaini101 commented Sep 19, 2025

Uh oh!

Shurtu-gal left a comment

Choose a reason for hiding this comment

Uh oh!

fmvilas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fmvilas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

github-actions bot commented Jun 21, 2025 •

edited

Loading

changeset-bot bot commented Jun 21, 2025 •

edited

Loading

AayushSaini101 commented Jun 21, 2025 •

edited

Loading

AayushSaini101 Sep 2, 2025 •

edited

Loading

fmvilas left a comment •

edited

Loading