Added multi-language support to the tokenizer

[shankeleven]

Solves #13

regex patterns have been modified to be unicode aware

just went through the regex patterns , thought it was feasible to include other language tokens by modifying the regex pattern to be unicode aware

[A-Za-z]+ -> \p{L}+ [0-9]+ -> \p{N}+ (or \p{Nd}+)
[^A-Za-z0-9\s]+ -> [^\p{L}\p{N}\s]+

"Hello, 世界!", \p{L}+ matches "Hello" and "世界"

have added tests for different languages
added a new depency regex , as it works better with unicode compared to re

[github-actions]

Manual validation results:

Size: 205 lines
Template: ✗
Linked issue: ✗

Run ID: 16790882125

[github-actions]

Manual validation results:

Size: 205 lines
Template: ✗
Linked issue: ✗

Run ID: 16791204003

[github-actions]

Manual Validation Results

Status: ❌ Failed

Check	Status	Details
PR Size	✅	205 lines
Template	❌	Missing required sections
Linked Issue	❌	Missing Fixes/Closes #XXX

Errors:

• ❌ Missing PR template sections: # Description, Fixes #, # How Has This Been Tested?, # Checklist
• ❌ No linked issue found

View workflow run

[github-actions]

Manual Validation Results

Status: ❌ Failed

Check	Status	Details
PR Size	✅	205 lines
Template	❌	Missing required sections
Linked Issue	❌	Missing Fixes/Closes #XXX

Errors:

• ❌ Missing PR template sections: # Description, Fixes #, # How Has This Been Tested?, # Checklist
• ❌ No linked issue found

View workflow run

[github-actions]

⚠️ Branch Update Required

Your branch is 24 commits behind main. Please update your branch to ensure CI checks run with the latest code:

git fetch origin main
git merge origin/main
git push

Note: Enable "Allow edits by maintainers" to allow automatic updates.

[github-actions]

⚠️ Branch Update Required

Your branch is 51 commits behind main. Please update your branch to ensure CI checks run with the latest code:

git fetch origin main
git merge origin/main
git push

Note: Enable "Allow edits by maintainers" to allow automatic updates.

[github-actions]

⚠️ Branch Update Required

Your branch is 86 commits behind main. Please update your branch to ensure CI checks run with the latest code:

git fetch origin main
git merge origin/main
git push

Note: Enable "Allow edits by maintainers" to allow automatic updates.

[github-actions]

⚠️ Branch Update Required

Your branch is 98 commits behind main. Please update your branch to ensure CI checks run with the latest code:

git fetch origin main
git merge origin/main
git push

Note: Enable "Allow edits by maintainers" to allow automatic updates.

[github-actions]

⚠️ Branch Update Required

Your branch is 106 commits behind main. Please update your branch to ensure CI checks run with the latest code:

git fetch origin main
git merge origin/main
git push

Note: Enable "Allow edits by maintainers" to allow automatic updates.

[github-actions]

⚠️ Branch Update Required

Your branch is 107 commits behind main. Please update your branch to ensure CI checks run with the latest code:

git fetch origin main
git merge origin/main
git push

Note: Enable "Allow edits by maintainers" to allow automatic updates.

[github-actions]

⚠️ Branch Update Required

Your branch is 109 commits behind main. Please update your branch to ensure CI checks run with the latest code:

git fetch origin main
git merge origin/main
git push

Note: Enable "Allow edits by maintainers" to allow automatic updates.

[github-actions]

⚠️ Branch Update Required

Your branch is 110 commits behind main. Please update your branch to ensure CI checks run with the latest code:

git fetch origin main
git merge origin/main
git push

Note: Enable "Allow edits by maintainers" to allow automatic updates.

[github-actions]

⚠️ Branch Update Required

Your branch is 111 commits behind main. Please update your branch to ensure CI checks run with the latest code:

git fetch origin main
git merge origin/main
git push

Note: Enable "Allow edits by maintainers" to allow automatic updates.