Hindi TN 2.0 - Accuracy Enhancements & New Telephone Class Integration #294

ngachchi · 2025-06-25T05:47:22Z

What does this PR do ?

This PR introduces Hindi Text Normalization 2.0, which features substantial accuracy improvements across multiple classes and the addition of a new Telephone class. It also integrates culturally relevant linguistic constructs to enhance natural language understanding.

Accuracy Improvements by Class:

Text Class	Accuracy
Cardinal	89.86%
Money	99.04%
Time	92.61%
Date	87.18%
Ordinal	38.46%
Decimal	100.00%
Fraction	99.66%
Measure	96.26%
Telephone	99.13%
Address	18.99%

Key Enhancements:

New Class: Telephone
- Added support for STD codes and Indian country code (+91) in landline number normalization.
- Classification based on keywords for pincode and the last four digits of credit cards to extend within the telephone class.
Linguistic Enrichment:
- Incorporated quarterly terms frequently used in Hindi for Fraction, Measure, and Time classes:
  - Savva (सवा), Saadhe (साढ़े), Ponne (पौने), Dedh (डेढ़), Dhai (ढाई)

Before your PR is "Ready for review"

Pre checks:

PR Type:

New Feature
Bugfix
Documentation
Test

If you haven't finished some of the above items you can still open "Draft" PR.

* Future Implementations for classes - Measure, Money, and Date (NVIDIA#258) * Future Implementations for classes - Measure, Money, and Date Signed-off-by: Namrata Gachchi <[email protected]> * Resolved the conflicts with mm_yyyy and date ranges and added the previously removed failing test cases. Signed-off-by: Namrata Gachchi <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed the unused empty string implementation Signed-off-by: Namrata Gachchi <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor fixes for the tagger files Signed-off-by: Namrata Gachchi <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * reformatted decimal final graph Signed-off-by: Namrata Gachchi <[email protected]> * incorporated the suggestion for decimal graph Signed-off-by: Namrata Gachchi <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Century implementations Signed-off-by: Namrata Gachchi <[email protected]> * Working on the yyyy format for the date class Signed-off-by: Namrata Gachchi <[email protected]> * reverted yyyy code Signed-off-by: Namrata Gachchi <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * working on future implementations Signed-off-by: Namrata Gachchi <[email protected]> * working on improving the date class accuracy Signed-off-by: Namrata Gachchi <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added year prefix for the date class Signed-off-by: Namrata Gachchi <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * working on the commma cases for date class Signed-off-by: Namrata Gachchi <[email protected]> * minor fixes Signed-off-by: Namrata Gachchi <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * implemented mixed fractions Signed-off-by: Namrata Gachchi <[email protected]> * rectified the test case Signed-off-by: Namrata Gachchi <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * working on quarterly measurements Signed-off-by: Namrata Gachchi <[email protected]> * reformatted the prefixes and suffixes for date tagger class Signed-off-by: Namrata Gachchi <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * replaced text tag with era tag for the date class Signed-off-by: Namrata Gachchi <[email protected]> * Removed the text tag reference from date class verbalizer Signed-off-by: Namrata Gachchi <[email protected]> --------- Signed-off-by: Namrata Gachchi <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * update jenkins cache Signed-off-by: Mariana Graterol Fuenmayor <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Potential fix for code scanning alert no. 821: Unused local variable Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: Mariana <[email protected]> --------- Signed-off-by: Namrata Gachchi <[email protected]> Signed-off-by: Mariana Graterol Fuenmayor <[email protected]> Signed-off-by: Mariana <[email protected]> Co-authored-by: Namrata Gachchi <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Signed-off-by: Namrata Gachchi <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <[email protected]>

nemo_text_processing/text_normalization/hi/taggers/date.py

nemo_text_processing/text_normalization/hi/taggers/telephone.py

nemo_text_processing/text_normalization/hi/verbalizers/fraction.py

Signed-off-by: Namrata Gachchi <[email protected]>

for more information, see https://pre-commit.ci

ngachchi

@mgrafu, could you please review this PR?

…e telephone class Signed-off-by: Namrata Gachchi <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <[email protected]>

for more information, see https://pre-commit.ci

nemo_text_processing/text_normalization/hi/data/telephone/STD_codes.tsv

Signed-off-by: Namrata Gachchi <[email protected]>

for more information, see https://pre-commit.ci

mgrafu · 2025-07-11T13:08:09Z

nemo_text_processing/text_normalization/hi/data/telephone/landline_digits.tsv

@@ -0,0 +1,8 @@
+२	दो


is this mapping any different than cardinals (lines 1-4)

These refer to the validation of landline numbers starting with specific digits within India.

but is the mapping any different from cardinals? if not, please import from cardinals and restrict the accepted inputs

I have removed the additional mappings from the TSV file and integrated them into the existing TSV files, as you suggested.

mgrafu · 2025-07-11T13:08:30Z

nemo_text_processing/text_normalization/hi/data/telephone/mobile_digits.tsv

@@ -0,0 +1,8 @@
+६	छह


is this mapping any different than cardinals (lines 1-4)

These refer to the validation of mobile numbers starting with specific digits within India.

but is the mapping any different from cardinals? if not, please import from cardinals and restrict the accepted inputs

I have removed the additional mappings from the TSV file and integrated them into the existing TSV files, as you suggested.

mgrafu · 2025-07-11T13:08:39Z

nemo_text_processing/text_normalization/hi/data/telephone/number.tsv

@@ -0,0 +1,20 @@
+०	शून्य


is this mapping any different than cardinals?

For Hindi digits, no, it's actually the same as cardinal single digits. But for English digits, yes, it's just a common resource for telephone class.

can you please use cardinal for Hindi digits and filter the inputs you need, and only add a file for English digits in that case? let's avoid repetition

yes sure, I've updated the same

mgrafu · 2025-07-11T13:09:05Z

nemo_text_processing/text_normalization/hi/data/whitelist/paune_mappings.tsv

@@ -0,0 +1,100 @@
+०	एक


is this mapping any different than cardinals (lines 1-4)?

Yes, actually 0.75 is converted to a quarter, so zero is mapped to one in paune_mappings.

we don't want a data file that is 100 lines -- please reuse cardinal when applicable or reapply with rules elsewhere

nemo_text_processing/text_normalization/hi/taggers/cardinal.py

nemo_text_processing/text_normalization/hi/taggers/telephone.py

nemo_text_processing/text_normalization/hi/taggers/word.py

Signed-off-by: Namrata Gachchi <[email protected]>

for more information, see https://pre-commit.ci

mgrafu · 2025-07-22T19:58:32Z

nemo_text_processing/text_normalization/hi/taggers/telephone.py

+    def __init__(self):
+        super().__init__(name="telephone", kind="classify")
+
+        mobile_number = generate_mobile(["नंबर", "मोबाइल", "फोन", "कॉल"])


can these inputs be part of a tsv file instead of hardcoding them here?

yes sure, I've removed these inputs and converted them to respective tsv files

nemo_text_processing/text_normalization/hi/taggers/word.py

tests/nemo_text_processing/hi/data_text_normalization/test_cases_fraction.txt

Signed-off-by: Namrata Gachchi <[email protected]>

for more information, see https://pre-commit.ci

nemo_text_processing/text_normalization/hi/data/telephone/number.tsv

Signed-off-by: Namrata Gachchi <[email protected]>

mgrafu and others added 4 commits April 24, 2025 11:20

telephone changes and other future implementations

e165e63

Signed-off-by: Namrata Gachchi <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

3963e4f

for more information, see https://pre-commit.ci

updated Jenkins file

775e08e

Signed-off-by: Namrata Gachchi <[email protected]>

github-advanced-security bot found potential problems Jun 25, 2025

View reviewed changes

ngachchi and others added 2 commits June 26, 2025 10:03

removed unused imports and variable

e0a4a85

Signed-off-by: Namrata Gachchi <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

2740172

for more information, see https://pre-commit.ci

ngachchi marked this pull request as ready for review July 1, 2025 11:15

ngachchi commented Jul 1, 2025

View reviewed changes

ngachchi and others added 2 commits July 7, 2025 16:34

pincode and last four digits of credit card implementations within th…

a09def4

…e telephone class Signed-off-by: Namrata Gachchi <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

052b396

for more information, see https://pre-commit.ci

mgrafu changed the base branch from main to staging_hi_tn July 7, 2025 17:44

ngachchi and others added 2 commits July 8, 2025 09:27

Merge branch 'staging_hi_tn' into telephone

4b188c9

Signed-off-by: Namrata Gachchi <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

81c664a

for more information, see https://pre-commit.ci

mgrafu reviewed Jul 8, 2025

View reviewed changes

nemo_text_processing/text_normalization/hi/data/telephone/STD_codes.tsv Outdated Show resolved Hide resolved

ngachchi and others added 2 commits July 9, 2025 09:57

removed dependency from STD_codes.tsv file

6f56d08

Signed-off-by: Namrata Gachchi <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

764b577

for more information, see https://pre-commit.ci

mgrafu reviewed Jul 11, 2025

View reviewed changes

nemo_text_processing/text_normalization/hi/taggers/cardinal.py Outdated Show resolved Hide resolved

mgrafu reviewed Jul 11, 2025

View reviewed changes

nemo_text_processing/text_normalization/hi/taggers/telephone.py Outdated Show resolved Hide resolved

mgrafu reviewed Jul 11, 2025

View reviewed changes

nemo_text_processing/text_normalization/hi/taggers/word.py Outdated Show resolved Hide resolved

ngachchi and others added 2 commits July 14, 2025 11:05

removed unused lines

aaaaa8f

Signed-off-by: Namrata Gachchi <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

f1d74e6

for more information, see https://pre-commit.ci

mgrafu reviewed Jul 22, 2025

View reviewed changes

nemo_text_processing/text_normalization/hi/taggers/word.py Show resolved Hide resolved

mgrafu reviewed Jul 22, 2025

View reviewed changes

tests/nemo_text_processing/hi/data_text_normalization/test_cases_fraction.txt Show resolved Hide resolved

added support for context words through tsv files

1e214d1

Signed-off-by: Namrata Gachchi <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

fd8cb94

for more information, see https://pre-commit.ci

mgrafu reviewed Jul 23, 2025

View reviewed changes

nemo_text_processing/text_normalization/hi/data/telephone/number.tsv Show resolved Hide resolved

removed repetitive digits validation through multiple tsv

285d0b3

Signed-off-by: Namrata Gachchi <[email protected]>

Hindi TN 2.0 - Accuracy Enhancements & New Telephone Class Integration #294

Are you sure you want to change the base?

Hindi TN 2.0 - Accuracy Enhancements & New Telephone Class Integration #294

Conversation

ngachchi commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Accuracy Improvements by Class:

Key Enhancements:

Before your PR is "Ready for review"

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ngachchi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgrafu Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ngachchi commented Jun 25, 2025 •

edited

Loading

ngachchi left a comment •

edited

Loading

mgrafu Jul 23, 2025 •

edited

Loading