Skip to content

Don't append a full stop after Hindi and Arabic sentence punctuation - #572

Open
ANDROIDHASSAN wants to merge 2 commits into
resemble-ai:masterfrom
ANDROIDHASSAN:fix/punc-norm-danda
Open

ANDROIDHASSAN wants to merge 2 commits into
resemble-ai:masterfrom
ANDROIDHASSAN:fix/punc-norm-danda

Conversation

@ANDROIDHASSAN

Copy link
Copy Markdown

punc_norm in mtl_tts.py appends . unless the text already ends in one of its sentence enders. Hindi and Arabic are supported languages, but their sentence punctuation was not in that set, so their input got a second stop:

before                         after
'मेरा नाम हसन है।'  -> 'मेरा नाम हसन है।.'    'मेरा नाम हसन है।'
'श्लोक ॥'          -> 'श्लोक ॥.'             'श्लोक ॥'
'كيف حالك؟'        -> 'كيف حالك؟.'           'كيف حالك؟'
'مرحبا،'           -> 'مرحبا،.'              'مرحبا،'
'Hello there'      -> 'Hello there.'         'Hello there.'   (unchanged)
'नमस्ते'           -> 'नमस्ते.'              'नमस्ते.'        (unchanged)

This adds the danda ।, double danda ॥, Arabic question mark ؟ and Arabic comma ، to sentence_enders, next to the CJK marks already there (the ASCII comma is already an ender, so the Arabic comma now behaves the same). All four characters are in the multilingual tokenizer's vocabulary, so the text now ends on the real punctuation token instead of an extra ..

Checked by running punc_norm from this branch on the inputs above.

punc_norm in the multilingual model adds '.' unless the text ends in one
of its sentence enders. The Devanagari danda (।) and the Arabic question
mark (؟) were missing, so Hindi and Arabic input got a second full stop:
'मेरा नाम हसन है।' became 'मेरा नाम हसन है।.'.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant