18RanText & sentiment

Language detection

I turned 22,000 sentences in 22 languages into word counts and trained multinomial naive Bayes, which scored 95.3% on a 20% test split. A Hindi sentence I typed came back as Hindi.

Original tutorial by Aman Kharwal: Language Detection (opens in a new tab)

What it printed in 2022

Numbers read from the saved notebook outputs. Cell numbers count every cell in the notebook, Markdown included, starting at 1.
Output	Value	Cell #
Test accuracy (multinomial naive Bayes)	95.3%	12
Languages	22	6

Cell # counts every cell from the top of the notebook, Markdown included, starting at 1.

The data

A balanced table of 1,000 sentences per language from the tutorial author's data repository.

What I'd do differently

Use character n-grams. Chinese, Japanese and Thai do not put spaces between words, so a word tokeniser sees each sentence as one rare token.
Print the confusion matrix to see which languages are mixed up.