\[ \begin{align}\begin{aligned}\newcommand\blank{~\underline{\hspace{1.2cm}}~}\\% Bold symbols (vectors) \newcommand\bs[1]{\mathbf{#1}}\\% Differential \newcommand\dd[2][]{\mathrm{d}^{#1}{#2}} % use as \dd, \dd{x}, or \dd[2]{x}\\% Poor man's siunitx \newcommand\unit[1]{\mathrm{#1}} \newcommand\num[1]{#1} \newcommand\qty[2]{#1~\unit{#2}}\\\newcommand\per{/} \newcommand\squared{{}^2} \newcommand\cubed{{}^3} % % Scale \newcommand\milli{\unit{m}} \newcommand\centi{\unit{c}} \newcommand\kilo{\unit{k}} \newcommand\mega{\unit{M}} % % Percent \newcommand\percent{\unit{{\kern-4mu}\%}} % % Angle \newcommand\radian{\unit{rad}} \newcommand\degree{\unit{{\kern-4mu}^\circ}} % % Time \newcommand\second{\unit{s}} \newcommand\s{\second} \newcommand\minute{\unit{min}} \newcommand\hour{\unit{h}} % % Distance \newcommand\meter{\unit{m}} \newcommand\m{\meter} \newcommand\inch{\unit{in}} \newcommand\foot{\unit{ft}} % % Force \newcommand\newton{\unit{N}} \newcommand\kip{\unit{kip}} % kilopound in "freedom" units - edit made by Sri % % Mass \newcommand\gram{\unit{g}} \newcommand\g{\gram} \newcommand\kilogram{\unit{kg}} \newcommand\kg{\kilogram} \newcommand\grain{\unit{grain}} \newcommand\ounce{\unit{oz}} \newcommand\pound{\unit{lbs}} % % Temperature \newcommand\kelvin{\unit{K}} \newcommand\K{\kelvin} \newcommand\celsius{\unit{{\kern-4mu}^\circ C}} \newcommand\C{\celsius} \newcommand\fahrenheit{\unit{{\kern-4mu}^\circ F}} \newcommand\F{\fahrenheit} % % Area \newcommand\sqft{\unit{sq\,\foot}} % square foot % % Volume \newcommand\liter{\unit{L}} \newcommand\gallon{\unit{gal}} % % Frequency \newcommand\hertz{\unit{Hz}} \newcommand\rpm{\unit{rpm}} % % Voltage \newcommand\volt{\unit{V}} \newcommand\V{\volt} \newcommand\millivolt{\milli\volt} \newcommand\mV{\milli\volt} \newcommand\kilovolt{\kilo\volt} \newcommand\kV{\kilo\volt} % % Current \newcommand\ampere{\unit{A}} \newcommand\A{\ampere} \newcommand\milliampereA{\milli\ampere} \newcommand\mA{\milli\ampere} \newcommand\kiloampereA{\kilo\ampere} \newcommand\kA{\kilo\ampere} % % Resistance \newcommand\ohm{\Omega} \newcommand\milliohm{\milli\ohm} \newcommand\kiloohm{\kilo\ohm} % correct SI spelling \newcommand\kilohm{\kilo\ohm} % "American" spelling used in siunitx \newcommand\megaohm{\mega\ohm} % correct SI spelling \newcommand\megohm{\mega\ohm} % "American" spelling used in siunitx % % Capacitance \newcommand\farad{\unit{F}} \newcommand\F{\farad} \newcommand\microfarad{\micro\farad} \newcommand\muF{\micro\farad} % % Inductance \newcommand\henry{\unit{H}} \newcommand\H{\henry} \newcommand\millihenry{\milli\henry} \newcommand\mH{\milli\henry} % % Power \newcommand\watt{\unit{W}} \newcommand\W{\watt} \newcommand\milliwatt{\milli\watt} \newcommand\mW{\milli\watt} \newcommand\kilowatt{\kilo\watt} \newcommand\kW{\kilo\watt} % % Energy \newcommand\joule{\unit{J}} \newcommand\J{\joule} % % Composite units % % Torque \newcommand\ozin{\unit{\ounce}\,\unit{in}} \newcommand\newtonmeter{\unit{\newton\,\meter}} % % Pressure \newcommand\psf{\unit{psf}} % pounds per square foot \newcommand\pcf{\unit{pcf}} % pounds per cubic foot \newcommand\pascal{\unit{Pa}} \newcommand\Pa{\pascal} \newcommand\ksi{\unit{ksi}} % kilopound per square inch \newcommand\bar{\unit{bar}} % % Bits \newcommand\bit{\unit{b}} \newcommand\byte{\unit{B}}\end{aligned}\end{align} \]

Sep 03, 2026 | 1014 words | 10 min read

16.3.2. Task 2

Learning Objectives

• Reuse previously written functions to analyze new data.

• Load and interpret saved model data from CSV files.

• Compare frequency models using a quantitative distance metric.

• Visualize analysis results using plots.

Introduction

In Section 16.3.1 you built normalized n-gram frequency models for known languages and saved them to CSV files using the n,ngram,frequency format. These models describe how frequently different character patterns appear in each language.

In this task, you will apply those models to analyze a text file in an unknown language. The user will select one unknown language text and one n-gram size to use for the analysis. You will compare that unknown text model to each known language model for the selected n-gram size and identify the closest language match.

Task Instructions

Make sure you have successfully completed Section 16.3.1 and have the CSV files containing the n-gram models for each known language. You will need to place these files in a folder named models within the same folder as your Python script. The CSV files should use the n,ngram,frequency format from Section 16.3.1.

Before creating the program, create a flowchart of the algorithm you will use and save it as py4_ind_2_username.pdf. Then start your program from a copy of the ENGR133_Python_Template.py Python template. Your program should be named py4_ind_2_username.py. You will also need to create a folder named unknown_texts within the same folder as your Python script. Then, download each of the unknown texts in Table 16.10 and place them into your unknown_texts folder. Develop a Python program that does the following:

1. Display the unknown text files and ask the user to select one file to analyze.

2. Ask the user to select an n-gram size from \(1\) to \(5\).

3. Load the selected n-gram models from the CSV files created in Section 16.3.1.

4. Create a relative frequency n-gram for the selected unknown text file.

5. For the selected unknown text, compute the distance between its n-gram model and each known language n-gram model.

6. Find the known language with the smallest distance score and report it as the best language model match.

7. Generate a plot for the selected unknown text showing the total difference score for each known language.

Table 16.10 Unknown Text Download

Unknown Text	Download Link
Unknown Text 1	sample_unknown_1.txt
Unknown Text 2	sample_unknown_2.txt
Unknown Text 3	sample_unknown_3.txt

Note

Make sure you saved the samples into your unknown_texts folder, and that your folder is located in the same directory as your Python script.

Reusing N-Gram Functions

In Section 16.3.1, you created functions to clean text, count n-grams, and normalize n-gram counts. Reuse those functions in this task instead of writing new versions of the same logic.

After the user selects an unknown text and an n-gram size, use your previous clean_text function to clean the unknown text before creating the n-gram. Then create the unknown text’s n-gram counts for the selected value of n using your previous create_n_gram function. Then use your previous normalize_n_gram function to convert those counts to relative frequencies.

Note

Remember to submit py4_ind_1_username.py in addition to this task’s deliverables if you import functions from it.

Load From CSV Function

Create a function named load_from_csv that takes in the selected n-gram size n and returns a dictionary where the keys are the known languages (e.g., “english”, “french”) and the values are only the relative frequency n-gram dictionary for the selected n-gram size.

For example, if the user selects n = 2, the returned dictionary should have the following structure:

models["english"] = {"th": 0.0254, "he": 0.0221, ...}
models["french"] = {"le": 0.0183, "es": 0.0175, ...}

When reading the CSV files, ensure that you correctly parse the n-grams and their relative frequencies for the selected n-gram size from the CSV files. You can use the csv module from the Python standard library to help with this task.

N-Gram Distance Function

Create a function named n_gram_dist that takes in two n-gram dictionaries (one for the unknown text and one for a known language) and returns the total distance between the two n-gram models using a distance metric.

The distance metric you will implement is the absolute difference between the relative frequencies of the n-grams in the two models. To calculate this, you will need to iterate through all n-grams that appear in either model and use the following formula:

(16.2)\[\begin{split}\text{Distance} = \left\{ \begin{array}{cl} |L-U| & \text{if n-gram is in both models,}\\ L & \text{if n-gram is only in language model,}\\ U & \text{if n-gram is only in unknown model.} \end{array} \right.\end{split}\]

Where \(L\) is the relative frequency of the n-gram in the known language model and \(U\) is the relative frequency of the n-gram in the unknown language model.

By summing the distances for all n-grams, you will get a total distance score that quantifies how different the two models are. A smaller distance indicates that the unknown text is more similar to the known language, while a larger distance indicates that it is less similar.

Score Language Function

Create a function named score_language that takes a dictionary of known language n-gram models (as returned by the load_from_csv function), and the selected unknown text normalized n-gram dictionary. This function will return a dictionary whose keys are the known language names and values are the total distance scores for each known language model compared to the unknown text model.

Use the n_gram_dist function to calculate the distance between the selected unknown text model and each known language model.

Plotting Function

Use the provided plot_language_scores function. It takes in a dictionary of total difference scores for each known language, the name of the unknown text, and the selected n-gram size. This function generates a bar plot showing the total difference score for each known language. The language with the lowest score is the best match.

import matplotlib.pyplot as plt

def plot_language_scores(scores, name, n):
    languages = list(scores.keys())
    distances = list(scores.values())
    fig, ax = plt.subplots()
    ax.bar(languages, distances)
    ax.set_title(f"{name} {n}-gram Language Scores")
    ax.set_xlabel("Language")
    ax.set_ylabel("Total Difference")
    ax.tick_params(axis='x', rotation=45)
    fig.tight_layout()
    plt.show()

Main Function

In your main function, you will need to do the following:

1. Display the unknown texts in the unknown_texts folder and have the user select a text to analyze.

2. Ask the user to select an n-gram size from \(1\) to \(5\).

3. Load the selected n-gram size from the known language models using the load_from_csv function.

4. Reuse functions from Section 16.3.1 to clean the selected unknown text and create a normalized n-gram model.

5. Use score_language to get the language distance scores for the selected n-gram size.

6. Print the best n-gram language model match for the unknown text. The best match is the language with the smallest total difference score.

7. Plot the language total difference scores for the selected n-gram size.

Sample Output

Test cases for language identification and visualization. Use the values in Table 16.11 below to test your program.

Table 16.11 Test Cases

Case	Unknown File Option	n-gram size
1	1	1
2	2	3
3	3	5

Ensure your program’s output matches the provided samples exactly. This includes all characters, white space, and punctuation. In the samples, user input is highlighted like this for clarity, but your program should not highlight user input in this way.

Case 1 Sample Output

$ python3 py4_ind_2_username.py Unknown Language File Options 1. unknown_1 2. unknown_2 3. unknown_3 Select a file to analyze: 1 Select an n-gram size (1-5): 1 The best language match for unknown_1 using 1-grams is the english model.

Fig. 16.5 Case_1_sample_output.png

Case 2 Sample Output

$ python3 py4_ind_2_username.py Unknown Language File Options 1. unknown_1 2. unknown_2 3. unknown_3 Select a file to analyze: 2 Select an n-gram size (1-5): 3 The best language match for unknown_2 using 3-grams is the french model.

Fig. 16.6 Case_2_sample_output.png

Case 3 Sample Output

$ python3 py4_ind_2_username.py Unknown Language File Options 1. unknown_1 2. unknown_2 3. unknown_3 Select a file to analyze: 3 Select an n-gram size (1-5): 5 The best language match for unknown_3 using 5-grams is the italian model.

Fig. 16.7 Case_3_sample_output.png

Table 16.12 Deliverables

Deliverables	Description
py4_ind_2_username.pdf	Flowchart(s) for this task.
py4_ind_2_username.py	Your completed Python code.