5 The most common words, and a report

5.1 A function for the most common words

count gives the counts in no useful order. To find the most common words, tally must sort the counts, most common first, and keep the first few. A function, top, does it. It sorts by count, highest first, and then by the word, so that two words with the same count keep one order. Then it keeps the first n.

A patch to src/tally/__init__.py.

A test says what top must give. It gives top three words with their counts, and asks for the two most common.

A patch to tests/test_top.py.

In src/tally/__init__.py, top stands between count and main. It sorts with the key (-pair[1], pair[0]). The key is the count with a minus sign, so that the highest count comes first, and then the word. The new file tests/test_top.py has one test, test_most_common_first.

uv run python -m unittest discover -s tests -q

The four tests pass.

@@ -11,6 +11,11 @@ def count(text: str) -> dict[str, int]:
     return counts
 
 
+def top(counts: dict[str, int], n: int) -> list[tuple[str, int]]:
+    "The n most common words, most common first."
+    return sorted(counts.items(), key=lambda pair: (-pair[1], pair[0]))[:n]
+
+
 def main() -> None:
     text = open(sys.argv[1]).read() if len(sys.argv) > 1 else sys.stdin.read()
     for word, n in sorted(count(text).items()):

@@ -0,0 +1,8 @@
+import unittest
+
+from tally import top
+
+
+class Top(unittest.TestCase):
+    def test_most_common_first(self):
+        self.assertEqual(top({"a": 1, "b": 3, "c": 2}, 2), [("b", 3), ("c", 2)])

5.2 tally --top N

tally prints every word of a text, and the corpus fills the screen. An option, --top N, makes main print only the N most common words, with top. Without the option, main prints every count, as before. main reads the option before the name of the file, so the command is tally --top 3 data/corpus.txt.

A patch to src/tally/__init__.py.

In src/tally/__init__.py, main now has a docstring, and the three lines under the docstring read the option. When the first argument is --top, the second is the number N, and the name of the file comes after them. The loop prints top(counts, n) when there is an N, and every count when there is not.

uv run tally --top 3 data/corpus.txt

The command prints three lines, the most common word first. They are 127 line, 126 class and 125 word.

@@ -17,6 +17,11 @@ def top(counts: dict[str, int], n: int) -> list[tuple[str, int]]:
 
 
 def main() -> None:
-    text = open(sys.argv[1]).read() if len(sys.argv) > 1 else sys.stdin.read()
-    for word, n in sorted(count(text).items()):
-        print(n, word)
+    "Count the words of a file, or of the standard input. With --top N, only the N most common."
+    args, n = sys.argv[1:], None
+    if args[:1] == ["--top"]:
+        n, args = int(args[1]), args[2:]
+    text = open(args[0]).read() if args else sys.stdin.read()
+    counts = count(text)
+    for word, times in top(counts, n) if n else sorted(counts.items()):
+        print(times, word)

5.3 A report, which just setup makes

The most common words of the corpus make a report, which others want to read without running tally. A recipe of the justfile, report, writes the ten most common words to build/top.txt. The recipe has two lines, and just prints each line before it runs it. The second line has two parts.

• mkdir -p build makes the folder build, and does nothing when the folder is there.
• uv run tally --top 10 data/corpus.txt counts the corpus, and keeps the ten most common words.
• > build/top.txt sends what tally prints to the file, in place of the terminal.

A patch to justfile.

The report is an artifact, as the corpus is. git ignores the folder build/, so git holds the recipe, and not the report. just setup makes the report when the file is missing. test -f build/top.txt is true when the file is there, and || runs just report only when it is not.

At the end of the justfile, the new recipe report makes the folder build, and writes the report into it. In the recipe setup, the new last line runs just report when build/top.txt is missing.

A report from an earlier run stays, because git ignores it. So remove the report first, and then just setup makes it again:

rm -f build/top.txt
just setup
cat build/top.txt

The file has ten lines, the most common word first.

@@ -6,6 +6,7 @@ default:
 setup:
     uv sync
     uv run python scripts/corpus.py data/corpus.txt
+    test -f build/top.txt || just report
 
 # Count the words of a file
 run file:
@@ -14,3 +15,8 @@ run file:
 # Run the tests
 test:
     uv run python -m unittest discover -s tests -q
+
+# Write the ten most common words of the corpus to build/top.txt
+report:
+    mkdir -p build
+    uv run tally --top 10 data/corpus.txt > build/top.txt