Files and Folders

Paths & File Handling

You should already know
Running local programs, handling expected errors and checking functions with assert.
You'll build
Generated text saved to disk, then a word-count report from a text file of your own.
You'll be able to
Read and write text files, locate them using paths and the working folder, and verify that a saved report answers the intended question.
You need
Your local Python setup, any plain-text editor, generator and tested word counter from chapters 10–12.

Your generator makes a passage, prints it, and finishes. You can run the instructions again, but that produces another set of random choices. How do you keep a passage you like?

We'll save it in a file. Then we'll turn the word counter into a program that reads your own text and writes a report you can reopen.

Save a Passage

Open my_generator.py. After the generation loop and the final print(output), add:

with open('generated.txt', 'w', encoding='utf-8') as destination:
    destination.write(output)
    destination.write('\n')

Save and run the program using the command from Installing Python:

Windows macOS / Linux
python my_generator.py python3 my_generator.py

Keep using py on Windows if that's the command you've established.

Find generated.txt in the course folder and open it in your editor. It should contain the same passage you just saw in the terminal, followed by a newline. Close it and reopen it. The passage is still there.

The parts of that new block have separate jobs:

write() doesn't add a newline automatically. That's why the second call writes '\n', the string representing a newline.

Write mode replaces an existing file's contents. Running the program again replaces the previous generated.txt. Keep that output filename distinct from your source code and input text. To keep two passages, rename the first output file before generating another.

Close the Input File Too

Earlier, we used a short command to read Shakespeare. Now replace that assignment in my_generator.py with a file block:

with open('shakespeare.txt', encoding='utf-8') as source:
    text = source.read()

When no mode is supplied, open() uses read mode. source.read() reads the text and returns a string. At the end of the block, Python closes the file, while text still refers to the string we read.

Leave followers = next_words(text) after the block, without indenting it. The rest of your generator doesn't change.

Run again. The follower checks should pass, a passage should print, and the same passage should be saved.

Try it: Add a short title and newline to the saved output before writing the passage. Reopen the file to check the line breaks. Does adding that title change what next_words() receives?

Which Folder Did Python Use?

A filename such as 'generated.txt' is a relative path. Python looks for it relative to the program's current working folder.

For our walkthrough, that folder is python-for-humans because we opened a terminal there. The terminal's dir or ls command shows its contents.

Your editor can open a source file from anywhere. Doing that doesn't change the terminal's working folder. Even starting Python with a full path to a source file doesn't automatically make that source file's folder the working folder.

Python can tell us where it is looking. Create a scratch file called where_am_i.py in your course folder:

import pathlib

folder = pathlib.Path.cwd()
print('Working folder:', folder)

pathlib is part of Python's standard library, like random. Path.cwd() produces a path representing the current working directory. Run the scratch file with python where_am_i.py or python3 where_am_i.py.

Here's how the pieces relate:

python-for-humans/       <- working folder for these runs
    my_generator.py     <- source: instructions Python reads
    shakespeare.txt     <- input: text the generator reads
    generated.txt       <- output: passage the generator writes
    where_am_i.py       <- scratch program that prints the working folder

Relative and Absolute Paths

A relative path describes a location from the working folder. For example, 'data/my_text.txt' names my_text.txt inside a data subfolder. Python accepts / in this relative path on Windows as well as macOS and Linux.

An absolute path describes the location from the filesystem's root. Its spelling depends on the system:

Windows: C:\Users\sam\Documents\python-for-humans\my_text.txt
macOS:   /Users/sam/Documents/python-for-humans/my_text.txt
Linux:   /home/sam/Documents/python-for-humans/my_text.txt

These are illustrations, not paths to type unchanged. For this chapter, keep input, source and output together and use relative filenames. That keeps the programs usable when you move the whole folder.

To investigate a wrong-folder error, use a new terminal. Change into the folder containing python-for-humans, then run the generator through its relative path:

Windows macOS / Linux
python python-for-humans/my_generator.py python3 python-for-humans/my_generator.py

Python can find the source file through that path, but open('shakespeare.txt') looks in the terminal's current folder. Assuming there's no Shakespeare file there, it raises FileNotFoundError.

The error arrives as several lines, called a traceback:

Traceback (most recent call last):
  File ".../python-for-humans/my_generator.py", line 3, in <module>
    with open('shakespeare.txt', encoding='utf-8') as source:
FileNotFoundError: [Errno 2] No such file or directory: 'shakespeare.txt'

Your path, line number and line of code will differ. Read a traceback from the bottom up. The last line names the kind of error and gives its message: here, the file Python couldn't find. The lines above say where Python was when it happened: which file, which line number, and the line itself. Longer tracebacks have more of these pairs, one for each function call that was under way, and the bottom pair is still the place to start.

The failure happens while reading the input, before the generator writes its output. Return to the python-for-humans folder and run the ordinary command again. Detailed path construction can wait until you need a more elaborate folder layout.

Start With Text You Can Check

Create my_text.txt in your plain-text editor and save it in the course folder using UTF-8. Start with exactly this sentence:

Dragon, pizza! DRAGON.

Before running anything, predict the word-count dictionary. Our tested rule should produce dragon twice and pizza once.

Open count_words_function.py. Keep your count_words() definition, its test definitions, and the calls that run those tests. Replace the normal example calls or Shakespeare analysis after the passed message with:

input_filename = 'my_text.txt'
output_filename = 'word_counts.txt'

if input_filename == output_filename:
    raise ValueError('Choose different input and output filenames.')

with open(input_filename, encoding='utf-8') as source:
    text = source.read()

counts = count_words(text)

with open(output_filename, 'w', encoding='utf-8') as report:
    for word in counts:
        report.write(f'{word}: {counts[word]}\n')

print('Report saved to', output_filename)

Use different filenames for source, input and output. The explicit check catches giving these two filename values exactly the same string. Keep the simple distinct names above for this walkthrough.

Run the program:

Windows macOS / Linux
python count_words_function.py python3 count_words_function.py

You should see the word-count checks pass, followed by the saved-report message. Open word_counts.txt in your editor:

dragon: 2
pizza: 1

Compare the report with the original sentence. Is it counting the words according to our chosen rule?

Follow the Data

The new code connects three jobs:

  1. Read the input file into a string.
  2. Pass that string to the tested counting function.
  3. Write each word and its count to the output file.

Only the first and third jobs deal with files. count_words() still accepts text and returns a dictionary. Its small-string tests still work without creating a file.

The report loop uses the dictionary keys in their insertion order: the order the words first appeared. This is not a ranking. Repeated words increase their counts without moving their position.

The f-string writes the word, a colon, its count and a newline. print() would display text in the terminal; report.write() sends text to the open report file.

If the input is empty, the dictionary is empty, so the loop writes no lines. Write mode still creates or replaces the report, leaving an empty file. That's an appropriate report for zero words.

Try it: Save an empty my_text.txt, run the analyzer, and reopen the report. Check that the program finishes and that no lines remain from the previous run. Then restore your sentence.

Use Something of Your Own

Replace the tiny sentence in my_text.txt with a modest amount of your own plain text: notes, a story, or dialogue you've written. Save, run the analyzer, and reopen the report.

You can also create a separate plain-text input file and change input_filename to its name. Keep the output filename separate. Neither change requires modifying count_words().

This program expects UTF-8 plain text. A Word document or PDF contains more structure than a text file, so changing its extension to .txt won't convert it. For this exercise, copy the words you want into your plain-text editor and save a new UTF-8 file.

Choose three words and inspect their source occurrences and reported counts. Remember that our cleanup rule strips only certain punctuation from the ends and ignores capitalization. The report answers a specific question about that rule, rather than every possible question about the writing.

We'll add ranked output and a spreadsheet CSV option in Modules and several files. Downloading public datasets and API responses comes in The Web. First, make this small file-to-report workflow something you can verify.

Optional: Pause and Look Inside

A traceback tells you where a program stopped. Some bugs don't stop anything: the program finishes and the report holds a count you didn't expect. One print() just before the line you doubt often settles it. When you have more than one question, a debugger is quicker. It pauses a running program so you can ask about its values for as long as you like. Python includes one, so this works whichever editor you use.

Put the dragon sentence back in my_text.txt. In the analyzer, add a call to breakpoint() as the first line inside the report loop:

with open(output_filename, 'w', encoding='utf-8') as report:
    for word in counts:
        breakpoint()
        report.write(f'{word}: {counts[word]}\n')

Save and run the analyzer with your usual command. This time Python stops partway through and shows something like:

> .../python-for-humans/count_words_function.py(31)<module>()
-> breakpoint()
(Pdb)

The arrow marks the line where Python has paused. (Pdb) is a third kind of prompt, after the terminal's and Python's >>>. It accepts debugger commands, and four are enough:

Type What it does
p word Print the current value of a name, here word.
n Move to the next line.
c Continue running until the next pause or the end of the program.
q Quit this run. Answer y if Python asks whether you're sure.

Type p word, then p counts. You should see 'dragon' and the whole dictionary. Nothing has been written to the report yet.

Type n. The arrow moves to the report.write(...) line. The arrow shows the line that is about to run, so that line hasn't happened yet. Type n again to run it.

Type c. The loop comes round, reaches breakpoint() again and pauses. Before typing p word, predict what it will show.

Type c once more to let the program finish. Then remove the breakpoint() line, save, and run the analyzer normally. If you left with q instead, the report may be incomplete until you do.

PyCharm, Visual Studio Code and Zed each have a debugger with buttons for the same steps. Whichever you use, the questions stay the same: what value does this name have right now, and is it the one I expected?

Your Turn

Predict Before Running

Change the source to Pizza! dragon pizza.. Predict the report's lines and their order. Then run the analyzer and compare.

Why does pizza appear first even if another text happened to give dragon a larger count?

Find a Misplaced Input

Make a data subfolder in your course folder and move my_text.txt into it. Run the analyzer without changing its filename value. Read the error and explain where Python looked.

Repair the path without changing the counting function. Then restore your preferred layout and verify the report again.

Compare the path after trying

With the terminal still in python-for-humans, the relative input path is 'data/my_text.txt'. The output remains in the working folder unless you also change its path.

Make and Verify Your Own Report

Without copying another worked input:

  1. Create a different short UTF-8 text file and work out its counts by hand.
  2. Point the analyzer at it and run the existing tests and analysis.
  3. Reopen the output and compare it with your expectation.
  4. Switch to a longer personal text without changing the counting function.
  5. Close the tools, then find the saved report again.

Explain where the source, input and output files are. What would happen to the report if you ran the analyzer with a different input but kept the same output filename?

On Your Own: Summarize Sample Expenses

Use a new setting to combine file reading, loops, dictionaries, functions and checks. Create expenses.txt with these invented amounts in cents, one category and amount per line:

food 1250
travel 300
food 625

Write category_totals(lines). It should accept a list of these lines and return a dictionary adding the amounts for each category. Call it on a tiny list in an assertion before connecting it to files.

Then write a program that reads expenses.txt, passes its lines to the helper and writes expense_totals.txt. Work out the expected report yourself before opening the hints. Run the program, close the tools and reopen the saved report.

Check repeated categories, an amount of zero and an empty input file. An empty input should produce an empty report, replacing any previous report's lines. Keep input and output filenames distinct. For this exercise, every nonempty line has exactly two space-separated fields, a one-word category and a nonnegative whole-number amount. These are sample purchases, not a general bank-file format.

Hint: Read a line into two familiar values

source.read().splitlines() gives you the lines. Inside the helper, fields = line.split() separates one line; use fields[0] for the category and int(fields[1]) for the amount. An empty file supplies no lines. Keep each category's running total in a dictionary.

Compare a complete approach after trying

Save this as expense_report.py:

def category_totals(lines):
    totals = {}
    for line in lines:
        fields = line.split()
        category = fields[0]
        amount = int(fields[1])
        if category in totals:
            totals[category] = totals[category] + amount
        else:
            totals[category] = amount
    return totals


def test_category_totals():
    assert category_totals(['food 1250', 'travel 300', 'food 625']) == {
        'food': 1875, 'travel': 300
    }
    assert category_totals(['food 0']) == {'food': 0}
    assert category_totals([]) == {}


test_category_totals()
with open('expenses.txt', encoding='utf-8') as source:
    lines = source.read().splitlines()

totals = category_totals(lines)
with open('expense_totals.txt', 'w', encoding='utf-8') as report:
    for category in totals:
        report.write(f'{category}: {totals[category]} cents\n')
print('Expense report saved.')

The first sample produces food: 1875 cents followed by travel: 300 cents. Integer cents let us add money without introducing rounding rules. A blank line inside a nonempty file is outside this exercise's stated format; it should expose a problem rather than silently inventing an amount.

Change the input to categories and amounts of your own. Explain which code calculates a total and which code writes it. We'll reuse this program for an independent module-extraction checkpoint next.

In Your Own Words

Explain why Python could find a source file but fail to find its input. Then describe how you checked that a saved report answers the question you intended.

You can now run a saved program, investigate a failure, check its helpers, and use your own files while keeping the results. Next, we'll organize reusable functions and their tests into separate files.