Layers of Logic

8.1

Java I/O: Streams, Scanner, BufferedReader

You have been using `Scanner` since Phase I without being told what it is. Here is the machine underneath it, and why reading one byte at a time is a thousand times slower than it needs to be.

Core19 min read4 exercises
01

Previously on

Back in Section 1.2 you were handed a Field Kit: three lines that let you read what the user types, with a promise that Section 8.1 would explain them.

import java.util.Scanner;
Scanner scanner = new Scanner(System.in);
String name = scanner.nextLine();

You have used it in exercises across six phases. This is where the promise is kept.

You will also need arrays from Section 4.1, because a buffer is an array. And inheritance from Section 7.1, because the I/O library is one of the deepest hierarchies in Java.

02

The problem

Your program lives inside the JVM. The keyboard, the screen, files and the network are all outside it.

Every one of those is different. A keyboard delivers characters slowly as somebody types. A file is a large block sitting on a disk. A network connection arrives in pieces, out of order, with pauses. Writing separate code for each would mean learning four unrelated systems.

There is a second problem, and it is about speed.

int b;
while ((b = System.in.read()) != -1) {
    process(b);
}

That reads one byte at a time. It is correct, and on a large file it is thousands of times slower than it needs to be. Not because moving bytes is slow, but because of what happens around each read.

03

The idea

Java’s answer to the first problem is one idea:

A stream is a flow of bytes, one after another, from somewhere to somewhere.

The source does not matter. A keyboard, a file, a network socket and a block of memory all present themselves as a stream, so the code that reads them is the same code.

DirectionBase class
Bytes inreading raw bytesInputStream
Bytes outwriting raw bytesOutputStream
Characters inreading textReader
Characters outwriting textWriter
Four abstract classes. Everything else in java.io extends one of them.

System.in is an InputStream

System.out.println("hello");    // System.out is a PrintStream
int b = System.in.read();       // System.in is an InputStream

System.in has been an InputStream all along. It gives you raw bytes, and nothing else. It does not know about letters, lines, or numbers.

Which is why the Field Kit needed a Scanner wrapped around it.

04

Under the hood

Going deeper

The stack of three, explained

Here is the line you will see in most serious Java input code:

BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));

Three objects. Each one solves a different problem, and each wraps the one below it.

BufferedReaderadds speed and lines
  • readLine()
  • an 8192 char buffer

Reads a big block at once, then serves your requests from memory.

InputStreamReaderadds meaning
  • bytes -> characters
  • character encoding

Turns raw bytes into letters. Needed the moment anyone types a non-English character.

System.inthe raw source
  • read() one byte

An InputStream. Bytes only. Knows nothing about text.

Each layer adds exactly one capability. This is called the decorator pattern, and it is why java.io has so many classes that each do so little.

Why buffering changes everything

Reading without a buffer:

What one unbuffered read costs

  1. Your program asks for one byteA method call. Cheap so far.
  2. The JVM makes a system callYour program stops. The operating system takes over. This switch is the expensive part.
  3. The OS fetches the dataFrom the disk, the keyboard, or the network.
  4. Control comes backAnother switch. Then you have your one byte.

Do that a million times for a one megabyte file and you have made a million system calls.

With a buffer, the first read fetches 8192 bytes into an array in memory. The next 8191 reads are served from that array with no operating system involvement at all.

Same total bytes. Roughly one eight-thousandth of the system calls.

Scanner versus BufferedReader

ScannerBufferedReader
Gives youints, doubles, words, lineslines of text only
Parsingbuilt inyou do it yourself
Buffer sizeabout 1 KBabout 8 KB
Speedslower, parses on every callmuch faster
Thread safenoyes, its methods are synchronized
Reach for it whensmall interactive programsfiles, or a lot of input

Scanner is doing more work per call. It reads, then applies a regular expression to find token boundaries, then converts. That convenience is worth it while you are learning, and expensive when the input is large.

The Scanner trap everybody hits

Scanner scanner = new Scanner(System.in);

System.out.print("Age: ");
int age = scanner.nextInt();

System.out.print("Name: ");
String name = scanner.nextLine();      // returns "" immediately, without waiting

The program appears to skip the name question.

The reason: nextInt() reads the digits and leaves the newline sitting in the stream. nextLine() then reads from wherever the cursor is to the next newline, which is zero characters away.

Three fixes, worst to best

  1. Add an extra scanner.nextLine() to swallow the newlineWorks, and every reader has to work out why that stray line is there.
  2. Read everything with nextLine() and parse yourselfint age = Integer.parseInt(scanner.nextLine());. Consistent, and no leftover newlines anywhere.
  3. Use BufferedReader for the whole programIt only has readLine(), so the problem cannot occur. This is what most real code does.

Closing streams

An open stream holds an operating system resource. Leave enough of them open and your program runs out of file handles, which fails in a way that looks nothing like the code that caused it.

// Old way, and easy to get wrong
BufferedReader reader = null;
try {
    reader = new BufferedReader(new FileReader("units.txt"));
    // ...
} finally {
    if (reader != null) reader.close();
}

// try-with-resources: closes automatically, whatever happens
try (BufferedReader reader = new BufferedReader(new FileReader("units.txt"))) {
    String line;
    while ((line = reader.readLine()) != null) {
        System.out.println(line);
    }
}

The second form closes the reader when the block ends, including when an exception is thrown. Anything you open in those brackets is closed for you.

Phase XII covers exceptions properly. For now: always use try-with-resources for files.

05

What it costs

The class count is intimidating. java.io has dozens of classes with long names. That is the price of the decorator design: each class does one small thing, so a normal job needs several of them.

The stacking is verbose with it. new BufferedReader(new InputStreamReader(System.in)) is a lot of typing to read one line, and nothing in it tells a newcomer what each layer is for.

Byte streams and character streams are also separate hierarchies. InputStream and Reader share no parent, so you cannot write one method that accepts either.

And everything throws checked exceptions. Every read can fail, so every read has to be wrapped or declared. Honest, and noisy in a small program.

What you get is one way to read everything. The code that reads a file also reads a network socket, a block of memory or a compressed archive, unchanged. And one wrapper turns a program that makes a million system calls into one that makes a hundred and twenty.

06

Check yourself

Answer from memory before opening the explanation. Then compare your rule with the answer and revisit the example if they differ.

  1. Why does `new BufferedReader(new InputStreamReader(System.in))` need three objects? What does each one do?

    Show the answer

    Each one solves a different problem, and stacking them is the design, not clumsiness.

    System.in is an InputStream. It gives you raw bytes, one at a time, and knows nothing about letters.

    InputStreamReader turns bytes into characters, using a character encoding. This matters the moment anybody types a letter that is not plain English, because such letters take more than one byte.

    BufferedReader reads a large block at once into an array in memory and serves your requests from there, so you are not asking the operating system for one character at a time. It also adds readLine().

    This stacking is called the decorator pattern. Each class adds one capability and wraps the one below it, which is why Java has so many I/O classes and so few of them do more than one thing.

  2. Reading a large file one byte at a time is enormously slower than reading it in blocks, even though it is the same number of bytes. Why?

    Show the answer

    Because the cost is not in the bytes. It is in the asking.

    Every unbuffered read is a system call: your program stops, the operating system takes over, fetches the data, and hands control back. That switch costs far more than moving one byte, and you are paying it once per byte.

    A buffer changes the ratio. One system call fetches 8192 bytes into an array, and the next 8191 reads are served from memory with no operating system involvement at all.

    Same total bytes, roughly one eight-thousandth of the system calls. This is the same idea as the processor cache from the Section 4.2 exercises: fetch a block, because the next thing you want is probably next to the last thing you wanted.

  3. Should you use `Scanner` or `BufferedReader`?

    Show the answer

    It depends on what you value, and the honest answer has a middle case.

    Scanner parses for you. nextInt(), nextDouble(), next(). Convenient, and slower, with a small buffer and parsing on every call. Good for small interactive programs and for learning.

    BufferedReader gives you lines of text and nothing else. You parse them yourself. Much faster, and better when input is large or you are reading files.

    The trap is mixing nextInt() and nextLine() on one Scanner. nextInt() leaves the newline in the stream, so the next nextLine() returns an empty String. Almost every beginner hits this, and it looks like the program skipped a question.

07

Exercises

Write each solution in VS Code, predict its result, and then run the checks. Reading an example does not replace implementing it.

4 exercises95 pointsabout 105 minutes

The Layers of Logic VS Code extension runs the checks for exercises marked checked. For a manual exercise, run the program and compare its behaviour with the stated requirements and sample output.
A

Unwrap the Field Kit

Warm up·20 min·15 points

checkedex-8-1-a

Take apart the three lines you were handed in Section 1.2 and have used ever since.

Start with the raw stream and read a single byte. Print it as a number and as a character. That is what System.in actually offers, and it is less than you expected.

Then add the layers one at a time, and after each one write down what you can now do that you could not before.

Finish with the interesting test: type a character that is not plain English, such as a letter from your own language or an emoji, and read one raw byte. You will not get a letter. Explain why, and you will have understood what InputStreamReader is for.

What to write

These exact signatures. The checks call them by name, so a different name or a different parameter type will not build.

class Unwrap {
    static InputStream raw(String text)
    static int firstRawByte(String text)
    static int byteCount(String text)
    static int charCount(String text)
    static String firstLine(String text)
    static List<String> allLines(String text)
    static String scannerFirstLine(String text)
    static boolean oneByteIsTheWholeCharacter(String text)
}

The source is a String rather than the keyboard, so the checks can feed it text you would find hard to type. raw() gives you the bottom of the stack. Build the other layers on top of it. byteCount reads one byte at a time, charCount reads one character at a time, and the two answers are not always the same number.

What your program must do

  • Read one raw byte and return the number rather than a character
  • Add the reader layer and count characters instead of bytes
  • Add the buffered layer and read whole lines with it
  • Feed in a letter that is not plain English and explain the two counts
Unwrap.java
import java.io.*;
import java.nio.charset.StandardCharsets;
import java.util.*;

public class Unwrap {

    // The bottom of the stack: bytes, and nothing else. Build everything on this.
    static InputStream raw(String text) {
        return new ByteArrayInputStream(text.getBytes(StandardCharsets.UTF_8));
    }

    // Layer 0: read ONE byte and return the number. -1 when there is nothing there.
    static int firstRawByte(String text) throws IOException { return 0; }  // TODO

    // Layer 0 again: count the bytes, one read() at a time.
    static int byteCount(String text) throws IOException { return 0; }  // TODO

    // TODO: layer 1. Wrap raw() so you get characters instead of bytes, then count them.
    static int charCount(String text) throws IOException { return 0; }  // TODO

    // TODO: layer 2. Wrap layer 1 so you get whole lines.
    static String firstLine(String text) throws IOException { return null; }  // TODO
    static List<String> allLines(String text) throws IOException { return null; }  // TODO

    // TODO: the same job with a Scanner. Compare the code with allLines.
    static String scannerFirstLine(String text) { return null; }  // TODO

    // Is one byte one character for this text? Answer it by measuring, not by guessing.
    static boolean oneByteIsTheWholeCharacter(String text) throws IOException { return true; }  // TODO

    public static void main(String[] args) throws IOException {
        // TODO: print the byte count and the character count for "Atlas" and for "é"
    }
}
Hint 1
The stack is new BufferedReader(new InputStreamReader(raw(text))). Build it inside out, and after each layer write down what you can now do that you could not before.
Hint 2
readLine() lives on BufferedReader and nowhere below it. That one method is the reason the outer layer is there at all.
Hint 3almost the answer
For é the byte count is 2 and the character count is 1, because UTF-8 spends two bytes on that letter. The first raw byte is 195, and the letter is code point 233. Reading one byte gave you a fragment, and putting the fragments back together is exactly the job InputStreamReader does.
What this is really testing

Whether the three lines you have used since Phase I are now three understood objects rather than a spell you copied.

B

Measure the System Calls

Hard·30 min·30 points

checkedex-8-1-b

Read the same file two ways and measure the difference.

Before running, predict the ratio. Most people guess something like twice as slow. It is far more than that.

Then explain the gap correctly. The trap is to say “buffering moves more bytes at once”. It does not move more bytes. The same number of characters cross the same boundary. What changes is how many times your program has to stop and ask the operating system for help.

Finish by switching to readLine() and measuring again. Then notice something: FileReader does not have a readLine method at all. Work out which layer added it, and you have understood the decorator design.

What to write

These exact signatures. The checks call them by name, so a different name or a different parameter type will not build.

class Buffered {
    static Path makeFile(int lines)
    static void deleteFile(Path path)
    static int countRawBytes(Path path)
    static int countBufferedBytes(Path path)
    static long timeRawBytes(Path path)
    static long timeBufferedBytes(Path path)
    static long timeUnbufferedChars(Path path)
    static long timeBufferedChars(Path path)
    static int countLines(Path path)
    static long timeByLine(Path path)
    static boolean fileReaderHasReadLine()
}

makeFile builds its own temp file with Files.createTempFile, so nothing here depends on a file already in the repo, and deleteFile removes it. Every time method returns milliseconds and must actually read the whole file. The checks warm the JVM up first and then compare ratios, never fixed millisecond counts.

What your program must do

  • Read the same file five ways and record every timing
  • Confirm all of them move exactly the same bytes
  • Explain the byte gap in terms of system calls, not bytes moved
  • Compare the byte gap with the character gap and account for the difference
Buffered.java
import java.io.*;
import java.nio.file.*;

public class Buffered {

    // Given. The exercise makes its own file and cleans it up.
    static Path makeFile(int lines) throws IOException {
        Path path = Files.createTempFile("layers-of-logic-buffered-", ".txt");
        try (PrintWriter out = new PrintWriter(Files.newBufferedWriter(path))) {
            for (int i = 0; i < lines; i++) out.println("unit " + i + " readiness " + (i % 100));
        }
        return path;
    }

    static void deleteFile(Path path) throws IOException {
        Files.deleteIfExists(path);
    }

    // ---- bytes. One read() on a FileInputStream is one system call. ----

    // TODO: read the file one byte at a time with a plain FileInputStream
    static int countRawBytes(Path path) throws IOException { return 0; }

    // TODO: the same, wrapped in a BufferedInputStream
    static int countBufferedBytes(Path path) throws IOException { return 0; }

    // TODO: time each of the two above. Return milliseconds.
    static long timeRawBytes(Path path) throws IOException { return 0; }
    static long timeBufferedBytes(Path path) throws IOException { return 0; }

    // ---- characters. Predict this gap before you measure it. ----

    // TODO: one char at a time from a plain FileReader
    static long timeUnbufferedChars(Path path) throws IOException { return 0; }

    // TODO: one char at a time from a BufferedReader
    static long timeBufferedChars(Path path) throws IOException { return 0; }

    // ---- lines ----

    // TODO: count the lines with readLine
    static int countLines(Path path) throws IOException { return 0; }
    static long timeByLine(Path path) throws IOException { return 0; }

    // TODO: does FileReader have a readLine method? Ask it, do not guess.
    static boolean fileReaderHasReadLine() { return true; }

    public static void main(String[] args) throws IOException {
        Path path = makeFile(120_000);
        try {
            // TODO: print all five timings and compare the two ratios
        } finally {
            deleteFile(path);
        }
    }
}
Hint 1
Both loops call read() the same number of times and the same bytes cross the boundary. What changes is what happens inside each call.
Hint 2
On a raw FileInputStream every read() is a system call: your program stops, the operating system takes over, one byte comes back, control returns. Wrapped in a BufferedInputStream, one system call fills an 8192 byte array and the next 8191 reads are served from memory.
Hint 3almost the answer
Now the part that surprises people. FileReader is nowhere near as slow as the raw byte stream, because it already holds a byte buffer of its own. What BufferedReader adds on top is a character buffer, and a readLine method that no layer below it has. Two different wins, and only the first one is about system calls.
What this is really testing

Whether buffering is an explanation you read or a number you produced. The gap is large enough to be obvious, which makes this a satisfying thing to measure.

C

The Scanner Skips a Question

Real work·20 min·20 points

checkedex-8-1-c

A program that appears to skip a question. It does not skip anything, and nothing failed.

Run it, then print the length of the name to prove the String is empty rather than null. Something was read successfully. It was just zero characters long.

Explain exactly what is left sitting in the stream after nextInt() and why the next nextLine() finds it immediately.

Then fix it three ways and compare them honestly. One of the three is the fix most people use, and it is the worst one, because it leaves a line of code that does nothing visible and that every future reader has to work out from first principles.

What to write

These exact signatures. The checks call them by name, so a different name or a different parameter type will not build.

class Skipped {
    static int buggyId(String input)
    static String buggyName(String input)
    static boolean buggyNameIsNull(String input)
    static String fixWithThrowawayNextLine(String input)
    static String fixWithParseInt(String input)
    static String fixWithBufferedReader(String input)
    static int fixedId(String input)
    static boolean throwawayLineIsTheClearestFix()
}

The input arrives as a String rather than the keyboard, so the checks can feed it several shapes. Build a Scanner over it with new Scanner(new ByteArrayInputStream(input.getBytes(UTF_8))). The buggy pair keeps the bug on purpose. The three fixes must all return the same name, including when the name has spaces in it.

What your program must do

  • Show the name comes back empty rather than null
  • Explain exactly what is left in the stream after nextInt
  • Fix it three different ways and make all three handle a name with spaces
  • Say which fix you would ship, and why
Skipped.java
import java.io.*;
import java.nio.charset.StandardCharsets;
import java.util.Scanner;

public class Skipped {

    // Given. A Scanner over a String, so the input is repeatable.
    private static Scanner scannerOn(String input) {
        return new Scanner(new ByteArrayInputStream(input.getBytes(StandardCharsets.UTF_8)),
                StandardCharsets.UTF_8);
    }

    // The bug, kept on purpose. Run it before you fix anything.
    static int buggyId(String input) {
        try (Scanner scanner = scannerOn(input)) {
            return scanner.nextInt();
        }
    }

    static String buggyName(String input) {
        try (Scanner scanner = scannerOn(input)) {
            scanner.nextInt();
            return scanner.nextLine();       // appears to be skipped
        }
    }

    // TODO: is the name null, or is it something zero characters long? Do not guess.
    static boolean buggyNameIsNull(String input) { return true; }

    // TODO fix 1: consume the leftover newline and throw it away
    static String fixWithThrowawayNextLine(String input) { return null; }

    // TODO fix 2: read whole lines only, and parse the number yourself
    static String fixWithParseInt(String input) { return null; }

    // TODO fix 3: no Scanner at all
    static String fixWithBufferedReader(String input) throws IOException { return null; }

    // TODO: the id, read the way fix 2 reads it
    static int fixedId(String input) { return 0; }

    // TODO: after writing all three, decide. Is the throwaway line the clearest?
    static boolean throwawayLineIsTheClearestFix() { return true; }

    public static void main(String[] args) throws IOException {
        String input = "101\nAtlas Recon\n";
        // TODO: print the buggy name and its length, then all three fixes
    }
}
Hint 1
Check buggyName(input).length() before you check anything else. Nothing failed. Something was read, and it was zero characters long.
Hint 2
nextInt() reads the digits and stops. The Enter key you pressed is still sitting there. nextLine() then reads from the cursor to the next newline, which is right there, so it returns an empty String.
Hint 3almost the answer
Three fixes: a throwaway scanner.nextLine(); read everything with nextLine() and use Integer.parseInt on a trimmed line; or use BufferedReader throughout. The second and third are better, because the first leaves a statement that does nothing visible and that every future reader has to decode.
What this is really testing

Whether you can explain the nextInt-then-nextLine bug rather than working around it by superstition. Almost every Java beginner hits this and most never learn why.

D

Load the Registry From a File

Real work·35 min·30 points·The Registry

checkedex-8-1-d

Read structured data from a file, which is what most real programs spend their time doing.

The sample data has a bad row in it on purpose. A program that crashes on row three and tells you nothing is much less useful than one that reports “line 4: readiness is not a number” and carries on with the rest.

Getting that right means putting the try/catch in the right place. Around the whole loop and you stop at the first problem. Around one row and you keep going, which is what you want.

Use try-with-resources from the start. It closes the file whatever happens, including when something throws, and it is one of the few pieces of Java syntax that is strictly better than what it replaced.

What to write

These exact signatures. The checks call them by name, so a different name or a different parameter type will not build.

class LoadRegistry {
    record Unit(String name, int id, double readiness, boolean active)
    record Loaded(List<Unit> units, List<String> problems)
    static Path writeFile(String contents)
    static Path writeSample()
    static void deleteFile(Path path)
    static Loaded load(Path path)
    static double averageReadiness(List<Unit> units)
}

writeSample writes the sample CSV to its own temp file with Files.createTempFile, so nothing depends on a file already in the repo. load skips the header, returns the rows it could read, and adds one problem string per row it could not. Each problem must start with "line N: " where N counts the header as line 1. Bad rows are skipped, never fatal. A missing file still throws.

What your program must do

  • Write the sample file, then read it back
  • Use try-with-resources so the file closes even if something throws
  • Skip the header, and report every bad row with its line number
  • Keep going after a bad row rather than stopping
LoadRegistry.java
import java.io.*;
import java.nio.file.*;
import java.util.*;

public class LoadRegistry {

    record Unit(String name, int id, double readiness, boolean active) { }

    // What one pass over the file produced: the good rows, and a note per bad one.
    record Loaded(List<Unit> units, List<String> problems) { }

    static final String SAMPLE = """
            name,id,readiness,active
            Atlas,101,88.5,true
            Beacon,102,42.0,true
            BadRow,not-a-number,50,true
            Cipher,103,95.5,false
            """;

    // Given. The exercise makes its own file and cleans it up.
    static Path writeFile(String contents) throws IOException {
        Path path = Files.createTempFile("layers-of-logic-registry-", ".csv");
        Files.writeString(path, contents);
        return path;
    }

    static Path writeSample() throws IOException { return writeFile(SAMPLE); }

    static void deleteFile(Path path) throws IOException { Files.deleteIfExists(path); }

    // TODO: read it back with try-with-resources and a BufferedReader.
    //   - skip the header, which is line 1
    //   - report every bad row as "line N: something useful"
    //   - keep going after a bad row rather than stopping
    //   - check the field count before reading position 3
    static Loaded load(Path path) throws IOException {
        return new Loaded(List.of(), List.of());  // TODO
    }

    // TODO: the average over the units that loaded. Nothing loaded is 0.0, not a crash.
    static double averageReadiness(List<Unit> units) { return 0.0; }

    public static void main(String[] args) throws IOException {
        Path path = writeSample();
        try {
            // TODO: load it, print each unit, print each problem, print a summary
        } finally {
            deleteFile(path);
        }
    }
}
Hint 1
line.split(",") gives you the fields. Check the array length before reading position 3, or a short row throws ArrayIndexOutOfBounds instead of your useful message.
Hint 2
Wrap the parsing of each row in its own try/catch for NumberFormatException. Catching around the whole loop would stop at the first bad row, which is exactly what you were asked not to do.
Hint 3almost the answer
try-with-resources is try (BufferedReader r = Files.newBufferedReader(path)) { ... }. The reader closes when the block ends, including when an exception is thrown, which the old finally version made easy to get wrong. Count lines from 1 as you read, so the header is line 1 and the first data row is line 2.
What this is really testing

Whether you can read structured data from a file, handle the bad rows, and close the file properly. Every real program does this, and the bad rows are always the interesting part.

08

After the credits

Two things from this section come back, and one of them comes back as a warning.

The buffering idea returns in Phase IX as StringBuilder. Joining a thousand Strings with + creates a thousand throwaway objects, because Strings cannot change. StringBuilder keeps a growing array and produces the final String once. Same trick, different resource, and you will recognise it immediately.

The word Stream returns in Phase XI, meaning something completely different. java.util.stream.Stream is about processing collections, has nothing to do with bytes, files or I/O, and shares only a name with everything in this section.

When you meet it, the fact that you can already say what java.io.InputStream is will keep the two apart in your head. Most people never quite separate them.

Threads you opened in this section

Buffering will return in Phase IX. Text and Types