Grep's Buffering + Pipes

I got bit by this again today, so I’m writing it down so I can reference it later.

When running docker logs -f CONTAINER_NAME | grep -v "foo" | jq, it is important to consider grep’s buffers.

I am using GNU grep and, by default, it uses block buffering when not connected to a terminal (which is my case, since I piped it to jq). This means jq doesn’t get any input until a large-ish amount of text (4k? 8k?) is sent to grep. Since container I was inspecting logs little, I got zero log lines.

To quote man grep:

–line-buffered

Force output to be line buffered. By default, output is line buffered when standard output is a terminal and block buffered otherwise.

This works:

docker logs -f CONTAINER_NAME | grep -v --line-buffered "foo" | jq

Julia Evan’s has a wonderful blog post on this: Why pipes sometimes get “stuck”: buffering

TLDR:

Use --line-buffered with grep when piping its output.