Why do deep networks behave like piecewise linear functions?

ReLU activations partition space into linear regions.

Each ReLU neuron creates a linear boundary. Composing many layers creates exponentially many linear regions. Thus, deep networks are globally nonlinear but locally linear in each region.