Walter first appears as a prisoner in Episode 2: Taken. He begs to Alan to turn the lights on and admits that he's in jail for beating someone up. He then appears The Anderson Farm in The Truth, having gone back to get more moonshine, but is attacked by his now taken friend Danny. He is attacked and is found bleeding on the floor, dying shortly after he's finished talking.
He appears once more in "The Writer" in Alan's Mind when Alan runs around a giant wheel and Walt is begging for the lights to be turned on.