Walter first appears as a prisoner in Episode 2: Taken, drunk on moonshine. He begs to Alan to turn the lights on and admits that he's in jail for beating someone up. He then appears on The Anderson Farm in Episode 4: The Truth, having gone back to get more moonshine, but is attacked by his now taken friend Danny. He is attacked and is found bleeding on the floor, dying shortly after he's finished talking.
He appears once more in The Writer in Alan's Mind when Alan runs around a giant wheel and Walt is begging for the lights to be turned on.