Randolph was born Emanuel Hirsch Cohen on June 1, 1915 in the Bronx; he attended City College prior to studying at the Federal Theater Project and with acclaimed acting teacher Stella Adler. He made his Broadway debut in 1938 in a production of William Shakespeare's Coriolanus and also appeared in the original productions of Come Back, Little Sheba, Command Decision, Seagulls Over Sorrento, Paint Your Wagon, The Visit, The Sound of Music, and Mother Courage and Her Children, in addition to many other shows.

Called before the House Un-American Activities Committee in the 1950s, Randolph was blacklisted when he refused to testify. He returned to film and television work in the mid-1960s; his film credits include Escape from the Planet of the Apes, Serpico, Heaven Can Wait, Prizzi's Honor, Christmas Vacation, A Price Above Rubies, and You've Got Mail. On the long-running Roseanne TV series, he played the title character's father.

Randolph's wife, actress Sarah Cunningham, died in 1986. The actor is survived by his daughter, his son, his brother, and his granddaughter.