Charlie McAdam grew up on Springvale Station, near Halls Creek, Western Australia. In 1947, when McAdam was around ten, he was removed from his family and taken to Beagle Bay Mission. By the time he was fifteen he decided he had enough of the starvation and floggings at Beagle Bay Mission, so he started causing trouble so that the priests would kick him out, which they did, sending him on his way in just a singlet and shorts.

For most of his life, McAdam worked as a stockman. He mustered horses and cattle, branded stock, broke in horses and eventually worked as head stockman. While McAdam was a young man, he boxed for money and worked as a rough rider at rodeos. When his son died of pneumonia, McAdam decided to quit work that took him away from his family. He settled down in Alice Springs with his family. McAdam took on jobs with the Central Australian Aboriginal Legal Aid Service, McMahon constructions, Department of Works and Housing, and the rural section of the Institute of Aboriginal Development, while Yami Lester was the director. Although McAdam was settled in Alice Springs every few years he would do mustering work.