He was still a notable star from, lets say, 1980 until 1985, the period of So Fine, Partners, Irreconciliable Differences and Fever Pitch. He walked away from The Champ, was in line for The Thorn Birds and didn't do Frst Blood. What could he have been?
I went through the list of top box office films from 1980 to 1985. It was a period of franchises (Bond, Burt in a car), comedies
1980 - don't laugh but American Gigolo, Flash Gordon.Definitely Somewhere in Time, Altered States.
1981 - Body Heat
1982 - An Officer and a Gentleman, Night Shift, Best Friends
1983 - Trading Places, The Big Chill, High Road to China
1984 - Romancing the Stone
1985 - Jagged Edge
There were a number of movies with roles that would've suited a young Ryan O'Neal type - Ordinary People, Caddyshack, Endless Love, Young Doctors in Love, Taps, Splash, Police Academy
Okay so there is my take out: there simply weren't as many star parts going that O'Neal would've been good for. Young O'Neal type roles, yes, plenty - that's what Tim Hutton, Tom Hanks, Steve Guttenberg got famous on. Young, funny handsome types. But leads in comedies went to more aggressive TV comedians like the SNL crowd, Richard Pryor, Michael Keaton and Steve Martin.
There was also a sharp drop in female driven comedies/ melodramas. These were strong in the 70s still with Jane Fonda, Babs, etc. But in the 80s it was fairly grim.
Some actors did have careers that might've suited O'Neal. William Hurt roles - especially Body Heat - would've suited him to a T. Ditto Richard Gere. He could've easily played Tom Selleck parts.
But the biggest problem... I think he just didn't appear in enough quality productions. And it does tell. Like look at Jeff Bridges - flop after flop but he keeps appearing in good things whjch mitigates. So he remained a kind-of star.
I think that's the trick if a star - you won't be one forever, so just try to make as much quality stuff as you can.