Heck, now he's played six years on professional teams -- longer than he played on teams growing up. He didn't start until his sophomore year at Madison County (Fla.) High School.
"I just never got into it," Cain said Friday at the Royals FanFest. "It was just never my thing. In high school, I kind of picked it up and went with it."
Now, at age 24, Cain has a shot at the Royals' center-field job after being acquired from Milwaukee in the six-player trade that netted the Brewers pitcher Zack Greinke.
A friendly man with a big smile, Cain just didn't play much of anything as a kid in Madison, a little town about 50 miles from Tallahassee.
"My dad passed away when I was 4 so my mom pretty much raised me and my brother, so it was just us three growing up," he said. "She was a single mom, she worked two jobs at times so that's probably the main reason I didn't play sports. She didn't have really time to take me to practice or whatnot."
He and brother Delvin, who is six years older, had their chores to help out Patricia Cain.
"Somebody had to keep the house clean so that was me and my brother's job," he said.
When Cain got to the ninth grade, he tried out for the basketball team and got cut. His mom thought football was too rough.
"She wouldn't let me play football so what was left? Baseball," he said.
He talked to a buddy who played high school ball -- Jeremy Haynes, now a pitcher in the Los Angeles Angels organization. He got encouragement and some practice-field help.
"We went out and hit a little early sometimes and did some defensive drills or whatever. So I ended up trying out for the team and made the JV team," Cain said.
After a year with the JV, as a junior he moved up to become a varsity bench-sitter. He didn't care for that.
"I just wasn't content sitting on the bench. That wasn't for me," he said. "I went out and worked hard and made sure that didn't happen."
Cain played well enough that scouts followed him and letters from big league clubs came in the mail. At first, he found that "shocking" because, after all, he was just playing to have some fun. It dawned on him that this could be a way to making a living.
Drafted by the Brewers in the 17th round of 2004's First-Year Player Draft, he didn't sign and became a "draft and follow" while he played a season for Tallahassee Community College. Then he signed with the Brewers and began his pro career in 2005 with a bang; he led the Arizona Rookie League with 73 hits and batted .356 in 50 games.
Royals manager Ned Yost, then the Brewers' skipper, remembered seeing Cain and shortstop Alcides Escobar, who also came in the Greinke trade, in training camp a year or two later.
"The first thing that went through my mind when I saw them was that these kids are going to be All-Stars," Yost said.
Cain had natural athletic ability -- he could hit, steal bases and cover the outfield -- but after getting such a late start in the game, he had a lot of catching up to do.
"I had a lot of tweaking to do," he said, "but I was able to pick it up quick and kind of learn on the fly. I'm the kind of guy, you throw me out there and I just try to get it done."
Not playing as a kid, he wasn't starstruck by any big leaguers.
"But when I started playing baseball, one guy that was my role model was Torii Hunter," he said. "I tried to model my game after him."
Coming back from leg injuries last season, the right-handed batter ripped off a .317 average combined at the Double-A and Triple-A levels and was promoted to the Brewers. In 43 games, he batted .306, had 13 RBIs and swiped seven bases in eight tries.
"I'm still learning," he said. "I'll take my bumps and bruises, but, at the same time, I feel like I can go out there and hold my own."
Cain will go to Spring Training next month with Melky Cabrera heading the center-field list and with Mitch Maier, Jarrod Dyson and Gregor Blanco also in the picture -- a lot of competition.
The Royals have outfitted Cain with uniform No. 6, formerly worn by their legendary center fielder, Willie Wilson. As the FanFest got under way, they posed together with the uniform.
"He was telling me to live up to it," Cain said.
Now that would be something.
Dick Kaegel is a reporter for MLB.com. This story was not subject to the approval of Major League Baseball or its clubs.