I think one trap people fall into (myself included) is confusing the expression of fitness versus what produces it. It's easy to remember all the activites a person did when he/she was younger, such as playing sports all day, dancing all night, etc. And from there, it's easy to make the association that if you perform these same activities now, you will be as fit as you were then.
This is a bit deceiving because when a person is younger, a lot of their fitness is "built-in". Hormones are at peak levels, aerobic capacity is high, muscle mass and strength are at good levels. It is these things that allow a young person to perform and enjoy a variety of physical activities, but it is not necessarily true that these activites are producing any fitness.
As a person gets older, the body switches into "decay mode". Then, unfortunately, you have to put in work to maintain all the different fitness components. This is sometimes not fun, but it is necessary if you want to continue enjoying physical activities as you age.
