In hindsight, this can be seen as a turning point for Astaire & Rogers - while still very profitable, it was less so than Swing Time and the next two would record losses. I'm trying not to be unduly influenced by hindsight (probably inevitable, but I try) but it did seem to be weaker - despite containing some of the most famous songs in the series, 'Let's Call the Whole Thing Off', 'They All Laughed' and 'They Can't Take That Away From Me'.
The movies generally had silly stories but this one seemed especially silly - something about Astaire and Rogers pretending to be married. I think they got away with it earlier on but it was starting to grate by now - all the contrived misunderstandings and so on. The supporting players felt overly familiar - Eric Blore, Edward Everett Horton, etc. Also it lacks some of the energy of the earlier works.
There's still plenty of incredible dancing and professionalism on display: tap solos, an African American soul number (common at the time eg Day at the Races), some waltzes, a masked dance at the end. It's good, just not as good as what had come before.