A critical question in the global warming debate concerns the causes of the observed trends of the Southern Hemisphere (SH) atmospheric circulation over recent decades. Secular trends have been identified in the frequency of occurrence of circulation regimes, namely the positive phase of the Southern Annular Mode (SAM) and the hemispheric wave-3 pattern which is associated with blocking. Previous studies into the causes of these secular trends have either been purely model based, have not included observational forcing data or have mixed external forcing with indices of internal climate variability impeding a systematic and unbiased attribution of the causes of the secular trends. Most model studies also focused mainly on the austral summer season. However, the changes to the storm tracks have occurred in all seasons and particularly in the austral winter and early spring when midlatitude blocking is most active and stratospheric ozone should not play a role. Here we systematically attribute the secular trends over the recent decades using a non-stationary clustering method applied to both reanalysis and observational forcing data from all seasons. While most previous studies emphasized the importance of stratospheric ozone depletion in causing austral summer SH circulation trends, we show observational evidence that anthropogenic greenhouse gas concentrations have been the major driver of these secular trends in the SAM and blocking when all seasons are considered. Our results suggest that the recovery of the ozone hole might delay the signal of global warming less strongly than previously thought and that effects from all seasons are likely crucial in understanding the causes of the secular trends.