Cerebral amyloid-β (Aβ) plaques are the hallmark biomarker of Alzheimer's disease (AD) and are detectable decades before clinical symptoms. Modifying risk factors associated with Aβ accrual offers an opportunity for AD prevention. While midlife vascular health is linked to AD; there is minimal longitudinal evidence regarding the effect of midlife lipids on Aβ. We examined the association between midlife lipids and Aβ 20 years later. One hundred and twenty-two women had serum lipid profiles in midlife (1992, 45-57 years), and cerebral imaging, genotyping, and cognition measured 20 years later (2012/13, 66-77 years). Imaging was performed in 2012/13 via F-18 Florbetaben positron emission tomography (PET) and standard uptake value ratios (SUVR) were calculated. Lipid profiles and other predictors of high PET-SUVR levels (>1.2) were evaluated using multivariable logistic regression. Increases in low-density lipoprotein (LDL) cholesterol in midlife were associated with Aβ, adjusting for age, education, cholesterol medication, and cognition (AdjOR1.81, 95% CI 1.08-3.01, p = 0.024), but attenuated on adjustment for apolipoprotein E4 (APOE ɛ4). Aβ risk increased in women with APOE ɛ4 and midlife cholesterol >6.2 mmol/L (AdjOR9.59, 95% CI 2.94-31.31, p < 0.001), APOE ɛ4 and LDL >3.3 mmol/L (AdjOR9.00, 95% CI 2.89-28.03, p < 0.001), and APOE ɛ4 and cholesterol to high-density lipoprotein ratio ≥3.25 (AdjOR8.32, 95% CI 2.32-29.89, p < 0.001). Presence of APOE ɛ4 and midlife dyslipidemia compounded the risk for Aβ deposition, although no independent effect of midlife lipids was found. Lipid-modifying treatment in midlife could mitigate the risk of Aβ in women with a genetic predisposition for AD. To better inform prevention, future consideration should be given toward managing dyslipidemia in women carrying the APOE ɛ4 allele.