Coastal eutrophication, under intensifying anthropogenic activities and climate change, is increasingly recognized as a nonlinear ecological response to interacting nutrient inputs, hydrodynamics, climate forcing, and sedimentary feedbacks, rather than nutrient enrichment alone. Evidence from urbanized bays suggests clear limitations of nutrient-reduction-only strategies in mitigating harmful algal blooms (HABs) and hypoxia. Conventional nutrient-based indices may therefore be insufficient for assessing ecological hazards in dynamic estuarine-coastal systems. Here, we developed a process-informed, data-driven, and adaptive framework for assessing eutrophication and related ecological risks in Hong Kong coastal waters. A multivariate dataset comprising over ten hydrodynamic, hydrochemical, biological-response, and sedimentary-process variables was compiled for 1993-2024. Principal component analysis identified six dominant system modes, indicating that HABs and bottom-water hypoxia are mainly associated with land-derived nutrient inputs, in situ biological production, vertical exchange, and sedimentary internal processes, while partly moderated by water ventilation, offshore mixing/flushing, and light limitation. Critical-dynamics indicators, including lag-1 autocorrelation, rolling variance, and Shannon entropy, revealed critical slowing-down signals during 2005-2010, suggesting a potential ecological state transition around 2011-2012. This transition may indicate a shift from externally driven HAB-hypoxia risk toward a compound state maintained by sedimentary internal loading and vertical mixing, which could prolong bottom hypoxia, increase red-tide susceptibility, and weaken the benefits of land-based nutrient reduction. Signals after 2020 further suggest another ongoing critical transition. Based on the six system modes, an XGBoost classification model was developed to distinguish ecological states from well-mixed healthy conditions to persistent and intense risk states. The model was trained with 1993-2017 data, validated with 2018-2022 data, and applied to predict eutrophication conditions and ecological risks during 2023-2024. The resulting classification index, derived from routine water-quality and physical variables, captured system states and showed predictive potential. Compared with the eutrophication index (E) commonly used in Chinese coastal waters, the new method showed stronger correlations with ecological risk indicators. Its correlation coefficient reached 0.157 with HAB occurrence across all periods and regions, and 0.294 with red-tide severity in affected waters, approximately eight times that of E based on the same metric. For bottom-water hypoxia events, the correlation coefficient increased by 94.9% relative to E. These results support the integration of system-mode extraction, critical-transition diagnosis, and XGBoost classification for dynamic eutrophication assessment and ecological risk early warning.