Reliable reconstruction of phase-resolved ocean wave fields from X-band marine radar remains a challenging inverse problem because radar backscatter is influenced by nonlinear imaging processes rather than directly representing sea-surface elevation. Existing empirical inversion techniques rely on modulation transfer functions and site-specific calibration, whereas conventional deep learning models primarily optimize pixel-level reconstruction without explicitly incorporating the governing physics of ocean waves. In this study, a physics-informed three-dimensional Fourier Neural Operator (3D-FNO) is developed to directly reconstruct spatiotemporal sea-surface elevation fields from sequences of X-band marine radar images. To enforce wave-physics consistency, a composite WaveLoss is introduced by integrating reconstruction, spectral, dispersion, spatial-gradient, and significant-wave-height constraints into the training objective. The model is trained and validated using paired synthetic radar observations and sea-surface elevation fields generated from Elfouhaily wave spectra under wind speeds ranging from 5 to 30 m s⁻¹. Model performance is evaluated using reconstruction quality, significant wave height (Hs), and spectral preservation. For the 15 m s⁻¹ test condition, the proposed physics-informed FNO achieved the lowest mean Hs bias (−0.30 ± 0.44 m), outperforming both the conventional 3D FFT/MTF inversion (3.18 ± 2.50 m) and the CNN-DAE model (−0.55 ± 0.54 m). In addition, the proposed method achieved the highest on/off-shell energy ratio (6.70), compared with 1.25 for the conventional 3D FFT/MTF inversion and 0.79 for the CNN-DAE model, indicating superior preservation of physically consistent wave spectra. These findings demonstrate that embedding wave-physics constraints into neural operator learning improves both the physical consistency of reconstructed wave fields and the reliability of wave-parameter estimation, providing a promising framework for operational X-band marine radar applications.