Spectral modeling of extreme and breaking wave sea states