THE BANDWIDTH OF NBFM (NARROW-BAND FM)
I think it was Wayne Green who started that rumor that NBFM only takes up bandwidth equal to twice the deviation, and can drop to zero bandwidth at very low deviations. But that is not really true; with FM, bandwidth is very roughly equal to the modulation frequency times two plus the deviation times two. The modulation frequency multiplier factor in this equation varies from about 1.2 to 2.0 for different types of modulating signal.
There is a complicating factor, though: officially, bandwidth is considered equal to the measured bandwidth at some number of dB down from either the bulk of the spectral power (in the case of a continuously modulated signal), or from the unmodulated carrier (if it is available for comparison). I have found that different governing bodies define this differently in their radio regulations. In some cases bandwidth is defined as the width of the frequency band containing 99% of the signal energy. In other cases it is considered the width of the frequency band in which the sidecurrent strength equals or exceeds -26dB relative to the unmodulated carrier level. I believe that these definitions are approximately equivalent for a single sine-wave modulating signal, but the differences cause a great deal of confusion.
Here is the glitch: if the deviation is very low, the sidecurrent energy can fall below the official measurement limit, and bandwidth would officially be considered to be zero! However, the total sidecurrent energy in this case would have to be 20 dB or more below the unmodulated carrier level, which is not very strong. And if we completely remove the sidecurrents, which at such low deviations are almost exclusively the first order ones at +/- the modulating frequency from the carrier, then all deviation disappears, because there are no sidecurrents to cause it.
By the way, this would also hold true for an AM signal: if modulation percentage is below about 10%, the total sideband power is less than 1% of carrier power, and each sideband is below -26dB relative to the carrier level, so bandwidth would officially be zero according to those definitions. Of course, we know this is not an accurate picture of the real situation. And likewise, if we completely remove the sidebands, we lose all modulation.
Up until fairly recently, the standard for commercial NBFM was 5 KHz deviation and 25 kHz channel spacing in any given area. More recently, a 2.5 KHz deviation standard has been introduced for 12.5 KHz local area channel spacings. In both standards, 6 dB per octave rising pre-emphasis from 300 to 3000 Hz is applied at the transmitter. (This audio preemphasis essentially turns the FM signal into a PM signal.) Peak clipping is applied, so that deviation will not exceed the maximum for the standard used, and low pass filtering is applied after clipping so that the audio band harmonics generated by the clipping will be suppressed and not cause wideband sidecurrents. I will discuss the 5 KHz deviation system.
A 3 KHz audio signal at 5 KHz deviation has a modulation index of 5/3, or about 1.67. This results in second order sidecurrents at +/- 6 KHz from the carrier of about -12 dB each, referenced to the unmodulated carrier level. Third order sidecurrents at +/- 9 KHz from the carrier appear at about -23 dB each, referenced to the unmodulated carrier level.
A 2 KHz audio signal at 5 KHz deviation has a modulation index of 5/2 = 2.5. It has second order sidecurrents at +/- 4 KHz from the carrier, each at -7 dB relative to the unmodulated carrier; third order sidecurrents at +/- 6 KHz from the carrier, each at -13 dB relative to the unmodulated carrier; fourth order sidecurrents at +/- 8 KHz from the carrier, each at -22 dB relative to the unmodulated carrier.
A 1 KHz audio signal at 5 KHz deviation has a modulation index of 5. It has fourth order sidecurrents at +/- 4 KHz from the carrier, each at -8 dB relative to the unmodulated carrier; fifth order sidecurrents at +/- 5 KHz from the carrier, each at -12 dB relative to the unmodulated carrier; sixth order sidecurrents at +/- 6 KHz from the carrier, each at -18dB relative to the unmodulated carrier; seventh order sidecurrents at +/- 7 KHz from the carrier, each at -25.5dB relative to the unmodulated carrier.
The general consensus is that a 5 KHz deviation NBFM signal is about 15 KHz wide. (In comparison, an AM system would be 6 KHz wide.) But the FM receiver bandwidth has to be a little wider, because our IF filters are not perfect. In an AM system, the filter imperfection causes a little loss of higher frequencies, and a slight improvement in S/N. In an FM system, sloppy IF response causes degraded signal reception in weak signal conditions. A sharp IF filter tends to have response ripple and uneven delay times for frequencies across its passband, particularly near the edges of its passband. This upsets the phase relationship between the sidebands and the carrier, turning the FM into AM at some frequencies. This is in turn causes distortion and loss of high frequency audio. (In an AM system, it usually only causes loss of highs. It takes worse passband distortion to cause much distortion.) The resulting audio-rate variations in amplitude of the FM signal at the detector also increase the effects of noise bursts that come from atmospheric noise and other interference.
The usual solution to the above problem is to use a wider IF passband than actually needed for the transmitted bandwidth alone. This pushes the regions of high delay time distortion out to where there is not much transmitted energy, so it has less effect. However, this makes the receiver wider, and therefore more noise and interference can get into it, so weak signal performance suffers somewhat. It is a compromise solution, but it has worked for many years.
In an AM system, we can have a great deal of sideband trimming without distortion, if selectivity is symmetrical and the carrier is not in a response depression. That technique only affects frequency response. We can even have seriously non-symmetrical response and deliberately use sideband diversity to avoid interference. This technique also produces loss of high frequencies, but it optimizes reception in conditions of bad interference. It can also produce some distortion when linear detectors are used, but the amount of distortion is not great as long as the carrier is not attenuated. Extremely narrow selectivity is not generally applicable with FM using typical FM receiver processing chains (IF limiters followed by discriminators, ratio detectors, etc.).
Advanced techniques improve the situation for FM and AM. Synchronous detection and digital filtering promise to reduce the issue to sideband energy, which will make NBFM and AM nearly equivalent in bandwidth and performance. NBFM will always be a little wider than AM, unless we really use QAM instead of NBFM to eliminate the higher order sidecurrents. This in turn requires power amplifiers with an amplification characteristic that is linear for small modulation percentages (10% or so).
SIGNAL TO NOISE IN DIRECT SPACE-WAVE, LINE-OF-SIGHT CONDITIONS (LOCAL, VHF/UHF, ETC)
FM has a capture effect that usually causes background noise to be lower than it would be for an AM signal of the same unmodulated carrier strength when envelope detection is used. For any amount of deviation above a modulation index of about one, there is a plateau of increase in S/N above that of equivalent-strength AM. In the case of FM with a modulation index of one, with strong signals, FM has a consistent 3dB advantage over AM. As signal level drops, both AM and FM get noisier, but the 3dB FM advantage holds down to the point where the AM signal has noise peaks equal to 79% modulation. Below this point, FM begins to lose its peak noise advantage. At the point where peak AM noise is equivalent to 100% modulation, FM peak noise is 1 dB worse than AM, although the FM 3dB RMS noise advantage holds for another 3dB below this. I think this advantage is due to limiting. Noise is a bunch of little vector lines, tending to hang around a general length, but pointing in all phase directions. In FM, they sometimes point parallel to the carrier, so they are limited out. Only the vector components perpendicular to the carrier affect phase and demodulated output, and there goes 3dB of noise. In the case of AM with envelope detection, noise affects the received carrier amplitude at all phase angles.
Note that in the case of AM using synchronous detection with product detection, noise vectors perpendicular to the carrier can not produce noise output from the detector, pretty much removing the 3dB FM advantage at a modulation index of one.
At a modulation index of four, FM has a 15 dB advantage in S/N over envelope-detected AM at strong signal levels, but the FM peak-noise improvement falls apart very suddenly around 8dB above the point where the AM signal has peak noise equal to 100% modulation. The FM RMS advantage falls apart more gradually, dropping from 15dB at about 6dB above the point where the AM signal has peak noise equal to 100% modulation, down to a 3dB RMS noise advantage at the point where the AM signal has peak noise equal to 100% modulation, and continuing to drop rapidly below the RMS performance of narrower FM at signal levels below that. The FM RMS advantage falls 12 dB over this 6dB carrier level drop, and continues to fall at this rate as the signal continues to get weaker.
If deviation is high, as in FM broadcast, signal to noise gets very good at strong signal levels, but it falls apart at low to moderate signal levels. We experience this as crackling breakup as we drive through signal nulls and weak-signal areas. Also, signal bandwidth is much greater than AM. NBFM (modulation index of 1, or slightly less) is much better than wide FM for weak signal work.
AM is preferred over FM for aircraft communications at VHF. Partially this is because AM does not have the capture effect. A weak distress signal will be heard better under a strong signal from a nearby aircraft. Also, there is not as much of a noise burst at the end of a transmission, before the squelch closes, as there is in FM systems.
SIGNAL TO NOISE IN SELECTIVE FADING, SKYWAVE CONDITIONS (DISTANT, MF/HF, ETC)
Ordinary FM receivers using IF limiting and discriminators do not deal very well with MF and HF FM, because interference and selective fading cause intermittent complete signal cancellations that can produce relatively loud noise bursts from the receiver, similar to the crackling breakup in wide FM noted above. As FM deviation is reduced below the point where signal bandwidth becomes equivalent to AM, the amplitude of the received noise is not affected, but the signal volume drops. Really strong noise such as nearby lightning is limited in the IF and causes a limited amount of peak noise, unlike AM. But weaker noise such as weak lightning and interfering signals have a somewhat worse effect on NBFM than on AM. Both modes benefit from some kind of audio peak clipping to limit the peak amplitude of noise blasts.
CONCLUSION
On MF and HF, I think performance is really a matter of the sideband energy and the detector technology. With current FCC Amateur rules, an NBFM transmitter can produce about the same or slightly more sideband energy as an AM transmitter, for a given official bandwidth. So NBFM has possibilities. For pure communications, SSB makes more sense.
On VHF and UHF, FM is by far the most popular mode, except in aircraft. Digital modes are becoming common, mostly for tactical security. Their performance is always inferior to a good FM system. Clean AM could be a bit more spectrally efficient than FM, but if badly set up it could be a disaster. FM tends to be cleaner if overmodulated, and overmodulation in FM systems tends to be self-correcting, because a severely overmodulated FM signal will break up badly in an FM receiver.
Overdriven SSB also splatters badly, yet this does not affect the functionality of the offending system. For this reason, and because of cost and frequency stability issues, VHF and UHF SSB is not commercially popular.