4.4 Receivers to Combat ISI and Their Analysis
Section 4.3 assumed AWGN, which is a relatively simple channel that allows the transmitted signal to arrive at the receiver without distortion, with the added noise being the only impairment.
Here, the goal is to briefly study the effects of sending a signal through a frequency-selective channel (or, equivalently, a dispersive channel when considering the time domain), such as e. g. the composing the LTI Gaussian channel of Eq. (4.1). Dispersive channels modify their input signals with respect to energy, spectrum, etc. To achieve a good performance in digital communication it is necessary to understand the effects of a frequency-selective channel and mitigate them.
With respect to the performance analysis of transmission systems in the presence of ISI, one strategy is to find an equivalent AWGN model. This allows the analysis to use concepts such as optimal matched filtering and deal with the effect of ISI as if it was non-white noise.
4.4.1 Intersymbol interference (ISI)
With a dispersive channel impulse response , one filtered symbol at the receiver can have a duration that exceeds its own interval of duration and extend to the time interval of neighbors symbols. This is called intersymbol interference (ISI). For example, the impulse response of an average plain old telephony system (POTS) channel has a duration of approximately 10 ms. In this case, if one uses bauds ( ms), a single symbol will influence approximately 25 neighboring symbols (i. e. the ISI will extend over approximately 25 symbols).
For example, assume a PAM with pulses of 100% duty cycle as in Figure 2.5. In this case, at the transmitter, a pulse corresponding to a new symbol begins just after the end of the pulse corresponding to the previous symbol. But the band limited channel can alter this harmony as indicated in Figure 2.6.
4.4.2 Pulse response
A channel that creates ISI is conveniently modeled by its overall pulse response. In the sequel, PAM is assumed for simplicity, but all development is valid for QAM (in this case the signals would be complex-valued) and even higher dimensions.
Note that extending the PAM generation of Block (2.15) to include the channel and receiver leads to the so-called ISI-channel model:
| (4.17) |
The shaping pulse at the transmitter is denoted here as . Also for simplicity, it is assumed that incorporates the impulse response of the transmitter filter used e. g. in Eq. (2.2). Similarly, the receiver system impulse response models all filtering processes that take place at the receiver.
Block (4.17) indicates that achieving zero ISI when dealing with imposes requirements on the overall pulse response
| (4.18) |
which is also called simply pulse response and corresponds to the cascade of all LTI systems used for modeling the complete chain composed of transmitter and channel6
Using Eq. (4.18), Block (4.17) can be depicted as:
| (4.19) |
From Block (4.19) and ignoring the receiver LTI , given that , zero ISI is achieved when . In words, if there is no ISI, when sampling for the -th symbol , this sample depends only on . Any signal that leads to this situation is known as a zero-ISI Nyquist pulse.
The pulse response energy is an important parameter when studying the ISI impact
| (4.20) |
In the sequel, examples of overall pulses and the associated ISI are discussed in the context of PAM.
4.4.3 Examples of intersymbol interference (ISI) in PAM
The following three examples illustrate how impacts a PAM signal, eventually creating ISI.
Example 4.6. Example of continuous-time Nyquist pulse shorter than .
Figure 4.17 suggests a first solution to avoid ISI: constrain the duration of to be shorter than the symbol duration . The drawback of this strategy is that the shorter the pulse, the larger its BW, as suggested by Eq. (A.50).
Example 4.7. Example of continuous-time Nyquist pulse longer than .
Figure 4.18 illustrates that a properly chosen pulse can achieve zero ISI: the amplitude of the pulse simply has to be zero at time instants . This is the distinctive property of a Nyquist pulse. When compared with Figure 4.17, the Nyquist pulse illustrated in Figure 4.18 can have its bandwidth reduced by using a pulse with support longer than .
Example 4.8. Example of continuous-time non-Nyquist pulse.
In contrast to Figure 4.17 and Figure 4.18, Figure 4.19 illustrates the consequence of using a pulse with support longer than and that is not a Nyquist pulse. In this case, taking in account only the power at the sampling instants (the “ISI power”), one calculates 0.834 Watts for the specified samples. The power from this interference would add to the power of the background noise in case they are not correlated.
4.4.4 Some practical aspects concerning the “pulse” to study ISI
The section discusses a model7 to study the performance of a communication system that faces ISI. First, some aspects of practical interest are discussed.
The overall pulse determines the ISI of a system but, as indicated in Eq. (4.18), it has the inconvenience of depending on the channel , which is not under control of the communication system designer. This can be circumvented by assuming that the channel was equalized and the combined effect of channel and equalizer is an ideal lowpass filter . Besides, the spectrum has to be assumed confined within the channel bandwidth .8 Under these two assumptions, does not distort the band-limited and, consequently, in Block (4.17). This way one gets rid of the channel in the model but should remember that the assumption is valid only if the transmitted signal is contained within the channel bandwidth, i. e., for .
With eliminated, the pulse response Eq. (4.18) becomes simply the transmitter shaping pulse. It is also convenient to take in account the filter at the receiver and, with an abuse of notation, call the signal (that is not anymore the channel pulse response):
| (4.21) |
For zero ISI, the signal must be a Nyquist pulse and the issue now is how to design and such that their convolution is a Nyquist pulse. When matched filtering is adopted, Eq. (4.3.2) in this context leads to . This issue is further discussed in Section 4.6, which discusses the square-root raised cosine shaping pulse.
There are applications that cannot afford using matched filtering because, for example, the channel cannot be equalized. In these cases, it may be convenient to use a Nyquist pulse at the transmitter (not a square-root version). For example, high-speed transmission in optical fibers may adopt the raised cosine discussed in Section 4.5 (a Nyquist pulse) at the transmitter and a (non-necessarily “matched”) Gaussian filter (see Eq. (E.46)) at the receiver.
The next sections will discuss the property shared by pulses that achieve zero ISI.
4.4.5 Nyquist Criterion for Zero ISI
As mentioned, a deleterious effect of channel dispersion is ISI, which can be generated by any bandlimited channel (with flat or non-flat frequency response). This section presents a criterion to guide the search for solutions to combat the ISI provoked by an ideal lowpass (or already equalized frequency-selective) channel. A PAM system is assumed. First, the continuous-time model of Eq. (2.15) is adopted and later the discrete-time of Eq. (2.16). The most important aspects are still valid when the channel is bandpass and frequency upconversion is used at the transmitter.
Zero ISI with a continuous-time model
First thing to note is that the convolution of with each impulse leads to the PAM expression
Aiming at a specific symbol , the sample to be observed is
and the associated ISI is
| (4.22) |
such that
If , then a zero ISI transmission leads to , as desired. Otherwise a scaling factor needs to be used.9
Eq. (4.22) indicates that should be zero at multiples of other than to achieve zero ISI. In other words, for being able to perfectly recover the transmitted symbol, has to have the following property:
| (4.23) |
When a shaping pulse obeys Eq. (4.23) it is called a Nyquist pulse. The question now is what are the features a pulse must obey to be a Nyquist pulse?
Instead of dealing only with , for the next development it is useful to adopt the auxiliary signal
which corresponds to sampling via an impulse train of period . Dealing with helps because Eq. (4.23) does not impose any restriction on the values that assumes at instants other than the multiples of . Hence, it is sensible to look at properties of instead of . Interpreting Eq. (4.23), the property of interest for can be stated as having .
The Fourier pair indicates that the property is equivalent to having or some other constant.10 Because determines , the question now is what property of the shaping pulse leads to . In other words, the Nyquist criterion can be derived by interpreting the desired property in the frequency domain.
The spectrum of is obtained by convolving with , the Fourier transform of the periodic impulse train. From Eq. (A.58), one has
| (4.24) |
To have with a constant value over all frequencies, Eq. (4.24) suggests that the replicas of should gracefully compose this constant level when summed up and, there should be no gaps (frequencies for which ) among the replicas.
Assuming is the spectrum of a bandlimited baseband pulse with bandwidth and support from to , it suffices to observe the interaction between and its first replica at the right (positive frequencies), located at . The support of is from to and, as indicated in Figure 4.20, to avoid a gap of between the two replicas, it is mandatory to have . Therefore, for a baseband real signal, the Nyquist criterion for zero ISI is
| (4.25) |
This criterion indicates that a channel with bandwidth limits the symbol rate to .

Example 4.9. Reasoning that leads to the Nyquist criterior for zero ISI. To illustrate Eq. (4.25), Figure 4.21 shows the replicas of , spaced by , that compose . In this case, the bandwidth Hz of is not large enough, and the gaps (where ) between the replicas indicate that it is not possible to obtain .

In fact, changing the triangular spectrum by a rectangular spectrum with , one can achieve , which is the pulse with smallest that achieves zero ISI. Several other options for can achieve zero ISI.
As further discussed in Table 4.3, for an analytic signal representing a bandlimited complex envelope (e. g., the one in Figure 3.6) with bandwidth and with support from to , Eq. (4.25) becomes
| (4.26) |
Zero ISI with a discrete-time model
The discrete-time model of Block (2.16) is adopted now and the emphasis is on showing some code, while equations were prioritized when discussing the continuous-time model. Assuming is the oversampling in Block (2.16), the desired property for the shaping pulse to achieve zero ISI is:
and the periodic impulse train is .
Some delay can be assumed, for example, to avoid non-causal pulses. In this case, the desired property is
and the receiver should find the correct instant to start sampling at the symbol rate.
Because has period , a natural tool to observe its spectrum is the discrete-time Fourier series. But it is convenient to use the DTFT for the spectra of both signals: and . The DTFT of is another impulse train
(recall that continuous-time impulses can be used to represent the spectrum of a periodic signal when one is interested in representing it as a Fourier transform instead of a Fourier series). The spectrum of is
| (4.27) |
Example 4.10. Example of Nyquist pulse with support not longer than oversampling . The function ak_plotNyquistPulse.m can be used to observe that Eq. (4.27) leads to a constant value when the pulse obeys the criterion. Figure 4.22 shows the obtained spectra for a pulse p=[1 1 1 1] with , using Listing 4.4.

1N0 = 1; %sample to start obtaining the symbols 2p=[1 1 1 1]; %shaping pulse 3L=4; %oversampling factor 4ak_plotNyquistPulse(p,L) %graphs in frequency domain
Note the pulse has non-zero samples only for and the oversampling is . Hence, it should be expected no ISI in this case. The top and bottom plots in Figure 4.22 show and , respectively. The plot in the middle illustrates that, according to Eq. (4.27), for there are four replicas of that sum up to compose .

Besides Figure 4.22, it is instructive to analyze the pulse p=[1 1 1 1] (or ) in time-domain. This is the goal of Figure 4.23, which shows on the top graph. Assuming the transmitted symbols are symbols=[3 -1 3 1], the bottom plot shows the result of the convolution between the upsampled symbols and , which corresponds to the transmitted signal. This graph shows the sampling instants, where the symbols can be recovered, as specified by N0 = 1. The middle graph shows the effect of shifting and scaling it by the corresponding symbol. For example, the parcel corresponding to the first symbol is . Note that for , there is only one parcel that has a non-zero value, as requested for zero ISI.
Example 4.11. Example of Nyquist pulse with support longer than oversampling . As previously discussed, it is possible to use a pulse with duration longer than samples and still achieve zero ISI. This is useful in practice because a longer shaping pulse (think it as a FIR filter) can have sharper transitions from passband to stopband, which will determine the effective filter’s bandwidth. To exemplify this possibility, it is assumed a pulse p=[0 -1 2 1 1 -2 0], oversampling and ak-here Figure 4.24 and Figure 4.25 were then obtained with Listing 4.5.


1N0 = 4; %sample to start obtaining the symbols 2p=[0 -1 2 1 1 -2 0]; %shaping pulse 3L=3; %oversampling factor 4ak_plotNyquistPulse(p,L) %graphs in frequency domain
It is not that evident, but Figure 4.25 indicates that the ISI is zero. Its bottom graph informs that the four symbols were perfectly recovered at samples . The value N0=4 leads to the first symbol being obtained at (N0-1). The bottom graph of Figure 4.24 shows that is a constant over , with a numerical error of .
Careful observation of the middle graph of Figure 4.24 indicates that, considering only the magnitude, the replicas of do not sum to a constant. Due to the delay imposed by N0, the phase must be taken in account, which was not necessary in Figure 4.22.
As discussed in Application 4.3, there is considerable flexibility for designing zero ISI pulses. But, in practice, two requirements are imposed when interpreting the pulse as a filter: 1) a relatively small bandwidth and 2) a large attenuation at the stopband. The raised cosines are popular options and are discussed in the sequel.
4.4.6 Cursor, precursor ISI and postcursor ISI
The cursor was briefly mentioned in Section 2.12.3. Generally, given a signal waveform, a reference time instant is sometimes called cursor. The segments of the signal before and after this point are called precursor and postcursor, respectively. The next paragraphs, apply the concept of cursor in the specific scenario of a digital receiver, in which it is also called synchronization delay, timing synchronization delay, or delay due to timing synchronization.
Considering again the transmitted and received signals in Figure 2.5 and Figure 2.6, respectively, note that it is not trivial to find the instant to sample the signal at the receiver (or the cursor) given a distorted signal, as in Figure 2.6.
Before defining the cursor more accurately, it is useful to observe in Figure 4.26, two examples of channel pulse responses to a single symbol (“one-shot”) of a PAM signal . In this case in case (a) spans approximately and potentially causes less ISI than case (b).

The timing references in Figure 4.26, with vertical lines separated by have the transmitted signal as reference. In the context of ISI analysis, it is sometimes convenient to have the timing specified by the receiver synchronizer as indicated in Figure 4.27.

Figure 4.27 indicates the cursor time (or simply cursor), which is typically estimated by the receiver’s synchronization algorithm. For a given reference symbol , the cursor is defined here as the time delay between the instant was generated at the transmitter and the time estimated by the receiver to obtain the amplitude value that will represent and be used for making the decision regarding the symbol. Considering distinct symbols as the reference (time “0”), the estimated cursor values may vary, and is their average.
Assuming the signal that will be used at receiver to make the decisions is already at the symbol rate , the samples of the pulse response separated by from may impact the decision of other symbols. In Figure 4.27, three samples before the cursor (precursor samples) may impact the three symbols that are transmitted before the reference symbol, while the sample at time impacts the symbol transmitted just after the reference symbol. This is an important aspect of ISI that can be summarized as follows:
-
Precursor: interference provoked by future symbols, which can be modeled as anticausal with respect to the reference symbol, and it is created by the parcels of before the cursors of the future symbols.
-
Postcursor: interference provoked by past symbols and created by the parcels of after the cursors of the past symbols.
An example using from Figure 4.26(a) is described in the next paragraphs.
Because the channel is LTI, it is relatively easy to observe its response to a sequence of symbols and interpret the final as the summation of the contribution of each symbol. Figure 4.28 illustrates a case with four PAM symbols composing and the corresponding responses that should be added to compose .

Observing Figure 4.28, suppose that the automatic synchronization procedure chooses (or for simplicity) as the indicated cursor to obtain the sample that represents the symbol transmitted at time . In this example, the signal at is the summation of three parcels: the parcel of interest, the precursor ISI (which in this case is provoked solely by the next symbol sent at ) and the postcursor ISI (provoked by the previous symbol, transmitted at ). In this case, only the two neighboring symbols are interfering with the reference symbol but, in general, due to the channel dispersion, it is possible that the symbol of interest suffers interference from many other symbols. As mentioned, the influence of “future” symbols can be conveniently modeled using noncausal signals and systems.