Comments on: Spline Smoothing by Steve McIntyre Wed, 20 Jun 2018 04:20:59 +0000 hourly 1 By: Mark T Sat, 12 Sep 2009 20:35:15 +0000 Yes, Hu, that’s what I was getting at when I said “all of them.” I was thinking circular convolution but wrote convolution. I had intended to write up something similar but got sidetracked.
If you want to see how signal processing folks get around the problem (and it is a problem), look up “overlap-add method” over at Wikipedia. That requires, however, successive FFTs from a long string of data, not just a single record the same length as the FFT.
For very large filter applications, the FFT method, even with the added cost of the overlapping, is much, much faster than a convolution. Implementing brickwall filters is still not recommended because there will still be ringing in the time domain, but other filters can be effectively implemented if it is done properly.

By: Hu McCulloch Fri, 11 Sep 2009 23:05:21 +0000 I think I now understand why the DFT Brickwall filter illustrated in #136 and used by Paul Clark gives such goofy end-values.

The following shows the Brickwall weights for y(150) with a cutoff of 15, to operate on data vector x(0:300). This is essentially the same as the graph in #156 above, except now t = 150 is exactly in the center of the sample, since there are now 301 points running from t = 0 to t = 300. The weights are exactly symmetrical about t = 150, and are a discrete modification of the sinc weights that apply in continuous, unbounded time.

The next graph shows the same thing, but now for y(250). The weights have just rotated to the right 100 time units, with the weights that have dropped of the right end being added back on the left end. This rotation occurs because the DFT actually thinks the data is periodic, ie x(301), if observed, would be exactly the same as x(0). In other words, the DFT brickwall filter is equivalent to using the symmetrical filter for the center point, and then endpadding the the missing last 100 or whatever points with the first 100 or whatever points.

With stationary noisy data this wouldn’t have an obvious effect. However, with uptrending data it creates a cyclic effect as the left and right ends of the data alternately get positive or negative weights.

When you rotate the weights clear to the end , the endpadding becomes even more noticeable with trending data, because now x(0) and x(1), etc are getting almost as much weight as x(300) and x(299), etc!

This is why the first and last smoothed values are almost equal in the first graph in #136 above.

The Brickwall DFT filter’s peculiar other-end-endpadding makes its end properties even stranger than Mann’s double-flip or Rahmstorf’s extrapolative endpadding. At least they are endpadding with data from the same end of the data series!

By: sky Tue, 08 Sep 2009 22:25:26 +0000 Yep, there is no free lunch in time-series analysis. HP may be linear, but the smoothing is inconsistent, being dependent on record-length. This also applies to the Fourier composition of any finite data series that some would “filter” by manipulating the F. coefficients. By contrast, genuine FIR filters always produce the same result over the applicable stretch of record. End of story.

By: John S. Tue, 08 Sep 2009 18:20:26 +0000 Re: Basil (#180),

There is a categorical difference between constructed indices and measurable reality. In the case of surface-station data, physical processes–known and unknown–determine the time history (signal) of temperature at the measurement site. “Global” indices constructed from highly incomplete geographic coverage of sites are nothing more than statistical samples of the concurrent average temperature troughout the globe. Such averages, even if known quite completely, do not necessarily conform to the laws of physics, because the dry bulb temperature alone is an inadequate measure of prevailing energy levels. And in your example of stock market indices, until the advent of index funds, futures and options, the common saying about index values was “you can’t trade on them.”

By: John S. Tue, 08 Sep 2009 17:53:36 +0000 Re: Hu McCulloch (#179),

Thank you for revealing that HP uses variable weights for a fixed value of the smoothing parameter. This makes it a linear curve-fitting operation (in contradistiction to proper linear filtering) and renders it unsuitable for scientific work, where the precise effect of data smoothing upon the spectral (or autocorrelation) structure needs to be known for a host of purposes.

By: Hu McCulloch Tue, 08 Sep 2009 15:41:01 +0000 RE #179,
Here is the graph of the HP weights for n = 100 and 5 selected y(t) values.

Graphing them all at once as in Fig. 1 in the post creates a visually interesting pattern, but it is hard to see the individual curves. Unlike the fixed-knot spline, the HP weights have modes that do not oscillate with t.

By: Basil Tue, 08 Sep 2009 13:09:24 +0000 Re: Geoff Sherrington (#173),

The four graphs you provide have mechanisms attributed to them. href=”#comment-354904″>Basil (#134)

What assurance can you give that these mechanisms or interpretations are both necessary and sufficient to explain the response for graphs 2-4?

Take the hypothetical case where climatologists were still unaware of ENSO, but were aware of some other unrelated physical processes. Would the graphs be labelled with these other processes? Would they be any more right or any more wrong than your attributions?

What if there exists an undiscovered mechanism that accounts for parts of graphs 2-4?

Well, we know now that Newtonian mechanics does not account for everything we can observe in the physical world, but the fact that there existed undiscovered mechanisms that account for part of what we observe in the physical world doesn’t invalidate the usefulness of classical mechanics. I didn’t suggest that the mechanisms I was attributing to the various graphs were capable of explaining them comprehensively. I was really making a much less grandiose claim than you would have make. It is a truism that climate varies on all time scales. I was just illustrating that with the graphs. That’s all. I wasn’t proposing a general theory of climate.

By: Basil Tue, 08 Sep 2009 12:50:38 +0000 Re: Hu McCulloch (#179), Since I don’t have matlab, when you can, I’d appreciate seeing your graph of this.

John S, #176

BTW, your chosen example also shows how incoherent the low-frequency temperature variations are in different regions, with differences that are as large or larger than the variations. Similar disparities are often seen between intra-regional stations. A “global temperature signal” is merely a statistical construct, rather than a ubiquitously measurable reality.

Yes, my chosen example illustrated how different the low frequency variation can be regionally, but within the 9 divisional data sets available to the US, they do break out into three roughly coherent regions, along geographically rational subdivisions (the two coasts, and the continental interior). As for a “global temperature signal,” that it is a statistical construct doesn’t mean that there might not be discernible influences on it. Just like stock market indexes remove the “systematic risk” of individual investments, a global average removes a lot of the variance in regional temperatures, leaving behind something notionally indicative of what is happening globally to terrestrial temperatures.

By: Hu McCulloch Tue, 08 Sep 2009 04:38:21 +0000 RE JohnS #178,
Wordpress isn’t letting me upload files to CA at the moment, but here is a MATLAB routine that computes the HP filter and its weights:

function [y,W] = hodrickp(x, lambda)
% y = hodrickp(x)
% y is smoothed version of x
% Formula derived in Ekkehart Schlicht, “Estimating the Smoothing Parameter
% in the So-Called Hodrick-Prescott filter”,
n = length(x);
P = toeplitz([-2 1 zeros(1,n-2)]);
P = P(2:n-1,:); % P generates second differences
W = inv(eye(n) + lambda*P’*P);
y = W*x;
% This routine is inefficient for large n. An interative routine
% based on the Extended Kalman Filter exists that takes
% less memory and is faster.

The weights in W are nearly symmetric near the center of the sample, and become highly skewed towards the ends. They do not respond to the data (x), but do respond to nearness to the endpoints. Furthermore, they are bilateral, in that they reach both forward and backwards (except at the last point, of course). Near the center, the weights have a bell-shaped center, with 2 negative wings, but no oscillations beyond the first overshoot, unlike the “Clark” filter in #156, or even the spline in the post.

By: John S. Mon, 07 Sep 2009 18:22:25 +0000 Re: Hu McCulloch (#175),

HP has been used very little ouside of econometrics and many of us here are trying to fathom its operational principles. If, for any set value of the smoothing parameter, HP indeed uses a fixed linear combination of data uniformly throughout the record, then the frequency response can be calculated from the coefficients (or weights). I’ve not seen this done anywhere. Although one can supposition some correlation structures that would render the Kalman filter no longer data-adaptive, it remains inherently a unilateral (i.e., causal) operation. This raises anew the question of end-treatment. Given the apparent zero-phase response of HP, one expects it to be–by contrast–bilaterally symmetric, with smoothed values provided everywhere by circular treatment of data, similar to IDFT reconstruction. Perhaps you can point us to the econometric literature where these specifics are discussed.