 Preface: RADAR analysis is a complicated topic. I had hoped to write this page in a manner easily accessible to friends, family, and random internet users. I've resigned to the conclusion that a worthy explanation requires more time and words than I care to invest. Instead, here is a smattering of examples that represent my work well.
There is a considerable amount of a processing involved to transform the raw radar signal into ocean data products. My MSc focused on the mathematical equations and processing algorithms that produce the final data products, and concluded that some of the more difficult data products were suffering from insufficient signal quality, rather than an error in theory. Motivated by this conclusion, I redirected my attention to the low-level behavior of the radar, i.e. the operation and performance. Analysis at this stage is with the original raw data, which is too dense to save. It must be automatically processed and reduced in real time, retaining the interesting information. ------------------------- Quality Control ------------------------- One feature desperately missing from the processing chain was Quality Control; the identification of bad data. These are highly sensitive radars operating on low power, making them susceptible to external errors, e.g. massive Australian defense radars. Occasionally, the radar would have internal problems, often due to the salt and rust from their seaside location. I wrote programs to automatically identify unusable data, and quietly exclude it from the further analysis. These quality controls were robust, in that they were accurate at separating the good from bad. This is a must for automated processing of large data sets. Ask any grad student sorting through MBs/GBs of non-QC'ed data. ------------------------- External Problems ------------------------- A radar is much like a radio in that it listens to an electromagnetic wave as a signal. And like a person at a busy party, it hears every conversation in the room. The human brain has some really neat tricks at "tuning out" the unwanted conversation1, but it is impossible to completely turn off the unwanted signals; they are invariably mixed in. When I began my research, I was presented with a complete data set and asked to investigate physical ocean processes. Upon inspection, it rapidly became clear the final data product contained far too much error to be of any use in further analysis. This is the point at which a reasonable graduate student would go looking for a different thesis. But my inherent mentality is "I can fix this". Specifically, step backwards through the processing, find the error, negate or remove it, and proceed. In my defense, any physicist would be enticed by a finished, and dense, data set.
 Working backwards, I eventually discovered the data was literally saturated with strong signals coming from specific directions. Ray-tracing showed the worst offender was coming from northern Australia. My best guess is the culprit was the JORN defense radar; since it's big, powerful, and covers our frequency.
To some extent, this might have been avoided with proper monitoring. Although, there is no such thing as quiet frequency bands. 
------------------------- Noise Removal ------------------------- In my thesis, I investigated a variety of methods for removing the external interference, or statistically improving the data. Ultimately, I concluded that none of these methods were satisfactory.2 Two of the more interesting attempts were a model-based method and eigen-decomposition.   ------------------------- Ship Calibrations ------------------------- For the radar to function, each antenna in the array has to have a known phase. Simply put, we have to know exactly when the EM wave arrived at the antenna. The propagation of the EM wave is affected (delayed) by the environment, the antenna, the cable, the radar hardware, and probably other sources I'm forgetting. All of these effects have to be known and removed. Our solution was to run a field calibration using a ship, a transmitter, and a GPS. Since the position of the ship is known from GPS, many of the phase effects can be quantified.  The ship traveling in a semi-arc around the antenna array. I dug fairly deep into the analysis of phases from multiple ship calibrations. My primary conclusion was the current calibration method was insufficiently accurate. This was because a single scalar value was being used to align the antenna's phases. But the results showed large, stable variations in phase as a function of EM incidence angle. The best analogy is wearing a pair of glasses where the lenses are wavy and warped. For standard processing, the phase origin is arbitrary; it has no effect on the results provided it remains constant. The origin defines a moment in time and space to which all the phases are referenced. For our calculations, only the relative phase between antennas matters. But, for the purpose of graphing and visualizing the phase information, a phase origin needs to be defined. Otherwise, the cyclic nature of phase makes it impossible to see any patterns. The previous processing method was defining the first antenna as the phase origin. This worked well for removing most of the shared variation and revealing the relative trends. It also had the unfortunate effect of doubling the intrinsic phase error for all antennas except the first. In this context, the "shared variation" can be equated to the mean phase. I found improved accuracy by defining the mean phase as: R = X'*Xmean phase = angle(mean(R))Where X is the complex timeseries voltage matrix, X' is its Hermitian transpose. R is the the un-normalized complex correlation, with the matrix product taken over time, yielding a NxN matrix, where N is the number of antennas. The correlation matrix R describes the rotation angle needed for maximum correlation between any two antennas. Taking the complex mean of R gives the rotation angle for maximum correlation to all antennas.3 Unwrapped phase. The trend is due to the ship's motion through space. Wrapped phase. The trend is due to the antennas spatial separation.  We had some problems at Koko Head with antenna mis-ordering. I discovered the error during analysis when the phase functions were clearly incorrect. The only prior indication was mediocre beamforming results. This sort of mishap is to be expected with dozens of cables and connectors. I would argue for software with auto-calibration and error detection. There are several good reasons not to do this. Once calibration phase data is obtained, one can estimate the "antenna functions" which I will define as the residual phase error as a function of EM incidence angle, after EM propagation theory has been taken into account. For the rest of this discussion I will refer to these phases as the phase residuals, since they are the difference between theoretical and measured phase. These are the phase residuals for all antennas.  These residuals are a strong argument against using a single phase calibration value; only an empirical function could remove that variation. 
Linear regression revealed a progressive phase trend between antennas. This prompted another round of verification that indeed the spatial measurements were correct. Frustrated by the inability to explain a clear and consistent trend, I tried a broader regression analysis using all the data. The regression model was all possible incidence angles in 3D space, using quaternions and the 4th-dimensional parameter as the error metric, i.e. minimizing the shortest-path angular error.  * To the extent of my knowledge. I can't claim to know the processing of the various research groups. Later on, we further confirmed these results using data from a separate site and the position of a known transmitter. ------------------------- Beamforming ------------------------- After quantifying the phase residuals, we were curious how they would propagate through the processing chain. The next step is beamforming, where the signals from multiple antennas are combined to yield directional information. I cannot think of any normal-life analogy, but I suspect if you've read this far you probably don't require an explanation ;) I termed my method "Observed Phase" beamforming. That is, I used the phase residuals as observed from the above ship calibrations. I allowed for each channel to have a scalar phase correction, in accordance with standard processing. The most salient result was that windowing functions are ineffective, even detrimental, in the presence of phase residuals. Windowing functions rely on the principle of coherence for constructive and destructive interference. Furthermore, their weights are very precisely calculated to coincide with the peaks and nulls of the sinc function. Significant phase errors ruin this careful alignment. The results can be seen in the figures below. Non-windowed and windowed beamforming on the left and right, respectively. The windowed and beamformed results (red) shows the inevitably broadening of the center beam, yet fail to reduce sidelobes as expected (black). Any hope for improved dynamic range is gone. Windowed or not, the beamforming is quite robust; yielding acceptable array functions despite the large phase residuals. Footnotes --------------------- 1 This feature has gotten me in trouble numerous times. 2 I could write an extended rant about the scientific bias against publishing or reporting on negative results. We must also share knowledge about what doesn't work, or continue to re-invent the square wheel. 3 The complex mean also weights the result by the magnitude of the correlation. This could be viewed as beneficial, e.g. suppressing weak channels. |
