Cops, planes, and tires

Here we’ll analyze time series data extracted from ΓRF, collected from the following sources:

  1. P25 trunked radio, police dispatching
  2. ADS-B aircraft telemetry
  3. Air traffic control, departure channel
  4. Tire pressure sensors

Hits were binned at 5 seconds, with zeros filling empty bins.  Data from 30 days was analyzed for all time series except TPMS (tire pressure sensors), for which 10 days of data was used.  Each time series appeared quite different than the others, visually.  The ratio of zeros in the P25 series was about half, whereas for the ATC departure channel series it was about 97%, for example.  The statistical properties of the different series varied widely as well.

Analysis techniques included:

  • Lomb-Scargle for identifying periodic elements of irregularly sampled series
  • Seasonal decomposition (using statsmodels.tsa.seasonal.seasonal_decompose)
  • Autocorrelation

Despite the superficial differences in the time series, their underlying characteristics were remarkably (though unsurprisingly) similar.

  • The series trend (as extracted via decomposition) hovered around some value, with excursions away from the mean now and then, possibly due to the effects of weather on radio propagation.
1
Seasonal Decomposition, ADS-B
  • Lomb-Scargle showed a strong daily periodicity for each series, as expected.  Each hit in each series is the result of human activity (e.g. flying a plane), and humans are periodic creatures
2.png
ATC Departure Lomb-Scargle plot
  • The histograms were somewhat asymmetrical, which I think suggests seasonality in the time series
3
Police dispatch activity histogram.  x-axis is hit number
  • For the residual, or “noise” (non-systematic) component of the series, the hit distribution appears more normal:
4
Police dispatch activity decomposition residual component, histogram
5
Kernel density estimation, police dispatch activity decomposition residual component
  • The correlograms, or autocorrelation plots, indicated daily periodicity as well
6
TPMS Correlogram.  The lag is bins.  Binning was at 5 seconds, so 17,280 bins = 1 day
7.png
P25 Police Dispatch Correlogram.  Is there a spike every 7 days as well (maybe at weekends)?
8
Decomposition of P25 police dispatch

Take a look at the decomposition above.  We’ve considered this time series additive, so in this model the sum of the trend, seasonal, and residual components should yield the initial series (labeled “observed” above).  What can we do with this data?

  • The seasonality can give us an idea of how busy the system we’re interested in will be at any given point in time
  • The trend might help us understand whether what we’re monitoring is changing over the long-term.  Are the police being called out more this month than they were this month last year?  Are more planes in the air now than a year ago?

Maybe the residual data can help us understand how often, and how far, we might deviate from (trend + seasonal)?  But if the trend is influenced by weather, as I’ve already speculated, then that would have to be taken into account as well…

Disclaimer: I’m not an expert in time series analysis and may have made some bad assumptions!  Please leave any helpful corrections / suggestions in the comments.

Many changes to the client and server

Over the past months there have been major changes.  Here are some.

Client

Only releases >= 1.0.4 work with the current server.

  • Users can manipulate p25 talkgroups on the command line (add, delete, and list, just like interesting freqs)
  • ZMQ socket is now dealer on client
  • Queue data on disconnect, and send it to the server after reconnect
  • Interesting freqs now belong to a user-defined group. You can specify freqs as “infrastructure” or “air”, for example.  Delete all your interesting freqs and readd them.

 

Server

A header shows various information about the station / user that’s logged in.

front_new

Graphs are now embedded Grafana graphs.  Much prettier, and much more interactive.

scanner_new

freqwatch_new

Some thoughts on long-term trends

Taking data from a couple of nodes and plotting it with Grafana results in graphs that are both pretty and interesting.  Below are a few 40-day plots, with some superficial analysis.  Holes in the plots are due to service disruptions.  (Right click->View Image to see a larger version)

long_term_1.JPG

Top graph is traffic from ATC frequencies for the nearby international airport, MCI.  Bottom is ADS-B.  Around the middle of the graph we see a streak of red begin, that lasts (with some interruption) until about July 10th.  There seems to have been some issues with the radio for departure west, or perhaps something near to my station was causing interference on this frequency.  I listened in while this was occurring, and indeed there were very frequent, short transmissions of static during this time.

The orange spikes that occur occasionally on the low end of the y-axis on the top graph are UHF, or military aircraft, frequencies.

long_term_2

Here is TPMS, or Tire Pressure Management System, hits near my station.  The high counts that occur once in awhile are due to idle vehicles.  These may be delivery vehicles like UPS, or people idling on the curb, waiting for something.  You can get a rough understanding of traffic ebb and flow down the street by my station from this graph.

long_term_3.JPG

Here are the ham radio stations.  Top graph is mostly repeaters, the bottom is storm / emergency networks.  The storm networks were busier in June, when Spring was winding down and weather was changing.  The repeaters are very periodic, and stratified, so that one can infer relatively how far from the station each is, in relation to the others.

ΓRF 1.0.0 client released

The 1.0.0 client includes improved p25 monitoring (using trunk-recorder), various bug fixes, and a huge stability improvement: clients now cache data if the connection to the server is down, and send it after the connection is reestablished.  The message times are now generated by the clients (in UTC), so keeping the client clocks accurate is now vital.

Below is a screenshot of the development server.  Today’s hits are in the top graph, and daily hits are in the bottom graph.  Clicking on a bar in the bottom graph takes you to the day in question.  (Right click->View Image to see a larger version)

new_int