ogn-gateway-java

Introduction

OGN gateway is a standalone java application which subscribes to OGN AircraftBeacons and ReceiverBeacons and does the following:

  • decodes received OGN APRS messages and logs aircraft beacons into IGC files ( if enabled )
  • relays received and decoded becons to ogn-gateway plugins for further processing. (See list of available plugins below)

Building the gateway (from cmd-line)

In order to build the gateway (and its auxiliary libraries) you need to have in your build system installed and configured:

- JDK 1.8+
- Apache Maven (3.0.3+)

The easiest way to check if JDK is already installed (and correctly configured) in your system is to open a console (linux) or a cmd prompt (windows) and type:

javac -version

which - if JDK is correctly configured (i.e. installed + JAVA_HOME env. variable set + JAVA_HOME/bin in the Path ) should print out something like:

javac 1.7.0_13

Unless you see such an output your JDK is not correctly configured. Please install and configure JDK before continuing.

Similarly, for maven, type:

mvn -version

If maven is correctly configured (i.e installed + M2_HOME env. variable set + M2_HOME/bin in the Path) it should result with something like:

Apache Maven 3.1.0 (893ca28a1da9d5f51ac03827af98bb730128f9f2; 2013-06-28 04:15:2+0200)
Maven home: C:\maven\apache-maven-3.0.3
Java version: 1.7.0_13, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1.7.0_13\jre

Unless you see such an output, Apache Maven is not correctly configured. Please refer to downloading and installing Maven before continuing.


OK, so JDK and Maven are there, time to build the gateway. Since ogn-gatway-java depends on ogn-commons-java and ogn-client-java you need to build all three modules in the following order:

  1. ogn-commons-java
  2. ogn-client-java
  3. ogn-gateway-java

All you need to do is:
1. Checkout each of the modules from OGN github
2. For modules 1) and 2) : enter their root folder and type:

mvn install

This will build, test and install artifacts 1) and 2) into maven local repository.
3. Enter root folder of ogn-gateway-java module and type:
mvn package

This will build, test and produce a deployable tar.gz with all the gateway needs inside the target folder. Voila! The gateway is ready for deployment!

Deploying the gateway (on a linux system)

1. Use SCP/FTP (or whichever tool you like) to copy over the ong-gateway's tar.gz to the host where the gateway is supposed to run.
2. untar the tar.gz (tar -zxf) to the destination location ( e.g. ~/ogn/ogn-gateway )
3. enter the ogn-gateway/conf subfolder and create two files inside: jmxremote.access and jmxremote.passwd. Make sure they have limited access rights! ( chmod 600 ) (won't work otherwise!)

Explanation: these two files contain usernames and credentials for remote access via JMX. Therefore they cannot be part of the deployable tar.gz and need to be created manually (or -even better - kept locally somewhere in the local filesystem and copied over to the ogn-gateway/conf right after untarring the tar.gz )
The content of the files should be following:
jmxremote.access :

ognjmx readonly
ognadm readwrite

jmxremote.passwd :

ognjmx the-password-you-want-to-set-for-this-user
ognadm the-password-you-want-to-set-for-this-user

Remember to "chmod 600" the two files ;-)

Launching the gateway

Prerequirements:

  • the gateway needs JRE 1.7+ to be installed on the machine where it is supposed to run
  • export OGN_HOME env. variable setting the location of all OGN modules. Preferably put the export it into your .bashrc Note: Unless set the startup script will take the default value i.e. ~/ogn

Start the gateway using the ogn-gateway-sh script you will find inside the gateway's root folder. There are following options:

[ognuser@ognhost ogn-gateway]$ ./ogn-gateway.sh
*****************************************************************************
 usage:
 ./ogn-gateway.sh start|stop|restart|status
*****************************************************************************

The structure of the gateway once deployed

The gateway's tar.gz once unpacked will show up the following structure:

[ognuser@ognhost ogn-gateway]$ ls -1
conf
lib
log
ogn-gateway.sh
plugins
tmp
version.txt
  • conf

The configuration folder. It contains the gateway.properties configuration file , logging configuration file (log4j.xml ) + JMX remote access credential files (jmxremote.access, jmxremote.passwd)

  • lib

Contains all libraries required by the gateway to work (ogn stuff + 3rd parties)

  • log

The folder where the log files are stored

  • plugins

The container folder for the OGN gateway's forwarding plugins (e.g plugin for FR24, plugin for live.glidernet.org,…)

  • tmp

Tmp folder, e.g. for storring the pid lock file

  • version.txt

Contains the gateway's version number as defined in the project's pom.xml, e.g: 1.0.0-SNAPSHOT

Log files description

Once started, the OGN gateway will be producing a number of logs. I shortly describe the default set below. Please note that the number of logs and their thresholds can easily be reconfigured (also at runtime) by manipulating the log4j.xml configuration file which can be found in the conf subfolder.

log folder contains the following log files:

  • ogn-gateway.log

Gateway-specific runtime information. Normally you should not see much in that log (since its default treshold is INFO). In case of normal operation there will be information in which mode the gateway has been started (simulation or not), which plugins have been loaded etc. By default this log uses rolling appender (it creates log files of max 10Mb and max. 2 backups) It can of course be tuned (conf/log4j.xml)

example:

2015-01-13 17:33:47.950 INFO  org.ogn.gateway.OgnGateway  - starting OGN gateway process
2015-01-13 17:33:49.349 INFO  org.ogn.commons.igc.IgcLogger  - creating igc logger [log-folder: log/igc, mode: ASYNC]
2015-01-13 17:33:49.418 INFO  org.ogn.gateway.OgnGateway  - simulation mode: true
2015-01-13 17:37:19.287 INFO  org.ogn.gateway.PluginsManager  - registering plug-in org.ogn.gateway.live.LiveGlidernetForwarder live.glidernet.org forwarder 1.0.0 relays OGN aircraft beacons to live.glidernet.org
2015-01-13 17:37:49.287 INFO  org.ogn.gateway.PluginsManager  - registering plug-in org.ogn.gateway.fr24.FR24Forwarder FR24 forwarder 0.0.1 relays OGN aircraft beacons to FlightRadar24 system
  • ogn-client.log

All the output produced by the ogn-client-java module ends up in this file. By default not much noise is expected (the default logging threshold for that logger is INFO, yet like in case of any other loggers it can be changed at any time if needed). By default it uses rolling appender with max. 5Mb file size and 2 backups (see conf/log4j.xml for details)

  • forwarded-beacons.log

This logger logs ALL the packets which are in theory supposed to be relayed to the ogn gateway plugins. In theory - because if a gateway works in simulation mode an entry will still be logged yet a beacon is in fact not relayed.
This logger uses daily-rolling appender - i.e. one file per day is created.

columns: timestamp, stealth-flag, ogn-is-tracked-flag, address-type, num-of-error-bits, plugin-name, plugin-version, aprs-raw-packet
example:

2015-03-30 18:02:15.951 false false FLARM 0 FR24 forwarder 1.0.0 FLRDD88EB>APRS,qAS,LOWG:/160213h4659.79N/01525.92E'/A=001115 id06DD88EB +020fpm +0.0rot 11.8dB 0e -3.8kHz
2015-03-30 18:02:31.001 false false FLARM 0 live.glidernet.org forwarder 1.0.0 FLRDD88EB>APRS,qAS,LOWG:/160228h4659.79N/01525.92E'/A=001112 id06DD88EB +020fpm +0.0rot 9.2dB 0e -4.0kHz
2015-03-30 18:02:33.029 false false ICAO 0 FR24 forwarder 1.0.0 ICA4B521A>APRS,qAS,LFNC:/160229h4439.73N/00641.36E'037/070/A=008830 id054B521A +119fpm +0.1rot 11.0dB 0e -0.3kHz gps3x4
2015-03-30 18:02:33.029 false false ICAO 0 live.glidernet.org forwarder 1.0.0 ICA4B521A>APRS,qAS,LFNC:/160229h4439.73N/00641.36E'037/070/A=008830 id054B521A +119fpm +0.1rot 11.0dB 0e -0.3kHz gps3x4

PS. I gave it a name proxy-forwarded.log because it is the gateway's proxy class's logger. Shall we rename it however to simply forwarded.log? It is just a question of modifying 1 line in log4j.xml

  • discarded-beacons.log

This logger logs ALL the packets which are discarded by the gateway (i.e. NOT relayed to the forwarder plugins) due to several reasons (e.g. the number of error bits exceeds the max. accepted etc..)
This logger uses daily-rolling appender - i.e. one file per day is created.

columns: timestamp, stealth-flag, is-tracked-flag, address-type, num-of-error-bits, raw-aprs-packet
example:

2015-03-30 17:14:07.916 false false ICAO 6 WK518>APRS,qAS,UKKIR:/151357h5310.15N\00011.83W^258/054/A=000623 id2143C38E +535fpm -1.0rot 4.8dB 6e -5.4kHz gps2x2
2015-03-30 17:14:50.961 false false ICAO 11 WK518>APRS,qAS,UKKIR:/151445h5309.90N\00013.11W^253/060/A=000574 id2143C38E -237fpm +0.1rot 4.5dB 11e -5.5kHz gps1x2
2015-03-30 17:20:29.825 false false ICAO 9 WK518>APRS,qAS,UKKIR:/152015h5309.63N\00012.12W^165/093/A=000531 id2143C38E -514fpm -0.2rot 5.0dB 9e -6.0kHz gps2x2
2015-03-30 17:32:25.867 false false ICAO 7 TE311>APRS,qAS,UKLSW:/153222h5303.49N\00010.43W^052/190/A=002024 id2143C396 -2533fpm -0.6rot 5.8dB 7e -7.7kHz gps2x1 hearC78E

PS. shall we rename it to simply discarded.log?

  • out.log

All stdout/err not caught by any other loggers would end up in this file. Useful to look into for debugging in case of problems.

  • procstate.log

Logs an entry each time ogn-gateway process is started or stopped.

columns: timestamp, host-name, operation, status
example:

2015-01-13 17:33:46.924 ognhost.org       START   OK

log/aprs subfolder:

  • aircraft-raw.log

No need to describe I hope. It uses a daily-rolling appender.

columns: timestamp, raw-aprs-packet
example:

2015-01-14 17:37:17.734 FLRDDDF2F>APRS,qAS,LFNF:/162201h4344.23N/00548.13E'180/065/A=001453 id06DDDF2F -989fpm +0.3rot 14.8dB 0e +17.5kHz gps3x4
2015-01-14 17:37:18.837 FLRDD88EB>APRS,qAS,LOWG:/163715h4659.79N/01525.92E'/A=001125 id06DD88EB +000fpm +0.0rot 8.0dB 0e -6.3kHz
  • aircraft-decoded.log

No need to describe I hope. It uses a daily-rolling appender.

example:

2015-01-14 17:36:36.717 D-6346 {
  "id" : "D-6346",
  "timestamp" : 1421253393000,
  "lat" : 52.91683333333334,
  "lon" : 8.823333333333332,
  "alt" : 36.9,
  "rawPacket" : "D-6346>APRS,qAS,Sykeihgr:/163633h5255.01N/00849.40E'/A=000121 id06DDACFC -019fpm +0.0rot 32.8dB 0e -3.2kHz gps2x3",
  "receiverName" : "Sykeihgr",
  "track" : 0,
  "groundSpeed" : 0.0,
  "address" : "DDACFC",
  "addressType" : "FLARM",
  "aircraftType" : "GLIDER",
  "stealth" : false,
  "climbRate" : -0.1,
  "turnRate" : 0.0,
  "signalStrength" : 32.8,
  "frequencyOffset" : -3.2,
  "gpsStatus" : "2x3",
  "errorCount" : 0,
  "heardAircraftIds" : [ ]
} {
  "regNumber" : "D-6346",
  "cn" : "46",
  "owner" : "SFV Hoya",
  "homeBase" : "HOYA",
  "model" : "Ka-8",
  "freq" : "122.475"
}
  • receivers-raw.log

No need to describe I hope. It uses a daily-rolling appender.

columns: timestamp, raw-aprs-packet
example:

2015-01-14 17:38:49.107 NewburyS>APRS,TCPIP*,qAC,GLIDERN1:/163848h5120.77NI00120.35W&/A=000518 CPU:3.7 RAM:166.8/458.7MB NTP:0.9ms/-19.4ppm +47.6C RF:+74-6.5ppm/-0.8dB
2015-01-14 17:38:49.179 Bueron>APRS,TCPIP*,qAC,GLIDERN2:/163848h4712.83NI00805.89E&/A=001830 CPU:0.9 RAM:121.3/458.7MB NTP:2.1ms/-40.1ppm +57.3C RF:+29+0.6ppm/+4.9dB
  • receivers-decoded.log

No need to describe I hope. It uses a daily-rolling appender.

2015-01-14 17:39:35.294 Sylwek {
  "id" : "Sylwek",
  "timestamp" : 1421253574000,
  "lat" : 50.0325,
  "lon" : 19.9425,
  "alt" : 229.8,
  "rawPacket" : "Sylwek>APRS,TCPIP*,qAC,GLIDERN2:/163934h5001.95NI01956.55E&/A=000754 v0.1.4 CPU:0.5 RAM:767.7/1016.9MB NTP:0.3ms/+2.5ppm RF:+62-1.3ppm/+0.96dB",
  "version" : "0.1.4",
  "cpuLoad" : 0.5,
  "cpuTemp" : "NaN",
  "totalRam" : 1016.9,
  "freeRam" : 767.7,
  "ntpError" : 0.3,
  "rtCrystalCorrection" : 2.5,
  "recCrystalCorrection" : 62,
  "recCrystalCorrectionFine" : -1.3,
  "recInputNoise" : 0.96,
  "serverName" : "GLIDERN2",
  "recAbsCorrection" : 60.7,
  "numericVersion" : 1004
}
  • unmached.log

If an OGN aprs packet contains fields which have unexpected format such packets will be logged into this log. This logger uses daily-rolling appender.

example (MotServlx has probably DVB-T dongle issue - and does not send RF information in the expected format - cristal correlation is missing):

2015-01-14 16:52:40.057 AprsReceiverBeacon - aprs-sentence:[MotServlx>APRS,TCPIP*,qAC,GLIDERN1:/155239h4535.55NI00551.73E&/A=001007 v0.1.4 CPU:0.0 RAM:104.7/867.3MB NTP:0.5ms/-2.6ppm RF:+60+0.0ppm] unmatched aprs parms: [RF:+60+0.0ppm]
2015-01-14 16:57:40.153 AprsReceiverBeacon - aprs-sentence:[MotServlx>APRS,TCPIP*,qAC,GLIDERN1:/155739h4535.55NI00551.73E&/A=001007 v0.1.4 CPU:0.0 RAM:104.7/867.3MB NTP:0.5ms/-2.6ppm RF:+60+0.0ppm] unmatched aprs parms: [RF:+60+0.0ppm]
2015-01-14 17:02:40.343 AprsReceiverBeacon - aprs-sentence:[MotServlx>APRS,TCPIP*,qAC,GLIDERN1:/160239h4535.55NI00551.73E&/A=001007 v0.1.4 CPU:0.0 RAM:104.2/867.3MB NTP:0.5ms/-2.6ppm RF:+60+0.0ppm] unmatched aprs parms: [RF:+60+0.0ppm]
2015-01-14 17:07:40.200 AprsReceiverBeacon - aprs-sentence:[MotServlx>APRS,TCPIP*,qAC,GLIDERN1:/160739h4535.55NI00551.73E&/A=001007 v0.1.4 CPU:0.0 RAM:104.2/867.3MB NTP:0.5ms/-2.6ppm RF:+60+0.0ppm] unmatched aprs parms: [RF:+60+0.0ppm]
2015-01-14 17:12:40.124 AprsReceiverBeacon - aprs-sentence:[MotServlx>APRS,TCPIP*,qAC,GLIDERN1:/161239h4535.55NI00551.73E&/A=001007 v0.1.4 CPU:0.0 RAM:104.0/867.3MB NTP:0.5ms/-2.6ppm RF:+60+0.0ppm] unmatched aprs parms: [RF:+60+0.0ppm]

log/igc subfolder:
OGN gateway uses IGC logger to log IGC files.
Note: The default location of IGC files can be changed by overwriting a ogn.gateway.igc.folder property ( in the gateway.properties file or setting corresponding JVM env. variable: -Dogn.gateway.igc.folder=…)

example:

[ognuser@ognhost 2015-01-14]$ pwd
/home/ognuser/ogn/ogn-gateway/log/igc/2015-01-14
[ognuser@ognhost 2015-01-14]$ ls -1
2015-01-14_D-6346.IGC
2015-01-14_D-KDSD.IGC
2015-01-14_D-KTCJ.IGC
2015-01-14_D-KTIO.IGC
2015-01-14_F-BNSQ.IGC
2015-01-14_F-CEAV.IGC
2015-01-14_F-CECN.IGC
...
...

The logging thresholds of ALL logs can be changed at runtime by modifying the log4j.xml configuration (which can be found in the /conf subfolder). The file is monitored and the new configuration will be taken into account usually in 30sec (of course that can also be configured as a property in the gateway.properties or passed as a JVM env. variable)

Introspecting the gateway process (JMX)

JMX is very handy to introspect the running java process. JMX beans expose a variety of metrics which you can monitor at real time and quickly spot problems (e.g. memory leaks, growing thread numbers, deadlocks etc..). There is a set of beans exposed by default by every java process but in addition ogn-gateway-java exports a set of its own "specific" beans, showing a set of interesting metrics.

Once the ogn-gateway-java process is up and running on its deployment machine, launch the jconsole from cmd line (note: jconsole is a part of JDK, so again it is assumed JDK is correctly configured on the machine from which you are about to connect):

jconsole

and in order to connect remotely to the ogn-gateway-java process, type the host, port, user-name (you can log in either as ognjmx -for read only access or as ognadm) and password as shown on the example below:

ogn-jconsole1.PNG

For more information on jconsole's features please refer to this manual

Configuration parameters

There is a set of configuration parameters which can be set in order to tune OGN gateway. These parameters can be specified in the ogn.properties file or they can be passed to the java process directly as JVM env variables (-Dvariabe=value). In both ways the effect is the same.
Unless overwritten, the gateway will use default values for all its parameters. To see the parameters names as well as their default values, please refer to the gateway's Configuration class definition (parameter names and their default values can be seen annotated with @Value).

Please also note that the whole Configuration class is a JMX bean and all its attributes can be introspected through JMX

The OGN gateway plugins

Implementing a new ogn plugin is generally quite easy (well..it depends of course what the plugin is supposed to do;-) ) and in practice one does not need to be aware at all of the whole ogn-gateway internals!. OGN gateway plugin is a standalone project which should produce a jar archive just complying with the two following rules:

  1. at least one of its classes must implement the OgnAircraftBeaconForwarder interface
  2. the JAR must contain META-INF/services/org.ogn.commons.beacon.forwarder.OgnAircraftBeaconForwarder text file and it should contain a fully qualified names of the class(es) implementing the mentioned interface, e.g:
# this class implements OGN forwarder for FlighRadar24
org.ogn.gateway.fr24.FR24Forwarder

to be completed For now, please try to do all the above. You should finish by having a running gateway with NO plugins installed - i.e beacons WILL NOT be delivered to any of the systems (FR24, live.glidernet…) still the gateway should be starting up and running correctly, logging all the stuff etc..

Meanwhile I'll finish the documentation and check-in the plugins to some private repo:)

Building and deploying a plugin

to be completed..

Available plugins

plugin name current version description requirements
ogn-gateway-live 1.0.0 a plugin forwarding AircraftBeacons to live.glidernet.org through HTTP POST live.glidernet.org requires authentication. Therefore the plugin needs live.glidernet.org.passwd property to be set (it can be put into ogn-gateway's property file (conf/gateway.properties) (prefered method!) or it can be set as a JVM env. variable at the gateway's startup (-Dlive.glidernet.org.passwd=…) Unless set a default value ( anonymous ) will be used
ogn-gateway-fr24 1.0.0 a plugin forwarding AircraftBeacons to FlightRadar24
ogn-gateway-sbb 1.0.0 under development a plugin starting an SBB server which forwards AircraftBeacons to connected clients according to SBS/Basestation TCP/30003 protocol
ogn-gateway-stats 1.0.0 This plugin is responsible for the analysis, computing and storing of selected information extracted from aircraft and receiver beacons in a SQLite database. These information include: number of aircraft beacons received by each individual OGN station (daily), max. range captured by each of the receivers (daily), max. altitudes captured by each of the receivers (daily), etc..
ogn-gateway-amqp 1.0.0 This plugin relays OGN beacons to AMQP (RabbitMQ) broker

Possible setups

Dependently on the use-case ogn-gateway can be set-up differently, as presented below.

Standalone APRS solution

ogn-gateway-pure-aprs-standalone.png

This is actually what we have in production at the moment. With the new gateway we can keep of course the same schema - this is the simplest solution which in first go can replace the currently operational gateway at any time.
This scenario, however has some well known weaknesses, mainly it is that OGN receivers send pure APRS messages to the OGN APRS tier and any client can subscribe to that data (e.g. the beacons with stealth-on), bypassing the gateway's filtering mechanism. Therefore in that scenario the receivers should be pre-filtering the date not sending the decoded packets with stealth flag on.

Clustered APRS solution

ogn-gateway-pure-aprs-clustered.png

A clustered version of the purly APRS-based solution. Both (or more) instances subscribe to the same APRS source and process data. Since both instances may receive same packets from the APRS servers, duplicates are filtered out.

Clustered own-protocol single-tier solution

ogn-gateway-1layer-mixed-clustered.png

In this solution we have a cluster of ogn-gateway's to which OGN receives are connected (round robin to distribute receiver connections +/- equally). The instances in the cluster can run on a single machine or be distributed on 2 (or N different machines). They no longer use pure APRS, but our own simple (binary) protocol (ogn-lib) delivering beacons to the gateways which in turn forward the preprocessed and filtered messages to the OGN gateway forwarder-plugins which in turn send them to our client systems (live.glidernet, fr24,…) as well as to the OGN APRS infrastructure (through the aprs plugin).

The main advantages are:
- OGN APRS clients can only subscribe to the data which is already pre-filtered.
- ogn-gateways are redundant. Switching off one does not affect the whole system.

Clustered own-protocol + APRS (double-tier) solution

ogn-gateway-2layer-mixed-clustered.png

In this solution we have a cluster of ogn-gateway's to which OGN receives are connected (round robin to distribute receiver connections +/- equally). The instances in the cluster can run on a single machine or be distributed on 2 (or N different machines).
They no longer use pure APRS, but our own simple (binary) protocol (ogn-lib) delivering beacons to the gateways which in turn forward the preprocessed and filtered messages to the OGN aprs infrastructure (through the ogn-gatewa's aprs plugin).

The remaining part (the upper tier) is a standalone ogn-gateway solution (as described above).

The main advantages are:
- OGN APRS clients can only subscribe to the data which is already pre-filtered.
- ogn-gateways are redundant. Switching off one does not affect the system.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License