Developer information

This chapter describes key areas useful for developers.

Docker compose

The docker setup is managed using make in the docker-compose directory. Key commands are:

• make status to check which containers are running,

• make build <container> to rebuild the image for the container,

• make build-nocache <container> to rebuild the image for the container from scratch,

• make restart <container> to restart a specific container, for example to effectuate a code change.

• make clean to remove all images and containers, and the tangodb volume. To do a deeper clean, we need to remove all volumes and rebuild all containers from scratch:

  make clean
  docker volume prune
  docker build-nocache
  

Since the Python code is taken from the host when the container starts, restarting is enough to use the code you have in your local git repo. Rebuilding is unnecessary.

Docker networking

The Docker containers started use a virtual network to communicate among each other. This means that:

• Containers address each other by a host name equal to the container name (f.e. elk for the elk stack, and databaseds for the TANGO_HOST),

• localhost cannot be used within the containers to access ports of other containers.

• host.docker.internal resolves to the actual host running the containers,

• All ports used by external parties need to be exposed explicitly in the docker-compose files. The container must open the same port as is thus exposed, or the port will not be reachable.

The networks are defined in docker-compose/networks.yml:

#
# Docker compose file that describes our docker networks.
#
version: '3.5'

networks:

  # Control network, with MTU=1500. Our default network.
  control:
    name: ${NETWORK_MODE}
    # The networks are created and destroyed in the Makefile.
    external: true

  # Data network, with MTU=9000, to receive the Jumbo frames from SDP
  data:
    # This name needs to be alphabetically before the control network.
    #
    # As we add the data-receiving devices to both control and data
    # network, we need to make sure the UDP data are forwarded to
    # the data network, not the control network. The way Docker decides
    # which network to actually expose ports on is by chosing the
    # first one in alphabetical order.
    name: 9000-${NETWORK_MODE}
    external: true

The $NETWORK_MODE defaults to tangonet in the docker-compose/Makefile.

CORBA

Tango devices use CORBA, which require all servers to be able to reach each other directly. Each CORBA device opens a port and advertises its address to the CORBA broker. The broker then forwards this address to any interested clients. A device within a docker container cannot know under which name it can be reached, however, and any port opened needs to be exposed explicitly in the docker-compose file for the device. To solve all this, we assign a unique port to each device, and explictly tell CORBA to use that port, and what the hostname is under which others can reach it. Each device thus has these lines in their compose file:

ports:
  - "5701:5701" # unique port for this DS
entrypoint:
  # configure CORBA to _listen_ on 0:port, but tell others we're _reachable_ through ${HOSTNAME}:port, since CORBA
  # can't know about our Docker port forwarding
  - python3 -u /opt/lofar/tango/devices/devices/sdp/sdp.py STAT -v -ORBendPoint giop:tcp:0:5701 -ORBendPointPublish giop:tcp:${HOSTNAME}:5701

Specifying the wrong $HOSTNAME or port can make your device unreachable, even if it is running. Note that $HOSTNAME is advertised as is, that is, it is resolved to an IP address by any client that wants to connect. This means the $HOSTNAME needs to be correct for both the other containers, and external clients.

The docker-compose/Makefile tries to set a good default for $HOSTNAME, but you can override it by exporting the environment variable yourself (and run make restart <container> to effectuate the change).

For more information, see:

• https://huihoo.org/ace_tao/ACE-5.2+TAO-1.2/TAO/docs/ORBEndpoint.html

• http://omniorb.sourceforge.net/omni42/omniNames.html

• https://sourceforge.net/p/omniorb/svn/HEAD/tree/trunk/omniORB/src/lib/omniORB/orbcore/tcp/tcpEndpoint.cc

Logging

The ELK stack collects the logs from the containers, as well as any external processes that send theirs. It is the Logstash part of ELK that is responsible for this. The following interfaces are available for this purpose:

Interface	Port	Note
Syslog	1514/udp	Recommended over TCP, as the ELK stack might be down.
Syslog	1514/tcp
JSON	5959/tcp	From python, recommended is the LogStash Async module.
Beats	5044/tcp	Use FileBeat to watch logs locally, and forward them to ELK.

We recommend making sure the contents of your log lines are parsed correctly, especially if logs are routed to the Syslog input. These configurations are stored in docker-compose/elk/logstash/conf.d. An example:

filter {
  if [program] == "tango-rest" {
    grok {
      match => {
        "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:level} %{GREEDYDATA:message}"
      }
      "overwrite" => [ "timestamp", "level", "message" ]
    }
    date {
      match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSS" ]
      timezone => "UTC"
    }
  }
}

Log from Python

The common.lofar_logging module provides an easy way to log to the ELK stack from a Python Tango device.

Log from Docker

Not all Docker containers run our Python programs, and can forward the logs themselves. For those, we use the syslog log driver in Docker. Extend the docker compose files with:

    logging:
      driver: syslog
      options:
        syslog-address: udp://${LOG_HOSTNAME}:1514
        syslog-format: rfc3164
        tag: "{{.Name}}"

Logs forwarded in this way are provided with the container name, their timestamp, and a log level guessed by Docker. It is thus wise to parse the message content further in Logstash (see above).