FAQ =================================== Connecting to devices -------------------------------------------------------------------------------------------------------------- My device is unreachable, but the device logs say it's running fine? `````````````````````````````````````````````````````````````````````````````````````````````````````````````` The ``$HOSTNAME`` may have been incorrectly guessed by ``docker-compose/Makefile``, or you accidently set it to an incorrect value. If you have ``$HOSTNAME`` set in the shell running ``make``, try:: unset HOSTNAME make build make stop make start If this does not work, you need to set ``$HOSTNAME`` to something that resolves to your machine, both for external parties and for docker containers. See :ref:`corba`. I get "API_CorbaException: TRANSIENT CORBA system exception: TRANSIENT_NoUsableProfile" when trying to connect to a device? ```````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````` See the previous answer. Docker -------------------------------------------------------------------------------------------------------------- How do I prevent my containers from starting when I boot my computer? ```````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````` You have to explicitly stop a container to prevent it from restarting. Use:: cd docker-compose make stop or plain ``make stop`` to stop all of them. Windows -------------------------------------------------------------------------------------------------------------- How do I develop from Windows? `````````````````````````````````````````````````````````````````````````````````````````````````````````````` Our setup is Linux-based, so the easiest way to develop is by using WSL2, which lets you run a Linux distro under Windows. You'll need to: - Install WSL2. See f.e. https://www.omgubuntu.co.uk/how-to-install-wsl2-on-windows-10 - Install `Docker Desktop `_ - Enable the WSL2 backend in Docker Desktop - We also recommend to install `Windows Terminal `_ .. _x11_on_windows: How do I run X11 applications on Windows? `````````````````````````````````````````````````````````````````````````````````````````````````````````````` If you need an X11 server on Windows: - Install `VcXsrv `_ - Disable access control during its startup, - Use ``export DISPLAY=host.docker.internal:0`` in WSL. You should now be able to run X11 applications from WSL and Docker. Try running ``xterm`` or ``xeyes`` to test. SSTs/XSTs -------------------------------------------------------------------------------------------------------------- .. _statistics-debugging: I am not receiving any XSTs and/or SSTs from SDP! `````````````````````````````````````````````````````````````````````````````````````````````````````````````` Are you sure? - Packets are arriving if ``sst.nof_packets_received`` / ``xst.nof_packets_received`` is increasing, - Packets are sent by SDP if ``sst.FPGA_sst_offload_nof_packets_R`` / ``xst.FPGA_xst_offload_nof_packets_R`` is increasing. In general, the settings ought to be correct after the following: The ``sdp.set_defaults()`` command, followed by ``sst.set_defaults()`` / ``xst.set_defaults()``, should reset that device to its default settings, which should result in a working system again. Also, check the following settings: - ``sdpfirmware.TR_fpga_mask_RW[x] == True``, to make sure we're actually configuring the FPGAs, - ``sdp.FPGA_communication_error_R[x] == False``, to verify the FPGAs can be reached by SDP. - ``sdp.FPGA_processing_enabled_R[x] == True``, to verify that the FPGAs are processing, or the values and timestamps will be zero, - ``sdp.FPGA_signal_input_bsn_R`` is increasing, to verify that the FPGA processing is subject to the clock. The SDP is not sending SST/XST packets! `````````````````````````````````````````````````````````````````````````````````````````````````````````````` Packets are sent if ``sst.FPGA_sst_offload_nof_packets_R`` / ``xst.FPGA_xst_offload_nof_packets_R`` is increasing. If not, check these settings: - SSTs: - ``sst.FPGA_sst_offload_enable_RW[x] == True``, to verify that the FPGAs are actually emitting the SSTs, - XSTs: - ``xst.FPGA_xst_offload_enable_RW[x] == True``, to verify that the FPGAs are actually emitting the SSTs, - ``xst.FPGA_xst_processing_enable_RW[x] == True``, to verify that the FPGAs are actually producing the SSTs, Some SSTs/XSTs packets do arrive, but not all, and/or the matrices remain zero? `````````````````````````````````````````````````````````````````````````````````````````````````````````````` So ``sst.nof_packets_received`` / ``xst.nof_packets_received`` is increasing, telling you packets are arriving. But they're apparently dropped or contain zeroes. The ``sst`` and ``xst`` devices expose several packet counters to indicate where incoming packets were dropped before or during processing: - ``nof_invalid_packets_R`` increases if packets arrive with an invalid header, or of the wrong statistic for this device, - ``nof_packets_dropped_R`` increases if packets could not be processed because the processing queue is full, so the CPU cannot keep up with the flow, - ``nof_payload_errors_R`` increases if the packet was marked by the FPGA to have an invalid payload, which causes the device to discard the packet, If no packets are received at all, check whether they are sent to the correct address: - SSTs: - ``sst.FPGA_sst_offload_hdr_eth_destination_mac_R[x] == ``, or the FPGAs will not send it to your machine. Use f.e. ``ip addr`` on the host to find the MAC address of your interface, and verify that its MTU is 9000, - ``sst.FPGA_sst_offload_hdr_ip_destination_address_R[x] == ``, or the packets will be dropped by the network or the kernel of your machine, - ``sst.FPGA_sst_offload_hdr_udp_destination_port_R[x] == 5001``, or the packets will not be sent to a port that the SST device listens on. - XSTs: - ``xst.FPGA_xst_offload_hdr_eth_destination_mac_R[x] == ``, or the FPGAs will not send it to your machine. Use f.e. ``ip addr`` on the host to find the MAC address of your interface, and verify that its MTU is 9000, - ``xst.FPGA_xst_offload_hdr_ip_destination_address_R[x] == ``, or the packets will be dropped by the network or the kernel of your machine, - ``xst.FPGA_xst_offload_hdr_udp_destination_port_R[x] == 5002``, or the packets will not be sent to a port that the XST device listens on. If this fails, see the next question. I am still not receiving XSTs and/or SSTs, even though the settings appear correct! `````````````````````````````````````````````````````````````````````````````````````````````````````````````` Let's see where the packets get stuck. Let us assume your MTU=9000 network interface is called ``em2`` (see ``ip addr`` to check): - Check whether the data arrives on ``em2``. Run ``tcpdump -i em2 udp -nn -vvv -c 10`` to capture the first 10 packets. Verify: - The destination MAC must match that of ``em2``, - The destination IP must match that of ``em2``, - The destination port is correct (5001 for SST, 5002 for XST), - The source IP falls within the netmask of ``em2`` (unless ``net.ipv4.conf.em2.rp_filter=0`` is configured), - TTL >= 2, - If you see no data at all, the network will have swallowed it. Try to use a direct network connection, or a hub (which broadcasts all packets, unlike a switch), to see what is being emitted by the FPGAs. - Check whether the data reaches user space on the host: - Turn off the ``sst`` or ``xst`` device. This will not stop the FPGAs from sending. - Run ``nc -u -l -p 5001 -vv`` (or port 5002 for XSTs). You should see raw packets being printed. - If not, the Linux kernel is swallowing the packets, even before it can be sent to our docker container. - Check whether the data reaches kernel space in the container: - Enter the docker device by running ``docker exec -it device-sst bash``. - Run ``sudo bash`` to become root, - Run ``apt-get install -y tcpdump`` to install tcpdump, - Check whether packets arrive using ``tcpdump -i eth0 udp -c 10 -nn``, - If not, Linux is not routing the packets to the docker container. - Check whether the data reaches user space in the container: - Turn off the ``sst`` or ``xst`` device. This will not stop the FPGAs from sending. - Enter the docker device by running ``docker exec -it device-sst bash``. - Run ``sudo bash`` to become root, - Run ``apt-get install -y netcat`` to install netcat, - Check whether packets arrive using ``nc -u -l -p 5001 -vv`` (or port 5002 for XSTs), - If not, Linux is not routing the packets to the docker container correctly. - If still on error was found, you've likely hit a bug in our software. Inspecting SST/XST packets `````````````````````````````````````````````````````````````````````````````````````````````````````````````` The fields ``sst.last_packet_R`` and ``xst.last_packet_R`` contain a raw dump of the last received packet for that statistic. Parsing these packets is aided greatly by using our packet parser:: from tangostationcontrol.devices.sdp.statistics_packet import SSTPacket, XSTPacket # print the headers of the last received packets print(SSTPacket(bytes(sst.last_packet_R)).header()) print(XSTPacket(bytes(xst.last_packet_R)).header()) Other containers -------------------------------------------------------------------------------------------------------------- TBA