FAQ
Connecting to devices
My device is unreachable, but the device logs say it’s running fine?
The $HOSTNAME
may have been incorrectly guessed by docker-compose/Makefile
, or you accidently set it to an incorrect value. If you have $HOSTNAME
set in the shell running make
, try:
unset HOSTNAME
make build
make stop
make start
If this does not work, you need to set $HOSTNAME
to something that resolves to your machine, both for external parties and for docker containers. See CORBA.
I get “API_CorbaException: TRANSIENT CORBA system exception: TRANSIENT_NoUsableProfile” when trying to connect to a device?
See the previous answer.
Docker
How do I prevent my containers from starting when I boot my computer?
You have to explicitly stop a container to prevent it from restarting. Use:
cd docker-compose
make stop <container>
or plain make stop
to stop all of them.
Windows
How do I develop from Windows?
Our setup is Linux-based, so the easiest way to develop is by using WSL2, which lets you run a Linux distro under Windows. You’ll need to:
Install WSL2. See f.e. https://www.omgubuntu.co.uk/how-to-install-wsl2-on-windows-10
Install Docker Desktop
Enable the WSL2 backend in Docker Desktop
We also recommend to install Windows Terminal
How do I run X11 applications on Windows?
If you need an X11 server on Windows:
Install VcXsrv
Disable access control during its startup,
Use
export DISPLAY=host.docker.internal:0
in WSL.
You should now be able to run X11 applications from WSL and Docker. Try running xterm
or xeyes
to test.
SSTs/XSTs
I am not receiving any XSTs and/or SSTs from SDP!
Are you sure?
Packets are arriving if
sst.nof_packets_received
/xst.nof_packets_received
is increasing,Packets are sent by SDP if
sst.FPGA_sst_offload_nof_packets_R
/xst.FPGA_xst_offload_nof_packets_R
is increasing.
In general, the settings ought to be correct after the following:
The sdp.set_defaults()
command, followed by sst.set_defaults()
/ xst.set_defaults()
, should reset that device to its default settings, which should result in a working system again. Also, check the following settings:
sdpfirmware.TR_fpga_mask_RW[x] == True
, to make sure we’re actually configuring the FPGAs,sdp.FPGA_communication_error_R[x] == False
, to verify the FPGAs can be reached by SDP.sdp.FPGA_processing_enabled_R[x] == True
, to verify that the FPGAs are processing, or the values and timestamps will be zero,sdp.FPGA_signal_input_bsn_R
is increasing, to verify that the FPGA processing is subject to the clock.
The SDP is not sending SST/XST packets!
Packets are sent if sst.FPGA_sst_offload_nof_packets_R
/ xst.FPGA_xst_offload_nof_packets_R
is increasing. If not, check these settings:
SSTs:
sst.FPGA_sst_offload_enable_RW[x] == True
, to verify that the FPGAs are actually emitting the SSTs,
XSTs:
xst.FPGA_xst_offload_enable_RW[x] == True
, to verify that the FPGAs are actually emitting the SSTs,xst.FPGA_xst_processing_enable_RW[x] == True
, to verify that the FPGAs are actually producing the SSTs,
Some SSTs/XSTs packets do arrive, but not all, and/or the matrices remain zero?
So sst.nof_packets_received
/ xst.nof_packets_received
is increasing, telling you packets are arriving. But they’re apparently dropped or contain zeroes.
The sst
and xst
devices expose several packet counters to indicate where incoming packets were dropped before or during processing:
nof_invalid_packets_R
increases if packets arrive with an invalid header, or of the wrong statistic for this device,nof_packets_dropped_R
increases if packets could not be processed because the processing queue is full, so the CPU cannot keep up with the flow,nof_payload_errors_R
increases if the packet was marked by the FPGA to have an invalid payload, which causes the device to discard the packet,
If no packets are received at all, check whether they are sent to the correct address:
SSTs:
sst.FPGA_sst_offload_hdr_eth_destination_mac_R[x] == <MAC of your machine's mtu=9000 interface>
, or the FPGAs will not send it to your machine. Use f.e.ip addr
on the host to find the MAC address of your interface, and verify that its MTU is 9000,sst.FPGA_sst_offload_hdr_ip_destination_address_R[x] == <IP of your machine's mtu=9000 interface>
, or the packets will be dropped by the network or the kernel of your machine,sst.FPGA_sst_offload_hdr_udp_destination_port_R[x] == 5001
, or the packets will not be sent to a port that the SST device listens on.
XSTs:
xst.FPGA_xst_offload_hdr_eth_destination_mac_R[x] == <MAC of your machine's mtu=9000 interface>
, or the FPGAs will not send it to your machine. Use f.e.ip addr
on the host to find the MAC address of your interface, and verify that its MTU is 9000,xst.FPGA_xst_offload_hdr_ip_destination_address_R[x] == <IP of your machine's mtu=9000 interface>
, or the packets will be dropped by the network or the kernel of your machine,xst.FPGA_xst_offload_hdr_udp_destination_port_R[x] == 5002
, or the packets will not be sent to a port that the XST device listens on.
If this fails, see the next question.
I am still not receiving XSTs and/or SSTs, even though the settings appear correct!
Let’s see where the packets get stuck. Let us assume your MTU=9000 network interface is called em2
(see ip addr
to check):
Check whether the data arrives on
em2
. Runtcpdump -i em2 udp -nn -vvv -c 10
to capture the first 10 packets. Verify:The destination MAC must match that of
em2
,The destination IP must match that of
em2
,The destination port is correct (5001 for SST, 5002 for XST),
The source IP falls within the netmask of
em2
(unlessnet.ipv4.conf.em2.rp_filter=0
is configured),TTL >= 2,
If you see no data at all, the network will have swallowed it. Try to use a direct network connection, or a hub (which broadcasts all packets, unlike a switch), to see what is being emitted by the FPGAs.
Check whether the data reaches user space on the host:
Turn off the
sst
orxst
device. This will not stop the FPGAs from sending.Run
nc -u -l -p 5001 -vv
(or port 5002 for XSTs). You should see raw packets being printed.If not, the Linux kernel is swallowing the packets, even before it can be sent to our docker container.
Check whether the data reaches kernel space in the container:
Enter the docker device by running
docker exec -it device-sst bash
.Run
sudo bash
to become root,Run
apt-get install -y tcpdump
to install tcpdump,Check whether packets arrive using
tcpdump -i eth0 udp -c 10 -nn
,If not, Linux is not routing the packets to the docker container.
Check whether the data reaches user space in the container:
Turn off the
sst
orxst
device. This will not stop the FPGAs from sending.Enter the docker device by running
docker exec -it device-sst bash
.Run
sudo bash
to become root,Run
apt-get install -y netcat
to install netcat,Check whether packets arrive using
nc -u -l -p 5001 -vv
(or port 5002 for XSTs),If not, Linux is not routing the packets to the docker container correctly.
If still on error was found, you’ve likely hit a bug in our software.
Inspecting SST/XST packets
The fields sst.last_packet_R
and xst.last_packet_R
contain a raw dump of the last received packet for that statistic. Parsing these packets is aided greatly by using our packet parser:
from tangostationcontrol.devices.sdp.statistics_packet import SSTPacket, XSTPacket
# print the headers of the last received packets
print(SSTPacket(bytes(sst.last_packet_R)).header())
print(XSTPacket(bytes(xst.last_packet_R)).header())
Other containers
TBA