I had the problem that the transmit FIFO size (= actual elements in FIFO) was...

I had the problem that the transmit FIFO size (= actual elements in FIFO) was slowly increasing over time, and was full after a few hours.

The reason was that the code hit the line "canerr("ERROR: No available mailbox\n");" in stm32_cansend, so can_xmit thinks it has sent the packet to the hardware, but actually has not. Therefore the transmit interrupt never happens which would call can_txdone, and so the size of the FIFO size does not decrease.

The reason why the code actually hit the mentioned line above, is because stm32can_txready uses a different (incomplete) condition than stm32can_send to determine if the mailbox can be used for sending, and thus can_xmit forwards the packet to stm32can_send. stm32can_txready considered mailboxes OK for sending if the mailbox was empty, but did not consider that mailboxes may not yet be used if the request completed bit is set - stm32can_txinterrupt has to process these mailboxes first.

Note that I have also modified stm32can_txinterrupt - I removed the if condition, because the CAN controller retries to send the packet until it succeeds. Also if the condition would not evaluate to true, can_txdone would not be called and the FIFO size would not decrease also.