vxWorks5.5.2 and varying interrupt servicing times
I have a system that is designed around PowerPC (PPC604) VME Single Board Computer (SBC). This SBC is pretty much a VME repeater. The SBC receives VME data messages in a UDP datagram and then ships them out via a Tundra Universe II chip. Now these datagram have special type codes like 'write 32 bit' 'read 32 bit' 'write 16 bit' 'read 16 bit' and so on. There is one special code call 'write 36 bit and wait' where the SBC will write data in the datagram out the VME bus and then be blocked by a binary semaphore until another card on the VME bus issues a IRQ7. When the SBC receives the IRQ 7 it then releases the binary semaphore (via a interrupt service routine attached to the VME IRQ 7 interrupt) and the SBC continues to execute buffered up datagram. This all works well and good, however, I have a real tight timeline for these VME writes after an IRQ 7 is asserted. For the most part it takes the SBC somewhere around 8-12 microseconds (us) to service the VME IRQ 7 interrupt. I've been measuring this on the VME backplane by monitoring the IRQ 7 and IACK lines (delta time between VME IRQ 7 and IACK being deserted). The time stated above was measured from IRQ7 being asserted until IACK is deserted. However, from time to time the time I've been seeing servicing time of up to 160 us (IRQ 7 being asserted until IACK being deserted). This extremely long interrupt servicing time happens about .1% of the time (so 5 interrupts every 5000). Unfortunately my system spec does not support any VME interrupt servicing times greater than 14 us. I'm puzzled as to what could be causing this. Does anyone have any ideas as to why this might be happening and how I could troubleshoot this?
Additionally details regarding the system:
The SBC sits in slot 1 and is the VME controller. This SBC also has a fiber gigabit Ethernet card attached to it on a PMC slot. Everything else is a standard card. All watch dog timers are disabled to reduce the numbers of interrupts on the board.
There are three other cards that exist on the bus. Only one of these cards can issue any IRQs (only IRQ7 is issued). None of the cards initiate VME bus traffic (only the SBC can do that)