Thanks all,
we've decided to break down the amount of information we process at a time. So instead of processing all of the IL data in one go, we do each sample as it's generated. Simpler and it works with the standard BAPI if we choose to use that instead.
Thanks for the input.