Lazy-Batched Stream Processing

[Denys Duchier]

Raphael Collet <raph at info.ucl.ac.be> writes:

>> For this reason, in my admitedly vague design, I propose to have, for each
>> transducer module Foo, both:
>> 	{Foo.encode      IN OUT}
>> and     {Foo.encodeBy BY IN OUT}
>> where Foo.encode just calls Foo.encodeBy with a reasonable default for the BY
>> argument
>> [...]
>
> Choosing the batch size is not easy.  Would it be possible to design an 
> abstraction with an adaptive batch size?  I know that guessing this kind 
> of magic number can be the source of much trouble.  All timeout values 
> turn out to be wrong one day.  Kevin has had a tough problem with Mozart 
> recently because of such a hardcoded magic number.

The batch size depends on the application as it needs to balance performing
possibly unnecessary work against computing values in advance; and also needs to
take into consideration the memory footprint of the precomputed batch of values.
This is why I propose that a transducer module Foo export Foo.encodeBy where the
batch size can be specified explicitly by the application writer.  Foo.encode is
merely a convenience function for what is envisioned as a typical application of
module Foo.

Are you suggesting that we also have an abstraction that allows the BY step to
be dynamically changed during stream processing?  It feels like overkill to me.

Cheers,

-- 
Dr. Denys Duchier - IRI & LIFL - CNRS, Lille, France
+33 (0)6 25 78 25 74    http://www.lifl.fr/~duchier/