Lazy-Batched Stream Processing

[Denys Duchier]

Raphael Collet <raph at info.ucl.ac.be> writes:

> Denys Duchier wrote:
>> [...]
>> fun lazy {OldLazyMap IN F}
>> proc {NewLazyMap IN F OUT}
>> [...]
>
> FYI, I have done a few comparisons between the two latter versions when 
> I introduced the WaitNeeded primitive.  There is an improvement, but the 
> speedup factor is less than 2.

A factor of 2 is considerable.

>> proc {LazyBatchedMap BY IN F OUT}
>> [...]
>
> Have you done some benchmark comparisons between the two latter 
> versions?  I am curious to know the actual overhead of this double 
> context switch...

This of course varies with the batch size.  I have used the following test,
where L1 is the infinite list "L1 = unit|L1":

   for _ in 1..1000000 _ in {LazyBatchedMap BY L1 Identity} do skip end

if BY=1 the speedup is of course 1, for BY=1000 it is about 3 (on my laptop),
and of 4.25 over the OldLazyMap.  On my office machine, the numbers are
resp. 3.54 and 5.15.

Cheers,

PS: ahem... there were some typos in the code I posted - sorry about that.

-- 
Dr. Denys Duchier - IRI & LIFL - CNRS, Lille, France
+33 (0)6 25 78 25 74    http://www.lifl.fr/~duchier/