Lazy-Batched Stream Processing

[Raphael Collet]

Denys Duchier wrote:
> Raphael Collet <raph at info.ucl.ac.be> writes:
> 
>>	 if {Value.isNeeded Ys} then
>>	    %% preemption was too fast, double batch size
>>	    BatchSize:=2*@BatchSize
> 
> ooo... that's a cool trick! but couldn't you just omit the Thread.preempt?
> there's no reason to force a thread switch here (in fact, if Ys IsNeeded, we
> want to continue producing values).

With another thread to adapt the batch size, then?  The thread is woken 
up once per batch, and checks whether the tail is already needed...

fun {Map F Xs}
    BatchSize={NewCell 1}
    BatchPort={NewPort
	      thread
		 for X in $ do
		    if {IsNeeded X} then BatchSize:=@BatchSize*2 end
		 end
	      end}
    proc {Loop K Xs ?Ys}
       if K<@BatchSize then
	 Ys = case Xs of X|Xr then {F X}|{Loop K-1 Xr} else nil end
       else
	 {Send BatchPort Ys} {WaitNeeded Ys} {Loop 0 Xs Ys}
       end
    end
in
    thread {Loop 0 Xs} end
end

raph