jeffrey vaughan

Hi,

I'm writing a lexer for a toy language and want to count the number of
instances of particular token. I tried to use the following:

datatype token = FOO | ...
val fooCount = ref 0
%%
....
%%
{foo} => {inc fooCount; FOO}

Unfortunately this shares the reference fooCount among all parsers.
That is, given
val l = Mlex.makeLexer(...)
val l' = Mlex.makeLexer(...)
l() and l'() both increment a single foo counter. This is bad.

I thought that using %arg could help, but it's limited because the
consumer code (just a test harness for now), must manually feed correct
values into Mlex.lex.

Anyhow I think my options are
1) define
val init() = fooCount := 0
and make sure to use lexers like:
val l = makeLexer(...)
... tire of l ...
val () = init()
val l' = makeLexer(...)

2) augment the lexer result to:
type lexresult = context * oldresulttype
and manually thread lexer state
val l = makeLexer(...)
val l' = makeLexer(...)
val (c1,r1) = l(emptyContext)
val (c'1,r'1) = l'(emptyContext)
val (c2,r2)= l(c1)

3) create an imperative context
type lexresult = context ref * oldresulttype
and use wrap the lexers created by makeLexer
fun makeLexer'(a) =
let
val c = ref emptyContext
val l = makeLexer(a)
in
fn() => l(c)
end
val l = makeLexer'(...)
val l' = makeLexer'(...)
<interleave l() and l'() willy-nilly>

I imagine that technique 2 may be preferable. What is the best way to
do this?

Thanks!

--Jeff

stephen weeks

I don't see the problem with using %arg, which is exactly for
per-lexer stuff. You can easily build a wrapper around makeLexer that
hides the counter from the consumer.

matthias blume

Jeffrey Vaughan <vaughan2@seas.upenn.nospamkthx.edu> writes:

I don't understand why this would be "limited". Just pass the counter
(i.e., the ref cell itself) using %arg. You can then maken a fresh
ref cell and use that every time you instantiate the lexer using
makeLexer. Alternatively, you could wrap the counter in a function
and pass that in. I do this sort of thing on a regular basis. (My
%arg type is usually a record of functions.)

Matthias