Icicle is a modern query language over eventful data streams for batch and
streaming workloads.
To make writing streaming queries collaborative and simple, we need to ensure
that different parties can write small, contained expressions, and execute them
along with those of others. It's critical that queries can't interfere with
one another, take an inordinate amount of time, or crash, so new queries can
be added or removed without hesitation.
Icicle uses a novel, modal type system to achieve these goals. Along with a principled
Hindley Milner style type inference algorithm, it guarantees that all queries must
be computable over a single pass of the data. It's light weight, with no type
signatures required, but total, so all queries are guaranteed to execute and not
throw runtime exceptions.
In this talk I'll introduce Icicle, and demonstrate how modal types are used to
handle error conditions and computation staging between streams of data and their
results.
We will also demonstrate how its strong typing leads to powerful optimisation
passes, including unboxing of almost all run time values.
Finally we'll wrap up with some benchmarking for batch and Kafka workloads, along
with comparisons of how different tools compose.