Transducers enable composable and memory efficient wrangling of very large or even infinite sets of data. With transducers, each item of the data is transformed by all operations in a single pass, as opposed to the data having to go through batch transformations one operation at a time.
The following example depicts the array manyNumbers going through two batch transformations, one with Array .filter and one with Array .map.
With transducers, you can express the above transformation as a single pass. Each item would be both filtered and mapped before the next item in the array. Batch transformations such as those with Array .map and .filter must create an intermediate array between each operation; transducers do not have this requirement and so do not incur a memory penalty.
The next example takes the above and converts it to use Rubico transducers.
Now the numbers are transformed in a single pass, avoiding the memory penalty 🎉. Transducers offer many benefits and expressive power, but can be difficult to pick up. You can build intuition for transducers by starting with reducers.
type Reducer = (accumulator any, value any)=>(nextAccumulator Promise|any)
type Transducer = Reducer=>Reducer
A reducer is a function that defines a relationship between an accumulator and an item in a transformation, and can be used in a reducing operation, such as with Rubico's reduce or vanilla JavaScript Array.prototype.reduce.
A transducer is a function that takes a reducer and returns another reducer. Transducers enable function chains with reducers - pass a reducer to a transducer to create a reducer with chained functionality. Imagine dominos falling over.
It's a good exercise to implement transducers on your own, however Rubico offers production-ready transducers via its Transducer module.
A transducer must be used with a reduce function such as Array .reduce. Rubico provides async-capable reduce functions as the transform and reduce operators.
The following example shows the function pipeline squaredOdds used as a transducer.
With Rubico's transducers, it is possible to transform asynchronous sources such as async generators.
The above is equivalent to the below with vanilla JavaScript for await:
Rubico's transducers are simple and useful for creating memory efficient data transformations. Get started with transducers at the docs.
Further reading:
