@pipe_util usage

[galen1090]

I'm trying to do some experiments with pipetools:

@pipe_util
def myaggregate(function):
    return partial(functools.reduce, function)

ret = glob.iglob | foreach(readf) | flatten | foreach(len) | myaggregate(operator.add)
print(ret(path))

and it works. But I'm trying to replace flatten with more_itertools.flatten for speed reason:

ret = glob.iglob | foreach(readf) | (more_itertools.flatten) | foreach(len) | myaggregate(operator.add)
print(ret(path))

with more_itertools.flatten takes 6.49 seconds, with flatten 13.68 seconds.
I have tried with:

@pipe_util
def myflatten():
    return partial(more_itertools.flatten)

but it doesn't work, what am I missing?
Thanks

[0101]

Hi!

I probably need to describe this better in the documentation. The @pipe_util decorator's purpose is to wrap higher-order functions, which take another function (or something else that can be converted to a function, like X or a regex) as argument. Such as foreach or where. flatten doesn't take any function to specify what it's going to do, so you don't really need pipe_util for anything.

As you can see in the example you provided, just using it directly works fine! You can just import it as flatten or even myflatten and you're done.

The only other feature that @pipe_util gives you, is that you can start the pipe with it, without explicitly prefixing it with pipe | or X |. But you can easily get that like this:

myflatten = X | more_itertools.flatten

Also note that pipetools.flatten functionality is slightly different than the one in more_itertools - it flattens arbitrarily deep nesting and ignores strings and dictionaries.

Also, for your first example, you don't need to explicitly define myaggregate, instead you can just use reduce directly:

... | foreach(len) | (functools.reduce, operator.add)

Or in this particular case even better:

... | foreach(len) | sum

So the main point is that you can just pipe regular functions and you shouldn't really need to use @pipe_util (unless you want to pass X without ~ or have automatic partial application)