Generator Functions as Data Sources?

[akloster]

In Pandas what I would often do to generate a DataFrame is to generate a list of Dictionaries, and create a DataFrame from there. Is that also possible with Odo? And would it be possible without keeping the whole list in memory?

The source of the records are normally not things like CSV or JSON, but either some computations, file exploration, etc. and I think it would be overkill or inconvenient to write a specialized backend.

[mrocklin]

There is an Iterator node that works well in odo. The one problem here is
that we can't investigate the data types/columns without destroying some of
the data (with how odo currently works) so you would need to specify a
datashape explicitly.

On Tue, Jun 16, 2015 at 10:08 PM, Andreas Klostermann <
notifications@github.com> wrote:

In Pandas what I would often do to generate a DataFrame is to generate a
list of Dictionaries, and create a DataFrame from there. Is that also
possible with Odo? And would it be possible without keeping the whole list
in memory?

The source of the records are normally not things like CSV or JSON, but
either some computations, file exploration, etc. and I think it would be
overkill or inconvenient to write a specialized backend.

—
Reply to this email directly or view it on GitHub
#230.

[llllllllll]

Why would you need to destroy some of the data? In part of my application where I was using odo with generators, I wrapped my stream in this class:

class PeekableIterator(object):
    """
    An iterator that can peek at the next value without consuming it.

    Parameters
    ----------
    stream : iterator
        The underlying iterator.
    """
    def __init__(self, stream):
        self._stream = stream
        self._peeked = LifoQueue()

    def __next__(self):
        try:
            return self._peeked.get_nowait()
        except Empty:
            return next(self._stream)

    def __iter__(self):
        return self

    def peek(self):
        """The next element from the iterator without consuming it.

        Returns
        -------
        top : any
            The next element in the iterator.
        """
        top = next(self)
        self._peeked.put_nowait(top)
        return top

and then used discover on the first element to set the dshape. There is an implied contract the the dshape of values in stream are all the same.