Is it possible to load data from Source to Dest between servers?

[shut000]

Is it possible to load data from Source to Dest between servers? Like what ETL usually does. Thanks!
And I don't know where to add questions, just put it here...

[jeffreyiacono]

hmm, you'd probably need to expand the ETLs class's ability to hold more than one connection (source_connection / destination_connection) so you could pull from one server and send to another. Let me know if that makes sense and answers your question.

[shut000]

thank you! I tried another one (activewarehouse-etl), it's more complex and support multiple servers. I'm new in Ruby and trying to break it down...

[Emerson]

interesyed in this functionality as well. Wondering what the API might look like.

[jeffreyiacono]

@Emerson @shut000 was just giving this a look over and have an idea on what could work here: right now we have ETL.connection that's used to speak with a single db via #query. This works fine when doing data processing within the same server / db, but totally see your use case. My thought: add a ETL.read_connection and ETL.write_connectionthat you can set and then interact with through two new methods#readand#writewhich would use either{read,write}_connection#query`. One requirement is we'd need to hold the read data in memory and then prepare it to be written, but that shouldn't been too bad!

Happy to take a pass at doing the above, let me know if it's still something you're interested in!