Is there a way to re-fetch and merge server collection changes into minimongo multiple times via a method call?

[Avijobo]

Hi,

I am a bit confused with the capabilities of this package. The readme states:

"With jam:pub-sub, you use Meteor.publish.once and the same Meteor.subscribe to have the data fetched via a Meteor Method and merged automatically in Minimongo so you can work with it as you're accustomed to."

Besides the initial goal of fetching the data once, this let me believe that I can also fetch the data multiple times via a method call, however I don't quite understand how. Is it just a matter of repeating "Meteor.publish.once" in a method call, with e.g. a different query filter?

If this would be possible, will the updated minimongo collections also re-trigger corresponding reactive computations on the client after each re-fetch?

Or am I misinterpreting this, and isn't this possible at all?

If this would work, this could give me a more efficient alternative to conditional reactive (pipeline) publications.

[jamauro]

Hmm, I'm not quite sure I understand what you're hoping to achieve. Can you provide some more context or a small example?

At the moment, Meteor.publish.once, combined with its corresponding Meteor.subscribe, fetches data once and merges it into minimongo. Under the hood, it's using a Meteor.method to make this happen. Whenever Meteor.subscribe is triggered, it will fetch again (unless you're caching the subscription).

It also allows for single-client reactivity so when they make actions they'll see them but those actions won't be broadcasted to other connected clients.

[Avijobo]

Yes, this is what I also understood as the main goal of the package.

This is what I was hoping to achieve:

I have a number of collections that contain different types of documents, but all have a relation to each other with conditional permissions.

A simplified example: a 'song' document in the 'songs' collection may contain a number of instrument 'track' documents in the 'tracks' collection. The way it is organized, is that all 'track' documents contain the _id of the corresponding song document. Going further, a track typically contains a number of audio segments which we call 'parts', so each 'part' document in the 'parts' collection on its turn contains the _id of its corresponding track document it belongs to, .etc etc. Now each of these documents may conditionally hide subsequent types, depending on user access rights (roles).

So, given a specific client, it means I need to dynamically publish only subsets of 'songs', 'tracks and 'parts', depending on other document's changing fields.

To achieve this, in Meteor v2.x, I used 'peerlibrary:reactive-publish', which worked perfectly for the last 5 years, never had a single problem with this package. Now since the upgrade to Meteor 3.x, I cannot use this library anymore because it does not allow async collection functions which I need for the conditional queries inside reactive-publish its reactive autorun() functions.

So I tried publish-composite. It works for 95% of the cases, but occasionally not all client side documents become (correctly) updated.

So as an other alternative I recently tried zodern:relay, which has 'reactive pipelines'. This works most of the time, behaves as expected in most cases, but in about 1% of the cases it crashes the server, so I cannot use this library either. (I reported these issues in the zodern:relay repo, but haven't got any response yet.)

Now I do have a lot of other collections 'dependencies', the above example is only a simplified scheme, in practice up to about 8 reactive pipelines are needed for the lowest document 'leafs' in the song 'tree'.

Ideally I would forget pub/sub for most of my collections and just have a means of fetching and updating document changes from the server into minimongo 'manually' by using a method, but this method would need to allow me to rerun e.g. publish.once() so that I can change the query filters depending on other collection queries output.

Because each 'song' document also contains a 'modifiedAt' field that is updated if any of its children docs ('track', 'part' etc...) changed, I could set up one regular Meteor.publish()/subscribe() that notifies the client to 'refetch' the other collections' data when the song changed.

Anyway, I was hoping to be able to achieve something like this with jam:pub-sub.

P.S. On a different matter: I also suspect that maybe the long delays I'm having with jam:transactions may come from these reactive publications?

[jamauro]

I see. Sounds like an interesting problem. :) So basically it sounds like you would ideally just fetch everything manually but you want all the results merged into minimongo on the client so using a regular Meteor.method won't get the job done. I'll have to think more on it.

Just curious, why not store the song as one document and embed all the rest of its data? Mongo is still not great with relational data.

[Avijobo]

Thanks for the quick response!

A song is modular, and may become quite large, may have alternative tracks, mixes, effects, layered recordings, annotations etc. This song e.g. consists of a few thousand documents internally, besides the audio/video/graphics data that it uses:

https://puremusicians.com/tryout/page/1%252520-%252520Gun%252520In%252520My%252520Hand

of which only a few documents may be altered during small edits, so it would not be efficient to store the entire song into one document. One other reason is that a lot of times I use the same functions both on server and client side that query collections, so I prefer having the same collections existing on both sides, be it that the client may only see a limited subset of a song, e.g. only the tracks or song sections that are permitted for him.

I do have to say that the 'this autorun()' function provided by reactive-publish which can be used inside a regular Meteor.publish() function was a nice solution that worked flawlessly. It is a pity that this library is not compatible anymore, as a reactive context on the server is very useful to me.

But indeed a way to manually fetch the documents via a method by only transporting the document changes and automerging/updating would be a very nice solution too. That's what I though pub-sub was also able too.

In all honesty, I am a bit desperately seeking for a reliable solution now since upgrading to Meteor 3.x, so any other ideas would also be greatly appeciated.

Thanks for listening anyway!