Find the 'top 100' metabolites

[joefutrelle]

Find the 'top 100' metabolites - we have a set of ~50,000 metabolites, many of which are found in multiple experiments and in multiple samples within an experiment. What are the most common metabolites? What is the distribution of metabolites?

[joefutrelle]

A related question is "given a randomly-selected metabolite, at what probability will it match a set of metabolites of size N" for N from 0 (no matches) to the maximum cluster size observed--that gets to the distribution question and it can be estimated via random selection.

[joefutrelle]

Here's an example histogram of that sort generated from tps4 and tps6 data, just showing cluster size and number of clusters of that size found, based on 4000 random samples

1: 3411
2: 369
3: 126
4: 53
5: 17
6: 8
7: 7
8: 4
14: 1
15: 1
17: 2
18: 1

[joefutrelle]

that capability committed to branch pdf

[redbluewater]

Can I interpret the output as follows:
(1) There are 3411 features (metabolites) that are only found in 1 sample
(2) In one case, there is a metabolite found in 18 samples
If so...the number of metabolites found repeatedly drops off quickly into long tail.

Now the question is, what are the mz and RT values for each of these metabolites? My first thought would be to calculate an average mz and RT value so I can use that to query back into the database to figure out which samples actually have the high frequency features. For now, I am not interested in the singletons. But I am interested in things that are found 'often'.

[joefutrelle]

That's the correct interpretation, except for some wrinkles:

1. The probability that a randomly-selected metabolite belongs to a cluster of occurrences is proportional to the size of the cluster, so this approach is (slightly?) biased towards metabolites that occur more often
2. It's theoretically possible that the size of a cluster found using this method depends on which member of the cluster is randomly selected as the query, as you allude to above (could be randomly hitting the edge of a cluster) ... it depends on how clumpy this data is
3. This approach doesn't exclude previously-found metabolites and so can count a cluster more than once

The next step is to use a true clustering algorithm to exhaustively and unambiguously label each metabolite--random selection is limited by the problems listed above. But it gives us a look at the long-tail nature of the PDF

[joefutrelle]

I should say,

There are 3411 features (metabolites) that are only found in 1 sample

is more accurately summarized as

Out of 4000 randomly selected metabolites, 3411 were only found in 1 sample

[joefutrelle]

I should probably correct my language to use the word "feature" where I'm using the word "metabolite", I hope I'm making sense ...

[redbluewater]

This does make sense. Honestly, in house we use metabolite and feature interchangeably.

[joefutrelle]

Looking at some scatterplots it seems to me that the largest recognizable clusters appear to be line-like with very closely matching mzs but "smeared" along the rt axis, much farther than our default tolerance of 30s. I'm not clear yet on what the best clustering algorithm is likely to be for this kind of data, and perhaps we should look at some of these graphics together--I can prepare some for our meeting this week.

[joefutrelle]

Here's an example of tps4 and tps6 data where you can see these linear-looking features smeared out along the rt axis (I'm not showing the entire dataset here):