Skip to content

Test statistic is unstable. #13

@jordantgh

Description

@jordantgh

The use of T <- max(weights * abs(object - rowMeans(nullobject))) as the basis for P-values seems questionable. In practise, this usually hinges on a single disparity within in the top handful of items (assuming list agreements are generally higher near the top). I understand that the idea may have been to preferentially weight the top ranks, but this seems a bit much, and it will produce wildly varying P-value estimates among simulations when you don't fix the seed.

In the absence of a strong reason for your particular choice of test statistic, why not use something a little more stable, or provide a choice, e.g.

  # ...
  if (style == "exp") {
    if (is.null(lambda)) {
      lambda <- 0.1
    }
    weights <- exp(-lambda * (ranks - 1))
    weights <- weights / sum(weights)
  } else if (style == "hyperbolic") {
    weights <- 1 / ranks
    weights <- weights / sum(weights)
  } else {
    stop("Invalid style argument")
  }

  T <- sum(weights * abs(object- rowMeans(null_object)))

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions