WN 0002: Parameters in Malloy

[lloydtabb]

See current Proposal at: https://github.com/malloydata/whatsnext/blob/main/wns/WN-0002/wn-0002.md

[skokenes]

+1 for support for Abstract views, etc. This use case especially I find myself hitting a lot while writing Malloy https://github.com/malloydata/whatsnext/blob/main/wns/WN-0002/wn-0002.md#abstract-views-let-us-build-reusable-query-shapes

Typically in a model, I will land on an interesting view and then want to look at that same view shape, but with different dimensions, measures, or some other parameter. Right now the only solution is to copy/paste and manually change values.

[mtoy-googly-moogly]

Here's the argument against looker string filters.

Malloy already has a facility for writing expressions which apply a series of tests to a value, it is wrong to re-invent something that exists. A abstract thingy which takes a filter should be taking a function and applying to the result, the source of the function could be a partial expression, or it could actually be a function ... in pseudo code ...

query:  foo(testFunc: (timestamp) => boolean) is unfilteredFoo {
  where: fooTime ? testFunc
}

run: foo(@2003)
run:foo(@2003 to @2005)
run:foo(>now.day - 2)

function:onMyBirthday(ts: timestamp) => boolean is {
  return month(ts) = 6 and day(ts) = 30
}

run: foo(onMyBirthday)

If we decide we want looker filters, first we need to make them work inthe language ... again just making up syntax

run: someSource extend { where: eventTime ?[|last week for 32 hours] }

Then looker filters would also work in

run: foo([between last month and two days ago])

[szawadski]

This would be a great add-on and would simplify lots of unclear workarounds. In particular, I am using now a virtual source to define a reference date that I inner join with other sources filter them wrt that reference date. Being able to use a globally defined parameter would make it more clear to read and simple to implement.

[christopherswenson]

I created a new WN 0014 for an MVP version of parameters: https://github.com/malloydata/whatsnext/blob/main/wns/WN-0014-mvp-parameters/WN-0014.md

[christopherswenson]

Thread for how we handle "Preview" lens on sources with parameters

(and also views on sources with parameters, and eventually views with their own parameters)

We almost certainly need to re-work the lenses to specify source name to preview (and for views, source name and view name), rather then simply code to run.

A couple options I can think of:

• Use VSCode UI elements:

  • In the worker, allow the run query task to respond with “needs parameters” (by compiling the model, extracting the source or view, and checking if it has/needs parameters)
  • Have the extension pop up an input box requesting the parameters, then re-run the "run query" task in the worker, passing in those parameters

• Use custom elements:

  • Implement some re-usable elements for specifying parameter values
  • When running a query, if it has required parameters, send a message to the webview to display parameter widgets
  • When the parameter widgets are changed (maybe with a commit "Run" button), send a message back to the worker to actually run the query

I personally think I prefer the "use custom elements" version. What do others think?

[skokenes]

I like the custom elements approach as well, from a UX perspective.@whscullin would be better suited to remark on the technical challenges it may present.

[carlineng]

My 2c on UX -- the user should be able to enter parameter values before they run a query. If a user must be prompted to enter parameter values on a "run" action, that's a few extra clicks that are required. If they attempt to run a query without valid parameter values, the UI should highlight the parameter input box; similar to web input forms where a required value is missing.

[christopherswenson]

This is a good point — if we start off with parameters always needing a default value specified on declaration (at least for the MVP), then it'd be really annoying to click the run button and have it run a version of the query with the default parameter value when you actually wanted to override it. Maybe there could just be two lenses here?

1. Run/preview with default parameter values
2. Specify parameter values then run

Option 2 could also work like "Load parameterized query in result panel without running" (and then allow the user to specify the parameter, then click run inside the result panel)

[christopherswenson]

Thread for how we handle parameters in Composer?

This could work a number of different ways.

One idea is that all source parameters show up as widgets in the result panel (similar to the "use custom elements" idea for VSCode). When we have view/query parameters, you could also add parameters to your query (and then they would also be added as widgets to the result panel). When we have dimension/measure/view parameters, we can have an additional context bar that pops up asking you to fill in the parameters whenever you add something with parameters (if you had added parameters to the query, you could use those parameters as the value for dimension/measure/view parameters included in the query).

[skokenes]

Why not show the parameter inputs in the left QuerySummary side panel?

[christopherswenson]

I'm thinking about the way that parameters will work in code. The composer currently only defines views, so the defined part technically does not include the source (or its parameterization), so it's like defining:

source: whatever(some parameters) is ... extend {
  view: this_is_what_the_composer_defines is {
    ...
  }
}

And it's not until you run it that you specify source parameters:

run: whatever(arguments) -> this_is_what_the_composer_defines

So from that standpoint it seems to make sense to specify the parameters outside the QuerySummary panel. It might confuse people if they can specify source parameters in the query panel, but then when they grab the code for the view: definition, it doesn't include those parameters. Then again, we can also (and I think already do?) generate run: code as well, so we could easily generate code with parameters filled in for that...

I guess it's mostly a UX choice here of where it feels most appropriate to include them. An argument in favor of having them in the query summary is that it would be annoying to have to look two different places to fully specify your query.

[carlineng]

So from that standpoint it seems to make sense to specify the parameters outside the QuerySummary panel. It might confuse people if they can specify source parameters in the query panel

Putting myself in the shoes of a user trying to run a query, the distinction between "source" vs "view" parameter is probably not going to be something I'm going to be thinking about. Composer is primarily an interface for building and running queries, so my instincts are going to be that the parameters are associated with the queries.

but then when they grab the code for the view: definition, it doesn't include those parameters

I can see how this would be confusing, but I suspect this isn't all that common of a workflow. I see Composer as more of a query running/data exploration tool, as opposed to a modeling tool.

All that is to say I think QuerySummary panel is probably the right place.

[christopherswenson]

Thread for "Extending Parameterized Sources is Cumbersome"

For sources with even a small number of parameters, extension becomes pretty cumbersome:

source: foo_base(
  param_1::string,
  param_2::string
  ...
) is ... extend { ... }

// ugh!
source: foo_ext(
  param_1::string,
  param_2::string,
  ...
) is foo_base(
  param_1 is param_1,
  param_2 is param_2,
  ...
) extend { ... }

This is a clear downside to parameterized sources behaving like functions, rather than parameters being a property of the source, e.g.:

source: foo_base is ... extend { 
  parameter: param_1::string
  parameter: param_2::string
  ...
}

// hooray, I got all the parameters for free!
source: foo_ext is foo_base extend { ... } 

This downside can be weighed against:

• function-like syntax is familiar
• function-like syntax makes the interface very clear

[lloydtabb]

I like the second form better. We're going to need..

   parameter: param_1:: string required
   parameter: param_2::string default 'something'

[lloydtabb]

Actually, maybe there is only 'default', if it is required, it needs to default to something so we can make the code lenses work.

[carlineng]

Parameters feel different enough from other things that go in source blocks (dimension, measure, join, where) that it feels like it should be in a separate space; so I prefer the function-like syntax. The latter are describing the data model; the former is a way for the user to control things about execution (this is poorly phrased, but hopefully you get the gist).

Naive question; why isn't this an option?

source: foo_base(
  param_1::string,
  param_2::string
  ...
) is ... extend { ... }

source: foo_ext(
  param_1::string,
  param_2::string,
  ...
) is foo_base extend { ... }

Since foo_ext extends foo_base, shouldn't it just inherit the same parameters with the same names? E.g., in Python if I have a Class A with a constructor, and I extend that to create Class B, B automatically inherits the constructor for A, so I can call new B(...) without having to explicitly create a constructor for B that passes all the args back down to A.

[christopherswenson]

I think your comparison with classes exactly highlights why we might want these to be defined inside the source rather then in a function-signature like syntax.

With dimensions, measures, and views, I'm pretty sure we want parameters to be treated like function parameters, where they are always part of the invocation of the thing — to pass them in, you must do so every time you name the thing. There's no such thing as "extending" a measure, dimension, or view.

Sources, on the other hand, behave much more like classes than functions, as you pointed out: you want to inherit source parameters when extending. So it makes more sense to me to put the parameters inside the declaration.

Now there's also the possibility of a mixed solution, where parameters that are declared inside a source are inherited and public, and parameters declared in the signature are not inherited (and private), and must be passed through during extension:

source: foo_base(
  param_1::string
) is ... extend { 
  parameter: param_2::string
}

// param_1 must be passed in
source: foo_ext(
  param_1::string
) is foo_base(param_1 is param_1) extend {
  // param_2 is automatically inherited 
}

Though I think the utility of any sort of mixed solution like this would need to be carefully considered, and is almost definitely not part of an MVP.

[christopherswenson]

There is a new proposal to add the ability to inherit the default value (as well as tags) from the base class when extending, with syntax like:

source: foo_base(param is "default value") is ...
source: foo_ext(param) is foo_base(param) extend { ... }

I.e., by not providing a type or default value in the definition, you opt in to using the type, default value, and tags from the base source definition.

There is an additional idea to allow:

source: foo_base(param is "default value") is ...
source: foo_ext(*) is foo_base(*) extend { ... }

This would copy all parameters from the base source. I feel a bit less sure about this latter syntax, since it muddies the clarity a bit too much for my taste. My preference would be to add the first behavior to the experiment, and see if people feel the need for the second syntax. But I'm curious what others think.

[szawadski]

Is it planned to have parameters on queries?

[christopherswenson]

Yes, we plan to have parameterized views and queries.