IN and !? or "negated apply"

[mtoy-googly-moogly]

This is just a placeholder to write down things.

Early in experimenting with apply, we discovered that

x = 7 | 8 expands to x = 7 or x = 8 worked as we expected and we were pleased, however, if i wanted everything EXCEPT 7 or 8, and wrote x != 7 | 8 that expands to x !=7 or x !=8 which is true for all numbers. ( #1353 )

Meanwhile, people who want the SQL IN operator are writing

expr ? (a | b | c) ... and wanting to know how to write NOT IN

So here are some things that we could think about ...

IN and NOT IN

I really don't like this, because this is now a weird SQL wart which has appeared in Malloy. If we want to stop making Malloy it's own language and turn it into mostly SQL, I want to do that on purpose, not back into it a little bit at a time.

So until we say "I was a mistake for Malloy to have it's own syntax" I am against this idea, but it needs to b e mentioned.

color in (red, green, blue) and state not in ('CA', 'NY')

in and !in

Using the in operator like sql, but using !in like the existing != and !~

color in (red, green, blue) and state !in ('CA', 'NY')

? and !?

Current use of apply, and adcd a new "not apply" ...

color ? (red | green | blue) and state !? ('CA' | 'NY')

~ instead of in

~ is already "funky generic matching operator" so maybe we could extend it to funky generic match lists ...

color ~ (red, green, blue) and state !~ ('CA', 'NY')

[christopherswenson]

Some points to consider:

Syntax for list:

• It would be nice if the RHS of the operator was syntactically equivalent to whatever we want literal arrays to look like, as well as being standalone (its syntax parsing is not dependent on the presence of the in operator). I would have assumed that would be [1, 2, 3].
• Do we ever anticipate having a distinction between tuples (1, 2, 3) and arrays
• I definitely feel comfortable moving away from alternation (1 | 2 | 3). It was a neat experiment, but not useful enough to merit the cognitive complexity

Syntax for in operation:

• I don't like not in because every other Boolean operator negation in the language uses !
• I'd be happy with !in: in is extremely clear what's happening, and !in fits the pattern of how we negate comparison operators in Malloy today. One downside is that in would be the only English operator in a language full of symbolic operators
• !? is pretty obscure. ? could work, but I don't think we want the apply operator to become a catch all "do the right thing" operator. I think it will make more sense to users to have specific operators for specific things. And if we're moving away from alternations, the uses of ? may start to become very limited or even go away completely.
• I have a similar fear with ~ that it becomes a catch all "do the right 'is kind of like'" operator. I think this one is more readable than !?, because ~ has a pretty well understood meaning of "is similar to," but in definitely wins on clarity

[christopherswenson]

Happy to discuss further! LMK what you think!

[mtoy-googly-moogly]

Comments on your comments.

? means "apply the predicate and return the boolean value of the application and !? would mean "apply the predicate and return the negation of the boolean value. That is x !? y is the same as (not (x ? y)) So it isn't a "do the right thing operator" it is just the negated version of apply

~ is already a dumping ground do the right thing operator -- we should kill it, or celebrate it (in the far future)

On [] vs () for matching lists ... just thinking this through, I think you still want () because if lists are an atomic data time then I think you still want to distinguish between elements in the "in" list ... maybe?

description in ([crunchy, spicy], [smooth, creamy], [runny, smelly])

[christopherswenson]

Yeah, I'm definitely on board with !? if we choose to use ? for the in operation, that follows the established pattern.

I'm just not sure I'm happy with ? being the in operator. I'll start with ~ though:

~ currently does two things:

• lhs ~ pattern = "is lhs LIKE pattern"
• lhs ~ regex = "does lhs match regex"

These are both string matching operations, and so ~ can reasonably be understood as the "string matching operator." The IN operator is not a string matching operator, so this is a clear expansion of the meaning, in a way which doesn't follow the pattern, hence the "do the right thing" comment.

? currently does the following:

• pick matching
• partial condition matching
• "do the right thing" equality/range checking https://docs.malloydata.dev/documentation/language/expressions#application

I guess if we're happy with the last of the above list, then maybe ?/!? is the right thing...

I almost wonder whether the first use (pick matching) is the outlier of ?, not the rule... Need to do more thinking about that.

I'm okay with (a, b, c), but I would want sentences like x = (1, 2, 3) or (1, 2, 3) + 7 to parse even if they are invalid and generate errors. I guess the question I would need to answer is: what is the Malloy type of (1, 2, 3)?

[mtoy-googly-moogly]

not arguing ... just adding to the conversation ...

? means "treat the rhs as a boolean predicate which take the lhs as an argument"

the most magic thing that happens is that granular times, when ask to express themselves as a boolean predicate, turn into a range
the least magic thing is that a value, when asked to express itself as a predicate, turns into an equality test

there was some thought that, beloved by someone, looker daet filters, would show up as

dateExpr ~ 'last week for 272 minutes'

which would put even more magic into ~ and maybe in a world where ? isn't an operator, that is where granular ranges would end up also

timeExpr1 ~ timeExpr2.month

[mtoy-googly-moogly]

... but maybe that would be timeExpr1 in timeExpr2.month ?

[skokenes]

My vote is in and !in. I agree that introducing not when everything else uses ! for negation seems strange.

I like in because it's clear about what it does. Yes, other operators in the language are symbolic. But most of those operators are commonly understood operators from existing domains. Things like +, - , * , / , > , < , = are all known symbols. ? and ~ are not known operators for "in". At the Rill meetup, Lloyd was asked "what changed about Malloy from v1 to v4". He said v1 was extremely concise, but hard to read. That resonated with me, and perhaps why I like in over the more ambiguous options.

Re: looking like SQL vs not looking like SQL, I could care less whether it looks like SQL or not. I think our goal is to make what we think is the best language for semantic modeling and querying, regardless of what SQL does or does not do? In that case, I hope we are voting against in/not in purely because we don't like it as syntax, and not (😉) because it happens to be what SQL uses.

[mtoy-googly-moogly]

re: SQL ness

People at the front line of trying to convince people to use Malloy, have consistently been reporting back that people would be way happier if we looked more like SQL. Meanwhile SQL is kind of and old and ugly language that could use some help.

I feel like I had a balance in my head for that tension. What I hear back from the universe is that my balance is wrong and we need to be "worse" according to my idea of balance in order to be "better" in terms of being a successful language.

I am trying to be humble and listen to users on this.

[skokenes]

Also I can't figure out why, but I like in (1, 2, 3) over in [1, 2, 3]. I guess for some reason I don't think of the set list as a literal array, even if thats what it is 🤷‍♂️

[mtoy-googly-moogly]

I see some support for in and !in

I could maybe live with that, I think I was hopeful people would like a wildy overloaded ~ because RIght now ~ and !~ mostly mean LIKE and NOT LIKE and I am staring at the slippery slope to LIKE and !LIKE and feeling very uncomfortable.

[christopherswenson]

Yeah these are all great points. Think we could use a VC to sync up on this.

[mtoy-googly-moogly]

right now i am just collecting data ... for sure before we write the WN we should meet with faces and hammer our compromises.