inconsistence of data labelling

[tangyaohua]

i found two turns of the same dialogue where similar sentences have different labelled code:

dialogue_id: 320a1a29-c2ea-40c3-9253-175af766d515, turn: 0
user: what's the wether like where i am?
agent: Sorry, I can only help with your calendar.
code: (FenceScope)

dialogue_id: 320a1a29-c2ea-40c3-9253-175af766d515, turn: 1
user: what's the weather in my area?
agent: It is clear with a temperature of 0.00 °F right now.
code: (Yield :output (WeatherQueryApi :place (AtPlace :place (Here)) :time (?= (Now))))

[hao-fang]

Thanks for reporting the issue!
We do expect a small amount of noisy labels in the released training data.

Our test set (used in the leaderboard) has much less noise -- see the description in our paper.

Data is split into training, development, and test sets. We review every dialogue in the test set with two additional annotators. 75% of turns pass through this double review process with no changes, which serves as an approximate measure of inter-annotator consensus on full programs.

[tangyaohua]

hi, i found some confusing labels about Yield output.

dialogue_id: d5ce06a0-bd3f-4abf-9089-b1466cf368fe, turn: 4
user: Make it 11AM
agent: I found 2 different people matching your query. Which of these did you mean?
code: (Yield :output (Yield :output (Execute :intension (NewClobber :intension (refer (ActionIntensionConstraint)) :slotConstraint (andConstraint (roleConstraint (append #(List[Path] []) #(Path "start"))) (Constraint[Constraint[Time]])) :value (intension (?= (NumberAM :number #(Number 11))))))))
(Yield
:output (Yield
:output (Execute
:intension (NewClobber
:intension (refer (ActionIntensionConstraint))
:slotConstraint (andConstraint
(roleConstraint (append #(List[Path] []) #(Path "start")))
(Constraint[Constraint[Time]]))
:value (intension (?= (NumberAM :number #(Number 11))))))))

this example has two "Yield output"?

dialogue_id: 60fe6779-4105-4299-97f1-89293a4634cc, turn: 0
user: What time is the meeting on Friday?
agent: __NULL
code: (:time (:start (singleton (:results (FindEventWrapperWithDefaults :constraint (EventOnDate :date (NextDOW :dow #(DayOfWeek "FRIDAY")) :event (Constraint[Event])))))))
(:time
(:start
(singleton
(:results
(FindEventWrapperWithDefaults
:constraint (EventOnDate
:date (NextDOW :dow #(DayOfWeek "FRIDAY"))
:event (Constraint[Event])))))))

and this one has no "Yield Output".

[sammthomson]

Thanks!

Those two examples are also annotation errors.

Multiple Yields are fine, but they shouldn't be nested like in the first example, they should be sequenced with do, like (do (Yield :output ...) (Yield :output ...)).

The second example will calculate a value but not describe it back to the user, which is also wrong (and probably why the agent utterance is null).

[tangyaohua]

dialogue_id: d5ea0d53-5e2f-4f41-97a2-b93636b86760, turn: 0
user: Do I have a meeting on my calendar that's marked as Tentative in the next 10 days and longer than 45 minutes?
agent: Yes, I found one event on your calendar. It's "meeting with Dr. Stein" tomorrow from 7:00 to 8:30 AM.
code: (do (Constraint[Event] :duration (?> (toMinutes # (Number 45))) :nonEmptyBase (EventDuringRange :event (Constraint[Event]) :range (NextPeriod :period (toDays # (Number 10.0))))) #(Path "showAs") (?= # (ShowAsStatus "Tentative")) (Yield :output (> (size (:results (FindEventWrapperWithDefaults :constraint (Constraint[Event])))) # (Number 0))))
(do
  (Constraint[Event]
    :duration (?> (toMinutes #(Number 45)))
    :nonEmptyBase (EventDuringRange
      :event (Constraint[Event])
      :range (NextPeriod :period (toDays #(Number 10.0)))))
  #(Path "showAs")
  (?= #(ShowAsStatus "Tentative"))
  (Yield
    :output (>
      (size
        (:results
          (FindEventWrapperWithDefaults
            :constraint (Constraint[Event]))))
      #
      (Number 0))))

this is weird.

[tangyaohua]

dialogue_id: 7daad84c-f3e9-489f-8cd7-5e2632f2e830, turn: 0
user: cancel my meeting with francis
agent: Is this what you want me to cancel?
code: (do (Constraint[Event]) #(Path "attendees") (AttendeeListHasRecipientConstraint :recipientConstraint (RecipientWithNameLike :constraint (Constraint[Recipient]) :name # (PersonName "francis"))) (Yield :output (DeleteCommitEventWrapper :event (DeletePreflightEventWrapper :id (:id (singleton (:results (FindEventWrapperWithDefaults :constraint (Constraint[Event])))))))))
(do
  (Constraint[Event])
  #(Path "attendees")
  (AttendeeListHasRecipientConstraint
    :recipientConstraint (RecipientWithNameLike
      :constraint (Constraint[Recipient])
      :name #
      (PersonName "francis")))
  (Yield
    :output (DeleteCommitEventWrapper
      :event (DeletePreflightEventWrapper
        :id (:id
          (singleton
            (:results
              (FindEventWrapperWithDefaults
                :constraint (Constraint[Event])))))))))

dialogue_id: a3922891-7dc7-400a-ba4c-47b9bf462960, turn: 0
user: Hey, can you mark down a working lunch with Ilya and Daniel for tomorrow at the Spaghetti Factory from 12:30 to 2 pm please?
agent: Is this good?
code: (let (x0 (DateAtTimeWithDefaults :date (Tomorrow) :time (HourMinutePm :hours #(Number 12) :minutes #(Number 30.0)))) (do #(String "Spaghetti Factory") (TimeAfterDateTime :dateTime x0 :time (NumberPM :number # (Number 2))) #(String "working lunch") (Yield :output (CreateCommitEventWrapper :event (CreatePreflightEventWrapper :constraint (Constraint[Event] :attendees (andConstraint (AttendeeListHasRecipient :recipient (Execute :intension (refer (extensionConstraint (RecipientWithNameLike :constraint (Constraint[Recipient]) :name # (PersonName "Ilya")))))) (AttendeeListHasRecipient :recipient (Execute :intension (refer (extensionConstraint (RecipientWithNameLike :constraint (Constraint[Recipient]) :name # (PersonName "Daniel"))))))) :end (?= (TimeAfterDateTime :dateTime x0 :time (NumberPM :number # (Number 2)))) :location (?= # (LocationKeyphrase "Spaghetti Factory")) :start (?= x0) :subject (?= # (String "working lunch"))))))))
(let
  (x0
    (DateAtTimeWithDefaults
      :date (Tomorrow)
      :time (HourMinutePm :hours #(Number 12) :minutes #(Number 30.0))))
  (do
    #(String "Spaghetti Factory")
    (TimeAfterDateTime
      :dateTime x0
      :time (NumberPM :number #(Number 2)))
    #(String "working lunch")
    (Yield
      :output (CreateCommitEventWrapper
        :event (CreatePreflightEventWrapper
          :constraint (Constraint[Event]
            :attendees (andConstraint
              (AttendeeListHasRecipient
                :recipient (Execute
                  :intension (refer
                    (extensionConstraint
                      (RecipientWithNameLike
                        :constraint (Constraint[Recipient])
                        :name #
                        (PersonName "Ilya"))))))
              (AttendeeListHasRecipient
                :recipient (Execute
                  :intension (refer
                    (extensionConstraint
                      (RecipientWithNameLike
                        :constraint (Constraint[Recipient])
                        :name #
                        (PersonName "Daniel")))))))
            :end (?=
              (TimeAfterDateTime
                :dateTime x0
                :time (NumberPM :number #(Number 2))))
            :location (?= #(LocationKeyphrase "Spaghetti Factory"))
            :start (?= x0)
            :subject (?= #(String "working lunch"))))))))
'''
and these two codes.

[tangyaohua]

dialogue_id: 9e74eea7-da1c-47aa-9657-a2d6f0fd3d34, turn: 2
user: Is it hotter than Phoenix in Tucson, or colder?
agent: Yes. The temperature is 0.00 °F right now.
code: (Yield :output (< (WeatherAggregate :property #(WeatherProp "temperature") :quantifier #(WeatherQuantifier "Summarize") :table (WeatherQueryApi :place (AtPlace :place (FindPlace :keyphrase # (LocationKeyphrase "Tucson"))) :time (?= (Now)))) (WeatherAggregate :property #(WeatherProp "temperature") :quantifier #(WeatherQuantifier "Summarize") :table (WeatherQueryApi :place (AtPlace :place (FindPlace :keyphrase # (LocationKeyphrase "Tucson"))) :time (?= (Now))))))
(Yield
  :output (<
    (WeatherAggregate
      :property #(WeatherProp "temperature")
      :quantifier #(WeatherQuantifier "Summarize")
      :table (WeatherQueryApi
        :place (AtPlace
          :place (FindPlace :keyphrase #(LocationKeyphrase "Tucson")))
        :time (?= (Now))))
    (WeatherAggregate
      :property #(WeatherProp "temperature")
      :quantifier #(WeatherQuantifier "Summarize")
      :table (WeatherQueryApi
        :place (AtPlace
          :place (FindPlace :keyphrase #(LocationKeyphrase "Tucson")))
        :time (?= (Now))))))

duplicated code.

[tangyaohua]

dialogue_id: c249e278-2a67-459b-b573-8afe10a5c87f, turn: 0
user: I need to create a new calender
agent: Does one of these work?
code: (CreateCommitEventWrapper :event (CreatePreflightEventWrapper :constraint (Constraint[Event])))
(CreateCommitEventWrapper
  :event (CreatePreflightEventWrapper
    :constraint (Constraint[Event])))

[tangyaohua]

dialogue_id: 107d8b55-1269-4c57-b8e3-25fad419cf5f, turn: 5
user: Is it going to rain in the entire area all day?
agent: __NULL
code: (Execute :intension (refer (extensionConstraint (Constraint[Event]))))
(Execute
  :intension (refer (extensionConstraint (Constraint[Event]))))

[tangyaohua]

dialogue_id: f255399e-961a-4fab-8a86-6a95b4bb52dd, turn: 2
user: I meant to say Lois Lane.
agent: How's this?
code: (Execute :intension (ReviseConstraint :rootLocation (roleConstraint #(Path "output")) :oldLocation (Constraint[Constraint[Event]]) :new (Constraint[Event] :attendees (andConstraint (AttendeeListExcludesRecipient :recipient (?= (Execute :intension (refer (extensionConstraint (RecipientWithNameLike :constraint (Constraint[Recipient]) :name #(PersonName "Clark Kent"))))))) (AttendeeListHasRecipient :recipient (Execute :intension (refer (extensionConstraint (RecipientWithNameLike :constraint (Constraint[Recipient]) :name #(PersonName "Lois Lane"))))))))))
(Execute
  :intension (ReviseConstraint
    :rootLocation (roleConstraint #(Path "output"))
    :oldLocation (Constraint[Constraint[Event]])
    :new (Constraint[Event]
      :attendees (andConstraint
        (AttendeeListExcludesRecipient
          :recipient (?=
            (Execute
              :intension (refer
                (extensionConstraint
                  (RecipientWithNameLike
                    :constraint (Constraint[Recipient])
                    :name #(PersonName "Clark Kent")))))))
        (AttendeeListHasRecipient
          :recipient (Execute
            :intension (refer
              (extensionConstraint
                (RecipientWithNameLike
                  :constraint (Constraint[Recipient])
                  :name #(PersonName "Lois Lane"))))))))))

[tangyaohua]

dialogue_id: 94bc3b21-218d-441c-a61d-931f64df7479, turn: 0
user: Add "Mark's Bachelor Party" to my calendar for this Saturday between 5 and 11 PM.
agent: Does this look right?
code: (let (x0 (Constraint[Event] :subject (?= # (String "Mark's Bachelor Party"))) x1 #(Path "start") x2 (DateAtTimeWithDefaults :date (NextDOW :dow # (DayOfWeek "SATURDAY")) :time (NumberPM :number # (Number 5))) x3 (CreateCommitEventWrapper) x4 #(Path "event") x5 (CreatePreflightEventWrapper) x6 #(Path "constraint")) (do (callCreateCommitEventWrapper (update x3 x4 (callCreatePreflightEventWrapper (update x5 x6 (Constraint[Event] :end (?= (TimeAfterDateTime :dateTime x2 :time (NumberPM :number #(Number 11)))) :nonEmptyBase (update x0 x1 (?= x2))))))) (Yield :output (callCreateCommitEventWrapper (update x3 x4 (callCreatePreflightEventWrapper (update x5 x6 (Constraint[Event] :end (?= (TimeAfterDateTime :dateTime x2 :time (NumberPM :number #(Number 11)))) :nonEmptyBase (update x0 x1 (?= x2))))))))))
(let
  (x0
    (Constraint[Event]
      :subject (?= #(String "Mark's Bachelor Party")))
    x1
    #(Path "start")
    x2
    (DateAtTimeWithDefaults
      :date (NextDOW :dow #(DayOfWeek "SATURDAY"))
      :time (NumberPM :number #(Number 5)))
    x3
    (CreateCommitEventWrapper)
    x4
    #(Path "event")
    x5
    (CreatePreflightEventWrapper)
    x6
    #(Path "constraint"))
  (do
    (callCreateCommitEventWrapper
      (update
        x3
        x4
        (callCreatePreflightEventWrapper
          (update
            x5
            x6
            (Constraint[Event]
              :end (?=
                (TimeAfterDateTime
                  :dateTime x2
                  :time (NumberPM :number #(Number 11))))
              :nonEmptyBase (update x0 x1 (?= x2)))))))
    (Yield
      :output (callCreateCommitEventWrapper
        (update
          x3
          x4
          (callCreatePreflightEventWrapper
            (update
              x5
              x6
              (Constraint[Event]
                :end (?=
                  (TimeAfterDateTime
                    :dateTime x2
                    :time (NumberPM :number #(Number 11))))
                :nonEmptyBase (update x0 x1 (?= x2))))))))))

[tangyaohua]

dialogue_id: d5ea0d53-5e2f-4f41-97a2-b93636b86760, turn: 0
user: Do I have a meeting on my calendar that's marked as Tentative in the next 10 days and longer than 45 minutes?
agent: Yes, I found one event on your calendar. It's "meeting with Dr. Stein" tomorrow from 7:00 to 8:30 AM.
code: (do (Constraint[Event] :duration (?> (toMinutes # (Number 45))) :nonEmptyBase (EventDuringRange :event (Constraint[Event]) :range (NextPeriod :period (toDays # (Number 10.0))))) #(Path "showAs") (?= # (ShowAsStatus "Tentative")) (Yield :output (> (size (:results (FindEventWrapperWithDefaults :constraint (Constraint[Event])))) # (Number 0))))
(do
  (Constraint[Event]
    :duration (?> (toMinutes #(Number 45)))
    :nonEmptyBase (EventDuringRange
      :event (Constraint[Event])
      :range (NextPeriod :period (toDays #(Number 10.0)))))
  #(Path "showAs")
  (?= #(ShowAsStatus "Tentative"))
  (Yield
    :output (>
      (size
        (:results
          (FindEventWrapperWithDefaults
            :constraint (Constraint[Event]))))
      #
      (Number 0))))

this is weird.

is this wrong?