time offset between squelcher <-> postgres output causes trouble

selects are faster than inserts on the eventdb
squelcher is faster than postgres output
-> events queue up at the output
but the squelcher relies on the data in the table, not the data that it had processed
-> squelcher does not see all the data
-> events are sent too often

possible solutions:

• squelcher has it's own cache (additionally?)
• a bot which only passes events to the squelcher, if the queues behind are all empty

[aaronkaplan]

Let's solve it like this: the squelcher bot shall have his own redis cache. The cache stores the hash of the unique key (TBD) and gives it a configurable TTL for automatic expiry.

The squelcher then checks:

  If the hash(tuple) is not in cache:  # cache expires on its own
    set_notify = True
    additionally_check_postgresql = True
    # add stub code for counting miss rate stats
  else:   # we saw it
    notify = False
    additionally_check_postgresql = False
    # add stub code for counting hit rate stats
  put_into_cache( { hash(tuple): (time.source, notify) } , ttl )

...
def check_in_postgresql():
   if additionally_check_postgresql:
      do the SELECT

In addition to the postgresql check. The postgresql check comes after this check

The database lookup could also be performed right on the INSERT by using a INSERT Trigger:

• An expert adds the TTL to the event by matching the event to a local configuration file (squelcher.conf) or fetches the information for other sources (Constituency Portal!)
• The postgresql output bot just writes the data to the database
• An INSERT trigger does the actual lookup right in time, using the TTL which is part of the data itself.
  This prevents the time offset and offloads the complexity to Postgres itself.