Fix: skip sincedb eviction if read mode completion deletes file during flush

If the value of the setting `sincedb_clean_after` is too short (1 ms) it happens that during the content reading loop (in read mode) the loops itself trigger the cleanup of the same value from the SincedbCollection and then it
also try to access it again creating an error.
This commit resolves the problem guarding against the presence in the sincebd_collection before accessing it.

Fixes: #272
Closes: #273

Author: andsel

CHANGELOG.md

@@ -1,3 +1,6 @@
+## 4.2.1
+  - Fix: skip sincedb eviction if read mode completion deletes file during flush [#273](https://github.com/logstash-plugins/logstash-input-file/pull/273)
+  
 ## 4.2.0
   - Fix: watched files performance with huge filesets [#268](https://github.com/logstash-plugins/logstash-input-file/pull/268) 
   - Updated logging to include full traces in debug (and trace) levels

lib/filewatch/read_mode/handlers/read_file.rb

@@ -19,8 +19,10 @@ def handle_specifically(watched_file)
           watched_file.listener.eof
           watched_file.file_close
           key = watched_file.sincedb_key
-          sincedb_collection.reading_completed(key)
-          sincedb_collection.clear_watched_file(key)
+          if sincedb_collection.get(key)
+            sincedb_collection.reading_completed(key)
+            sincedb_collection.clear_watched_file(key)
+          end
           watched_file.listener.deleted
           # NOTE: on top of un-watching we should also remove from the watched files collection
           # if the file is getting deleted (on completion), that part currently resides in

lib/filewatch/settings.rb

@@ -6,7 +6,7 @@ class Settings
     attr_reader :max_active, :max_warn_msg, :lastwarn_max_files
     attr_reader :sincedb_write_interval, :stat_interval, :discover_interval
     attr_reader :exclude, :start_new_files_at, :file_chunk_count, :file_chunk_size
-    attr_reader :sincedb_path, :sincedb_write_interval, :sincedb_expiry_duration
+    attr_reader :sincedb_path, :sincedb_expiry_duration
     attr_reader :file_sort_by, :file_sort_direction
     attr_reader :exit_after_read
     attr_reader :check_archive_validity
@@ -41,7 +41,6 @@ def add_options(opts)
       @file_chunk_size = @opts[:file_chunk_size]
       @close_older = @opts[:close_older]
       @ignore_older = @opts[:ignore_older]
-      @sincedb_write_interval = @opts[:sincedb_write_interval]
       @stat_interval = @opts[:stat_interval]
       @discover_interval = @opts[:discover_interval]
       @exclude = Array(@opts[:exclude])

lib/filewatch/sincedb_collection.rb

@@ -216,7 +216,7 @@ def sincedb_write(time = Time.now.to_i)
         @write_method.call
         @serializer.expired_keys.each do |key|
           @sincedb[key].unset_watched_file
-          delete(key) # delete
+          delete(key)
           logger.trace? && logger.trace("sincedb_write: cleaned", :key => key)
         end
         @sincedb_last_write = time

logstash-input-file.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-input-file'
-  s.version         = '4.2.0'
+  s.version         = '4.2.1'
   s.licenses        = ['Apache-2.0']
   s.summary         = "Streams events from files"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"