Use persistent_term instead of ets

[ukutaht]

I just learned about persistent_term as an alternative to ets which is optimized for read performance: https://www.erlang.org/doc/man/persistent_term.html

I haven't tested it out but it should give a performance boost.

[mneudert]

Well then, please take a look at the persistent_term branch :)

It is an untested "hack" that provides a new configuration option to switch between the current :ets and a new :persistent_term storage. Probably not far off from a final implementation as with the big warning posted in the Erlang documentation the :ets storage may be the "sane by default without known problems" way to go.

So I would say the option should exist with both storages available so one can measure and select what fits best.

---

Checks done with the existing mix bench.lookup in the mentioned branch as it was readily available.

A really small database:

# parallel: System.schedulers_online() / 2
Name                                           ips        average  deviation         median         99th %
IPv4 lookup (PersistentTerm)              221.39 K        4.52 μs   ±401.02%        4.26 μs        8.03 μs
IPv4 lookup (ETS)                         220.72 K        4.53 μs   ±380.09%        4.26 μs        8.03 μs
IPv4 in IPV6 lookup (PersistentTerm)      127.35 K        7.85 μs   ±189.80%        7.47 μs       14.74 μs
IPv4 in IPV6 lookup (ETS)                 124.42 K        8.04 μs   ±171.39%        7.61 μs       15.23 μs

# parallel: System.schedulers_online() * 4
Name                                           ips        average  deviation         median         99th %
IPv4 lookup (PersistentTerm)               23.76 K       42.09 μs   ±487.82%        8.59 μs     1116.28 μs
IPv4 lookup (ETS)                          23.50 K       42.55 μs   ±484.59%        8.66 μs     1140.80 μs
IPv4 in IPV6 lookup (ETS)                  11.60 K       86.23 μs   ±350.15%       18.09 μs     1572.90 μs
IPv4 in IPV6 lookup (PersistentTerm)       11.58 K       86.34 μs   ±338.61%       18.09 μs     1548.95 μs

A roundabout 100MB database:

# parallel: System.schedulers_online() / 2
Name                                           ips        average  deviation         median         99th %
IPv4 lookup (PersistentTerm)               36.86 K       27.13 μs    ±47.72%       26.19 μs       46.10 μs
IPv4 lookup (ETS)                          35.79 K       27.94 μs    ±27.98%       26.68 μs       47.56 μs
IPv4 in IPV6 lookup (PersistentTerm)       29.24 K       34.20 μs    ±26.64%       32.90 μs       56.36 μs
IPv4 in IPV6 lookup (ETS)                  28.35 K       35.28 μs   ±105.28%       33.66 μs       59.09 μs

# parallel: System.schedulers_online() * 4
Name                                           ips        average  deviation         median         99th %
IPv4 lookup (ETS)                           4.84 K      206.62 μs   ±158.49%       47.14 μs     1034.85 μs
IPv4 lookup (PersistentTerm)                4.82 K      207.47 μs   ±161.45%       47.28 μs     1037.22 μs
IPv4 in IPV6 lookup (ETS)                   4.55 K      219.58 μs   ±153.16%       54.13 μs     1087.92 μs
IPv4 in IPV6 lookup (PersistentTerm)        4.08 K      245.01 μs   ±145.81%       56.85 μs     1228.60 μs

[ukutaht]

Nice, didn't know you've already explored this!

Interesting, the difference in throughput does not seem huge