KeyDict usage slow

[whoburg]

ConstraintSet.varkeys is slow. This affects Model.__getitem__. Is there any way we could store Model._varkeys and have a boolean flag that gets set by other methods when the varkeys need to be recomputed? Otherwise, I end up storing the varkeys and then accessing those instead of the model, to prevent multiple calls to varkey.

Timing example from a model with only ~400 variables:

In [18]: t = time.time(); (M["C_L"], M["C_D"]); print time.time() - t
2.10050082207

In [19]: t = time.time(); mvk = M.varkeys; (mvk["C_L"], mvk["C_D"]); print time.time() - t
1.06283092499

[whoburg]

Here's the type of local hack I'm doing till this gets resolved:

Inside a Model :

      @property
      def varkeys(self):
          # local speedy varkey access -- this is temporary until we
          # implement this more generally in gpkit
          if self._fastvks:
              return self._fastvks
          # pylint:disable=no-member
          self._fastvks = Model.varkeys.fget(self)
          return self._fastvks

      def subinplace(self, *args, **kwargs):
          self._fastvks = None
          Model.subinplace(self, *args, **kwargs)

[bqpd]

Yeah, this is probably a good call. I need to make KeyDict and its usage a whole lot faster.

[whoburg]

I think this is partially addressed by #787, so I've changed the title to reflect the remaining issues with KeyDict speed.

[bqpd]

Tried to fix it and could not. :-/
Partially addressed in #861

[bqpd]

Closing this issue, as we're aware that KeyDict is slow...

[whoburg]

I still think this is a big issue. Being aware of it does not mean it is fixed! I think we need to discuss this, perhaps as a design discussion -- without a fast keydict, I don't see how we'll meet our goal of GPkit not creating overhead larger than 100% of the solver time.

[bqpd]

For large SPs I'm not seeing KeyDict usage show up in the profiler anymore! I think it's mostly a problem with GP sweeps at this point?

[whoburg]

I'm interested in several cases, each of which tests a different potential slowness mechanism:

• Large GP single solve (where we've had trouble getting GPkit faster than 100% of the solver time)
• Large SP overall wall time (maybe we can save time by not rebuilding model between SP iterations)
• Large GP sweep (maybe we can save time by not rebuilding model between sweep points)