Fix and allow streamable server cameras

[Baharis]

Context

Currently, Instamatic does not allow cameras running on server to stream images at all. There are two layers of security that prevent this at different levels. From my discussion with @stefsmeets I concluded that the reason behind that is mostly historical/practical. The initial development of Instamatic was done on a relatively slow camera that did not allow streaming, and since then a lot of work was done on fast local cameras that did not require the client-server architecture.

The main goal of this PR is to allow cameras running on server, such as the new instamatic-TEM-emulator, to stream their data via shared memory. In vacuum, this could be achieved by removing only two lines of code, as noted in #140 discussion. However, this introduced another problem - what if due to practical reasons someone actually wanted their camera to be non-streamable? To address this case, this PR also introduces a new camera config parameter, config.camera.streamable that, when set to either True or False, overwrites the default class variable for given camera.

Other than allowing streamable server cameras, this PR introduces a few code base improvements. Firstly, it aligns the code conventions used in the camera client code to ones introduced to microscope files by @viljarjf in #99. Furthermore, it allows cameras to call methods that do not exist in their namespace (but may still be accessible on the server) to reflect an analogous change introduced in #108. It slightly improves some docstrings and type hints.

Development note

Having documented all relevant changes, I decided to share some of my notes on writing this thing, because in the process of adding 3 lines for 3 hours I learned that apparently the Instamatic client-server architecture for cameras is heavily reliant on black magic. Buckle up and grab some popcorn!

instamatic.camera.camera_base:CameraBase is a relatively recent class, introduced to Instamatic by @viljarjf in #91. Acting as the base class of all cameras, it includes a very helpful method load_defaults that populates the camera namespace with the camera config. Therefore, it is incredibly easy to add new instance variable and even overwrite camera class variables - just add key: value to camera.yaml and poof! Suddenly your ctrl.cam object has access to any value desired!

On paper, it works great! However, it acts really strange when observed via the lens of CamClient i.e. when using a server. CamClient overloads __getattr__ in a very ingenious way. The old implementation first determines self._dct (namespace of local camera class) and self._attr_dct, attribute space of the remote camera class. If the local class has attr_name in its namespace, __getattr__ will call server implementation of attr_name but wrap it in local attr_name so that the Exceptions are easy to understand. If attr_name does not exist in local namespace but does remotely, it will be called without wrapping:

    def __getattr__(self, attr_name):
        if attr_name in self._dct:
            wrapped = self._dct[attr_name]
        elif attr_name in self._attr_dct:
            dct = {'attr_name': attr_name}
            return self._eval_dct(dct)
        else:
            raise AttributeError(f'`{self.__class__.__name__}` object has no attribute `{attr_name}`')

        @wraps(wrapped)
        def wrapper(*args, **kwargs):
            dct = {'attr_name': attr_name, 'args': args, 'kwargs': kwargs}
            return self._eval_dct(dct)

        return wrapper

So here is when things get very funny: streamable is a class attribute of every camera class. Therefore, streamable exists in the local class namespace as logged in self._dct. Therefore, whenever camera.streamable is called, __getattr__ first looks it up in the local class and runs wrapped = self._dct[attr_name] which evaluates to... True. Since no other checks are run after that, the bottom wrapping code runs as normal and, by a reason I still fully fail to understand, by adding streamable = False to the config and calling camera.streamable, instead of getting a False, you get a False wrapped in True that must be called to get False! Nota bene this behavior can't be reproduced in an interactive session, as this should be impossible.

Attempts to fix this behavior was very annoying because of recursion and the odd nature of self._dct and self._attr_dct. Ultimately, I managed to fix it by changing the first part of __getattr__ to the following:

    def __getattr__(self, attr_name):
        if attr_name in self._dct:
            if attr_name in object.__getattribute__(self, '_attr_dct'):
                return self._eval_dct({'attr_name': attr_name})
            wrapped = self._dct[attr_name]
        elif attr_name in self._attr_dct:
            dct = {'attr_name': attr_name}
            return self._eval_dct(dct)
        else:
            wrapped = None

The top addition of conditional if attr_name in object.__getattribute__(self, '_attr_dct') checks whether an attribute is present in both local class and on server, in which case it takes its value from the latter. This makes behavior of class variable normal again, in particular self.streamable = True now. You will also see that I prevent the AttributeError from being raised if attr_name is not found, in line with #108. This change is necessary for external server camera to work! AFAIK in #91, get_camera_dimensions() method was moved from individual cameras to the CameraBase - but the scope of CameraBase is not read by CamClient which believes that this method is thus undefined and falsely raises AttributeError whenever you try to display any image from server camera using FakeVideoStream. So this issue is also fixed here.

Major changes

• Cameras run on camera server can now be used to stream data to GUI, as governed by their class' streamable variable;

Minor changes

• CamClient: When calling a method via getattr, try evaluating instead of raising KeyError if it is not found in the interface. This is in line with Tecnai bigfixes, Microscope client documentation and EAFP #108 microscope logic and potentially allows more cameras with "incomplete" local interface template class e.g. the new Instamatic-TEM-emulator to be streamable. Should a method actually be absent, existing CamClient should handle the error.

Code maintanence

• Unused method in microscope Client was removed;
• The microscope and camera factory functions and client interfaces were aligned.

[stefsmeets]

Nice fix! This code still confuses me, but it does the trick. I think it would have been more ingenious to recognize this is basically what an RPC framework does, but I wasn't aware at the time ;-)

[Baharis]

Huh, thanks for the introspection, because thanks to you I just learned what RPC is. It might be a good idea if we ever have time to completely rewrite the interface BUT the current implementation has some benefits, e.g. it is super light-weight. The lightest alternative, xmlrpc, would not suffice as a full replacement, because it supports methods only, not non-callable attributes, and requires the full interface to be known a priori, and both these things are not true for the current design.

Unless I hear objections, I will merge this code to main in ~one to two days. If I have time, I will also propose some paragraph about the instamatic-TEM-emulator this PR enables to the documentation, because I think it is very useful.

[stefsmeets]

Yeah, I looked into some rpc frameworks at some point and then realized it's more effort than it's worth. Like you said, the implementation is quite light and robust enough that it does not cause frequent headaches.