readout.py: KeyError: 'MX device not found'

[two9ers]

I'm getting the following error when running readout.py
Apologies for formatting issues, having a bit of trouble with that also...

MATE emulator (MX)
Traceback (most recent call last)
File "readout.py", line 13, in
port = bus.find_device(MateNET.DEVICE_MX)
File "/home/pi/pymate-master/pymate/matenet/matenet.py", line 213, in find_device
raise KeyError('%s device not found' % MateNET.DEVICE_TYPES[device_type])
KeyError: 'MX device not found'

I added some print commands and set port 4:

bus = MateNET(SERIAL_PORT)
print "bus", bus

#port = bus.find_device(MateNET.DEVICE_MX)
port = 4
print "port", port

mate = MateMXDevice(bus, port)
print "mate", mate

mate.scan()

print "Revision:", mate.revision

New error shows up:

MATE emulator (MX)
bus <pymate.matenet.matenet.MateNET object at 0xb669e470>
port 4
mate <pymate.matenet.mx.MateMXDevice object at 0xb669e830>
Revision: 44451.173.000
Getting log page... (day:-1)
MX Log Page:
Day: -0
0Ah 361.6kWh
1Vpk 0.1Apk 0.000kWpk
Min: 0.0V Max: 1.5V
Absorb: 0min Float: 0min

Status:
Traceback (most recent call last):
File "readout.py", line 34, in
status = mate.get_status()
File "/home/pi/pymate-master/pymate/matenet/mx.py", line 167, in get_status
return MXStatusPacket.from_buffer(resp)
File "/home/pi/pymate-master/pymate/matenet/mx.py", line 50, in from_buffer
values = cls.fmt.unpack(data)
struct.error: unpack requires a string argument of length 13

[jorticus]

Yeah I think this is probably related to issue #22.

Can you enable full debug logging? Just add the following to the top of readout.py:

import logging
log = logging.getLogger('mate')
log.setLevel(logging.DEBUG)
log.addHandler(logging.StreamHandler())

I expect you'll see something similar where there are extra 00's at the start of the packet.

[jorticus]

Could you try the latest changes in branch issue-22?
I've added a more thorough workaround for the issue I believe you are encountering.
Please also enable the debug logs in addition to this.

[two9ers]

The latest from branch issue-22:

~/pymate-issue-22 $ python readout.py 
MATE emulator (MX)
Send [Port0, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 00
TX: [0] 02 00 00 00 00 00 02
RX: 00 00 01 00 01
RETRY
TX: [1] 00
TX: [0] 02 00 00 00 00 00 02
RX: 00 00 01 00 01
RETRY
TX: [1] 00
TX: [0] 02 00 00 00 00 00 02
RX: 00 00 01 00 01
Traceback (most recent call last):
    File "readout.py", line 17, in <module>
    port = bus.find_device(MateNET.DEVICE_MX)
  File "/home/pi/pymate-issue-22/pymate/matenet/matenet.py", line 209, in find_device
    dtype = self.query(0x00, port=i)
  File "/home/pi/pymate-issue-22/pymate/matenet/matenet.py", line 134, in query
    resp = self.send(MateNET.TYPE_QUERY, addr=reg, param=param, port=port, response_len=MateNET.QueryResponse.size)
  File "/home/pi/pymate-issue-22/pymate/matenet/matenet.py", line 87, in send
    rxbuf = self.port.recv(response_len)
  File "/home/pi/pymate-issue-22/pymate/matenet/matenet_ser.py", line 171, in recv
    return MateNETSerial._parse_packet(rawdata, expected_len)
  File "/home/pi/pymate-issue-22/pymate/matenet/matenet_ser.py", line 119, in _parse_packet
    if self.TRIM_LARGE_PACKETS:
NameError: global name 'self' is not defined
~/pymate-issue-22 $

[jorticus]

Ooops, just pushed a fix, please pull & try again. I'm not able to test the code right now (don't have the hardware set up)

[two9ers]

New error with the latest:

pi@datapi:~/pymate-issue-22 $ python readout.py 
MATE emulator (MX)
Send [Port0, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 00
TX: [0] 02 00 00 00 00 00 02
RX: 00 00 01 00 01
RETRY
TX: [1] 00
TX: [0] 02 00 00 00 00 00 02
RX: 00 00 01 00 01
RETRY
TX: [1] 00
TX: [0] 02 00 00 00 00 00 02
RX: 00 00 01 00 01
Traceback (most recent call last):
  File "readout.py", line 17, in <module>
    port = bus.find_device(MateNET.DEVICE_MX)
  File "/home/pi/pymate-issue-22/pymate/matenet/matenet.py", line 209, in find_device
    dtype = self.query(0x00, port=i)
  File "/home/pi/pymate-issue-22/pymate/matenet/matenet.py", line 134, in query
    resp = self.send(MateNET.TYPE_QUERY, addr=reg, param=param, port=port, response_len=MateNET.QueryResponse.size)
  File "/home/pi/pymate-issue-22/pymate/matenet/matenet.py", line 87, in send
    rxbuf = self.port.recv(response_len)
  File "/home/pi/pymate-issue-22/pymate/matenet/matenet_ser.py", line 170, in recv
    return MateNETSerial._parse_packet(rawdata, expected_len)
  File "/home/pi/pymate-issue-22/pymate/matenet/matenet_ser.py", line 112, in _parse_packet
    raise RuntimeError("Error receiving mate packet - Received packet too small (%d bytes)" % (len(data)))
RuntimeError: Error receiving mate packet - Received packet too small (2 bytes)

[jorticus]

Thank you, getting closer... I forgot to account for additional bytes in the response packet.
Please try the latest changes (0a6be1c).

[two9ers]

Latest results. I turned off debugging as scan.py completes successfuly.

pi@datapi:~/pymate-issue-22 $ python scan.py 
MATE Bus Scan
Port0: Hub (Rev: 001.000.001)
Port1: FX (Rev: 000.000.000)
Port2: FX (Rev: 000.000.000)
Port3: MX (Rev: 11683.045.000)
Port4: MX (Rev: 11691.045.000)

Error from readout.py, Not sure why it goes past port 4 as I have a 4 port hub and in scan.py I have changed:
"if dtype == MateNET.DEVICE_HUB:
for i in range(1,5):"

pi@datapi:~/pymate-issue-22 $ python readout.py 
MATE emulator (MX)
Send [Port0, Type=0x02, Addr=0x0000, Param=0x0000]    
TX: [1] 00
TX: [0] 02 00 00 00 00 00 02
RX: 00 00 01 00 01
Send [Port1, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 01
TX: [0] 02 00 00 00 00 00 03
RX: 02 00 02 00 04
Send [Port2, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 02
TX: [0] 02 00 00 00 00 00 04
RX: 02 00 02 00 04
Send [Port3, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 03
TX: [0] 02 00 00 00 00 00 05
RX: 00 00 02 11 03 00 16
Send [Port4, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 04
TX: [0] 02 00 00 00 00 00 06
RX: 00 00 02 11 03 00 16
Send [Port5, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 05
TX: [0] 02 00 00 00 00 00 07
RETRY
TX: [1] 05
TX: [0] 02 00 00 00 00 00 07
RETRY
TX: [1] 05
TX: [0] 02 00 00 00 00 00 07
RETRY
Send [Port6, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 06
TX: [0] 02 00 00 00 00 00 08
RETRY
TX: [1] 06
TX: [0] 02 00 00 00 00 00 08
RETRY
TX: [1] 06
TX: [0] 02 00 00 00 00 00 08
RETRY
Send [Port7, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 07
TX: [0] 02 00 00 00 00 00 09
RETRY
TX: [1] 07
TX: [0] 02 00 00 00 00 00 09
RETRY
TX: [1] 07
TX: [0] 02 00 00 00 00 00 09
RETRY
Send [Port8, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 08
TX: [0] 02 00 00 00 00 00 0a
RETRY
TX: [1] 08
TX: [0] 02 00 00 00 00 00 0a
RETRY
TX: [1] 08
TX: [0] 02 00 00 00 00 00 0a
RETRY
Send [Port9, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 09
TX: [0] 02 00 00 00 00 00 0b
RETRY
TX: [1] 09
TX: [0] 02 00 00 00 00 00 0b
RETRY
TX: [1] 09
TX: [0] 02 00 00 00 00 00 0b
RETRY
Traceback (most recent call last):
  File "readout.py", line 17, in <module>
    port = bus.find_device(MateNET.DEVICE_MX)
  File "/home/pi/pymate-issue-22/pymate/matenet/matenet.py", line 219, in find_device
    raise KeyError('%s device not found' % MateNET.DEVICE_TYPES[device_type])
KeyError: 'MX device not found'

If I specify port 4:
"# Find an MX device on the bus
#port = bus.find_device(MateNET.DEVICE_MX)
port = 4"

pi@datapi:~/pymate-issue-22 $ python readout.py 
MATE emulator (MX)
Send [Port4, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 04
TX: [0] 02 00 00 00 00 00 06
RX: 00 00 02 11 03 00 16
Send [Port4, Type=0x02, Addr=0x0002, Param=0x0000]
TX: [1] 04
TX: [0] 02 00 02 00 00 00 08
RX: 00 00 02 ad ab 01 5a
Send [Port4, Type=0x02, Addr=0x0003, Param=0x0000]
TX: [1] 04
TX: [0] 02 00 03 00 00 00 09
RX: 00 00 02 00 ad 00 af
Send [Port4, Type=0x02, Addr=0x0004, Param=0x0000]
TX: [1] 04
TX: [0] 02 00 04 00 00 00 0a
RX: 00 00 02 00 00 00 02
Revision: 44459.173.000
Getting log page... (day:-1)
Traceback (most recent call last):
  File "readout.py", line 30, in <module>
    logpage = mate.get_logpage(-1)
  File "/home/pi/pymate-issue-22/pymate/matenet/mx.py", line 177, in get_logpage
    resp = self.send(MateNET.TYPE_LOG, addr=0, param=-day, response_len=MXLogPagePacket.size)
TypeError: send() got an unexpected keyword argument 'response_len'

[jorticus]

For the first error - readout.py calls find_device() (see matenet.py L197) which scans each port 0-9, then returns when it finds the first connected MX. It seems I missed a fix for a bug that was found a while ago, so I've pushed that change to the branch. (Some MX devices have something in the upper byte for the scan result, which throws off the ID detection)

If you know your network won't change, you can hard-code the ports for each device and skip the call to find_device(). You'll probably need to do that anyway since you have two MX units.

The second error was a simple python mistake, and I've pushed a new change to the branch.

Thanks!

[two9ers]

I really appreciate your help on this, and wish I could set up a port forward so you could ssh into the datapi, but I'm off-grid and working with phone data. Can't wait for Starlink!

The latest. I have skipped find_device(), specified port = 3, and requested logpage = mate.get_logpage(-2). The returned logpage is not accurate. And then the get_status error...

pi@datapi:~/pymate-issue-22 $ python readout.py 
MATE emulator (MX)
Send [Port3, Type=0x02, Addr=0x0000, Param=0x0000]
TX: [1] 03
TX: [0] 02 00 00 00 00 00 05
RX: 00 00 02 11 03 00 16
Send [Port3, Type=0x02, Addr=0x0002, Param=0x0000]
TX: [1] 03
TX: [0] 02 00 02 00 00 00 07
RX: 00 00 02 0c 29 00 37
Send [Port3, Type=0x02, Addr=0x0003, Param=0x0000]
TX: [1] 03
TX: [0] 02 00 03 00 00 00 08
RX: 00 00 02 00 0c 00 0e
Send [Port3, Type=0x02, Addr=0x0004, Param=0x0000]
TX: [1] 03
TX: [0] 02 00 04 00 00 00 09
RX: 00 00 02 00 00 00 02
Revision: 3113.012.000
Getting log page... (day:-2)
Send [Port3, Type=0x16, Addr=0x0000, Param=0x0002]
TX: [1] 03
TX: [0] 16 00 00 00 02 00 1b
RX: 00 00 2a 00 99 00 58 00 00 00 00 00 00 00 00 01 1b
MX Log Page:
    Day: -0
    0Ah 244.8kWh
    0Vpk 51.2Apk 0.000kWpk
    Min: 0.0V Max: 1.0V
    Absorb: 88min Float: 0min

Status:
Send [Port3, Type=0x04, Addr=0x0001, Param=0x0000]
TX: [1] 03
TX: [0] 04 00 01 00 00 00 08
RX: 00 03 00 80 80 00 8f 00 00 00 00 00 f8 02 da 03 66
Traceback (most recent call last):
  File "readout.py", line 35, in <module>
    status = mate.get_status()
  File "/home/pi/pymate-issue-22/pymate/matenet/mx.py", line 169, in get_status
    return MXStatusPacket.from_buffer(resp)
  File "/home/pi/pymate-issue-22/pymate/matenet/mx.py", line 51, in from_buffer
    values = cls.fmt.unpack(data)
struct.error: unpack requires a string argument of length 13

[jorticus]

Hmm if your logpage is not correct, then maybe we're on the wrong path. It might just be that your serial link is not 100% reliable or can't handle all the devices connected to the hub (I only have 1 MX & FX device to test with).

It will be tricky to debug further without an oscilloscope or logic analyzer to prove that pymate is receiving the data correctly. It could be as simple as the opto-isolator circuit having wrong resistor values (corrupting incoming bytes).

Do you get a correct logpage if you connect directly to one of your MX units?