Correct way to maintain a connection to a PLC.

[Leery2495]

Hi,
Just a quick question if possible. In the examples provided when connecting to the processor we use with PLC() as comm:,
I am writing an application that basically reads tags provided and displays them. The application has connection to multiple PLC's.
What is the correct way to store the connection to the PLC so that I can recall it at any time.
The reason I want to do this is that opening and closing the connection is resulting in the EXCP 0300 error on my processor cards.
Any help is appreciated.

[Leery2495]

I seen this post but dont fully understand what it is saying.
#9 (comment)

[dmroeder]

When you first read or write a tag, the connection is established. If you have not closed the connection (calling close directly or indirectly), you have about a minute and a half to make another read or write before the PLC will drop the connection. If you keep making reads or writes before the PLC closes the connection, the same connection will be used.

That particular exception is the card not handling sheer volume of connections. ENBT's didn't handle flushing connections very well, so if you open/close connections too quickly, it would eventually cause an error in the card. This is what was happening in issue #9

Consider:

import pylogix
for i in range(10000):
    with pylogix.PLC("192.168.1.10") as comm:
        ret = comm.Read('MyTag')

Each iteration of the loop will open a connection, read the tag, close the connection. 10,000 times. This not what you want to do, it would be better to put the loop inside the pylogix context so that only one connection is made:

import pylogix
with pylogix.PLC("192.168.1.10") as comm:
    for i in range(10000):
        ret = comm.Read('MyTag')

[Leery2495]

When you first read or write a tag, the connection is established. If you have not closed the connection (calling close directly or indirectly), you have about a minute and a half to make another read or write before the PLC will drop the connection. If you keep making reads or writes before the PLC closes the connection, the same connection will be used.

That particular exception is the card not handling sheer volume of connections. ENBT's didn't handle flushing connections very well, so if you open/close connections too quickly, it would eventually cause an error in the card. This is what was happening in issue #9

Consider:

import pylogix
for i in range(10000):
    with pylogix.PLC("192.168.1.10") as comm:
        ret = comm.Read('MyTag')

Each iteration of the loop will open a connection, read the tag, close the connection. 10,000 times. This not what you want to do, it would be better to put the loop inside the pylogix context so that only one connection is made:

import pylogix
with pylogix.PLC("192.168.1.10") as comm:
    for i in range(10000):
        ret = comm.Read('MyTag')

Thanks @dmroeder
So because I'm going back and forward between different PLC's. What is the best way of ensuring that the connection is still open? Can I store the comm object in memory somehow so that a different thread can pick it up and use it?

[kyle-github]

you have about a minute and a half to make another read or write before the PLC will drop the connection.

Oops. Hit comment by accident.

I have had very poor luck waiting that long. In my testing I see relatively consistent drops after about five seconds. I must be setting up the connection differently than you.

I ended up writing code within my library to automatically close the connection, nicely, after five seconds of idle time and reconnect automatically when read our write are called.

[dmroeder]

you have about a minute and a half to make another read or write before the PLC will drop the connection.

Oops. Hit comment by accident.

I have had very poor luck waiting that long. In my testing I see relatively consistent drops after about five seconds. I must be setting up the connection differently than you.

I ended up writing code within my library to automatically close the connection, nicely, after five seconds of idle time and reconnect automatically when read our write are called.

It might be that not all controllers and/or firmware revisions are equal. I tested on a CompactLogix and the connection was flushed after about a minute and a half. Certainly never experienced 5 seconds, as long as ForwardClose and/or UnregisterSession is not called. Maybe a controller is more aggressive about connections the more connections it currently has. It's been a while since I've looked into this but the connection parameters in the forward open might matter as well.

@Leery2495 I've found that the best result when using threads is for each thread to have it's own instance of pylogix, rather than share it with each thread.

[evaldes2015]

My experience is also that the connections drop at about 5 seconds.

In every application I've built that talks to Rockwell PLCs I always "heartbeat" the PLC if more than 2 seconds have passed with no activity on the connection.

[ottowayi]

The timeout is set as part of the forward open, so based on the _buildCIPForwardOpen method the timeout should be about 14 seconds.
CIPPriority is 0x0A which means the tick time is 1024ms and CIPTimeoutTicks is 0x0E or 14 ticks, so the timeout values is 1024 * 14 = 14336ms or ~14 seconds. The unconnected send method also uses the same values, so it should be the same for unconnected messages as well.

Any message can work as a heartbeat to maintain the connection, in cases where I've needed it simple things like reading the PLC time or getting the program name is what I've used.

[kyle-github]

I went with a connection close instead of a keep-alive. I found that my own code either:

1. Read frequently and thus maintained its own keep-alives.
2. Read/wrote very infrequently and thus was just hogging PLC connection resources.

I didn't seem to have two many cases in between. A very informal question to my users at the time showed similar results. In the case that reads/writes are done infrequently, the time overhead of setting up a new connection is generally OK.

But perhaps I should add a keep-alive as an option.

@Leery2495 I've found that the best result when using threads is for each thread to have it's own instance of pylogix, rather than share it with each thread.

Does each instance have its own connection? If so, I'd be a bit careful. At least one of my old PLCs only supports 32 connections. The environments my library grew up in were fairly heavily networked and PLCs often had several open connections from other PLCs and other systems. Thus, I try pretty hard to minimize the number of connections. Should make that easier to manage though. Hmm, going to file myself an enhancement ticket.

[TheFern2]

I'm all for a connection keep alive option for pylogix as well, I think we've had a fair share of the same question around connection timeout. I think we can just read a system tag or clock on a background thread, no?

[kyle-github]

I should mention that I did definitely see a case where allowing the PLC to drop the connections itself was a problem. This was with a very old ENBT in a ControlLogix (perhaps they fixed this now) and it took so long for the connection resources to get cleaned up in the ENBT that I was able to run out of connection resources simply by waiting 30 seconds and reconnecting without explicitly disconnecting. I did that repeatedly just to see what would happen and the ENBT locked up. I have a ControlLogix with a L81 CPU and that definitely cleans up faster than that.

In general I strongly suggest making sure you are careful about cleaning up as the PLC might not be very fast or efficient.

[Leery2495]

Thanks for all the input everyone.
I should clarify. As it stands i'm forming a connection once every ten seconds, I then read all the tags I need to and close the connection. In my view this should be making one or two connections only in the timeout window. But for some reason my module is still faulting with the EXCP 0300 error. The reads themselves are celery driven as part of a bigger application, therefore each read may be done by any worker.
I am doing the same thing on another system with no issues but with this I am getting the fault roughly once every two days.
Does anybody know a better way for me to debug this as it sounds like my current method may not be the issue.
Thanks again.

[TheFern2]

The quick and dirty way would be to create one connection object per plc, and do not call close, if you have a cleanup function like ctrl + c sigint or something along those lines you can close connections there. Then create a keep alive function to read a dummy tag just to keep the connection alive. That should hopefully prevent the error from happening on this ent card.

What code are you currently using?

[dmroeder]

This exception is specific to the ENBT, as far as I'm aware. I'm guessing that the other systems that seem to work fine are not 1756-ENBT, or they are of a different firmware revision. Rockwell recommends flashing the module to 6.006 or higher as it addresses issues regarding this exception. The ENBT's web page gives some good information regarding the number of connections, loading and other information. I can help you analyze some screenshots of the page. You can email them to me if you are more comfortable with that.

Edit: I see, exactly what @kyle-github was talking about.

[dmroeder]

@Leery2495 I found an ENBT to test against. Opening/closing connections doesn't seem to be the problem, opening new connections without closing connections is a bigger problem. Of course sharing instances between threads can be an issue too.

I'd verify in wireshark that you are not accidentally opening a new connection with each read, if you are, and you don't have an easy way to prevent that, then make sure you close it.

A simple example, you can quickly open too many connections by doing something like this:

for i in range(1000):
    comm = pylogix.PLC("192.168.1.9")
    ret = comm.Read("ProductPointer")
    print(ret.Status)

[Leery2495]

@shared task
def machine_scan():
    processors = Processors.objects.filter(enabled = True)
    for p in processors:
        machines = Machine.objects.filter(machine_processor = p)
        with PLC() as comm:
            comm.IPAddress = p.processor_ipaddress
            comm.ProcessorSlot = p.processor_slot
            if p.processor_routing:
                comm.Route = literal_eval(p.processor_routing)
            for m in machines:
                try:
                    m.scan(comm)
        comm.Close()

Sorry it took so long to get back to you. Only back in the office today.
This is the code that is causing me the issues. the m.scan() function is just a list of all the tags I wish to read from each machine and where to put the results so I didnt think including it was relevant. From what you have said I am wondering if because i'm setting the IP route for each connection it is creating a new connection for every read. Perhaps I need something that checks if the comm config is the same and if so doesnt set it again.
And to clarify both the card that has no issues and the one that does are same card with same firmware revision so definitely something i'm doing wrong in the code.

Thanks again.