Add proxy protocol support for real client IP logging

[xtrusia]

currently, log shows proxy ip but need actual client ip

This PR suggest to add required module and
configuration method to setup options for Apache2 templates
and options for haproxy templates

#918

[freyes]

you are putting this feature behind a config option , does this mean all charms will be updated to include a new option in their config.yaml? ... can't we just make this the default behavior?, is there any cons on making this the default?

[xtrusia]

you are putting this feature behind a config option , does this mean all charms will be updated to include a new option in their config.yaml? ... can't we just make this the default behavior?, is there any cons on making this the default?

You're right.
I actually thought the same at first, and it seemed fine to just make it the default behavior.

But I had a question during testing. For example, in the keystone charm, there are its own templates for apache2. e.g wsgi-openstack-api.conf and openstack_https_frontend.conf.

I tested this, and it turns out that if a charm includes its own template files, it uses those instead of the ones from charmhelpers.
So even if we make this the default behavior in charmhelpers, we'd still need to update each individual charm that overrides the templates.

Also, fixes would need to be applied across haproxy, apache templates, and modules all at once, since they're all interdependent.
So charmhelpers(sync with charm) and the keystone charm itself(templates update) should be updated together, otherwise things will break I think.

If we adopt conf option, we can split commit to sync with charmhelpers and change each charms.

But yes. making this the default also makes sense to me as well.

[freyes]

On Mon, 2025-03-24 at 17:55 -0700, Seyeong Kim wrote: > you are putting this feature behind a config option , does this mean all charms will be updated > to include a new option in their config.yaml? ... can't we just make this the default behavior?, > is there any cons on making this the default? You're right. I actually thought the same at first, and it seemed fine to just make it the default behavior. But I had a question during testing. For example, in the keystone charm, there are its own templates for apache2. e.g wsgi-openstack-api.conf and openstack_https_frontend.conf. I tested this, and it turns out that if a charm includes its own template files, it uses those instead of the ones from charmhelpers. So even if we make this the default behavior in charmhelpers, we'd still need to update each individual charm that overrides the templates. Also, fixes would need to be applied across haproxy, apache templates, and modules all at once, since they're all interdependent. So charmhelpers(sync with charm) and the keystone charm itself(templates update) should be updated together, otherwise things will break I think. If we adopt conf option, we can split commit to sync with charmhelpers and change each charms. But yes. making this the default also makes sense to me as well. — Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you commented.Message ID: <juju/charm- ***@***.***> xtrusiaxtrusia left a comment (juju/charm-helpers#919) [1] > you are putting this feature behind a config option , does this mean all charms will be updated > to include a new option in their config.yaml? ... can't we just make this the default behavior?, > is there any cons on making this the default? You're right. I actually thought the same at first, and it seemed fine to just make it the default behavior. But I had a question during testing. For example, in the keystone charm, there are its own templates for apache2. e.g wsgi-openstack-api.conf and openstack_https_frontend.conf.

This is true, although not that common. AFAIK charm-keystone is an outlier here rather the norm.

I tested this, and it turns out that if a charm includes its own template files, it uses those instead of the ones from charmhelpers. So even if we make this the default behavior in charmhelpers, we'd still need to update each individual charm that overrides the templates.

we would need the usual "charm-helper sync" dance, yes.

Also, fixes would need to be applied across haproxy, apache templates, and modules all at once, since they're all interdependent. So charmhelpers(sync with charm) and the keystone charm itself(templates update) should be updated together, otherwise things will break I think. If we adopt conf option, we can split commit to sync with charmhelpers and change each charms.

I'm not following you on this one, you could have a charmhelpers sync for keystone and a subsequent patch that modifies the keystone templates, what makes you think things will break? [1] view it on GitHub #919 (comment) [2] unsubscribe https://github.com/notifications/unsubscribe-auth/AAAYLH7VUVJWBGAOXTTFH632WCSRVAVCNFSM6AAAAABZMOX736VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBZG42TSMRRGE

[xtrusia]

@freyes If we can do both at once, no issue.
if haproxy.conf has send-proxy-v2 but apache2 conf doesn't have it (vice versa) it doesn't seem to work. so I mentioned it.

So, would you like me to change it to default instead of adding config option? I'll change the fix tomorrow.

Thanks so much

[freyes]

something we haven't talked about is what happens when you have only side of the connection with the proxy protocol enabled, specifically:

1. haproxy set with send-proxy-v2, but apache doesn't have remoteip enabled
2. haproxy is NOT set with send-proxy-v2, but apache has enabled remoteip

I would assume in the case of (2) nothing happens, and the old behavior where the client ip address is not received by apache, right?, but what about in the case of (1)?

[xtrusia]

something we haven't talked about is what happens when you have only side of the connection with the proxy protocol enabled, specifically:

1. haproxy set with send-proxy-v2, but apache doesn't have remoteip enabled
2. haproxy is NOT set with send-proxy-v2, but apache has enabled remoteip

I would assume in the case of (2) nothing happens, and the old behavior where the client ip address is not received by apache, right?, but what about in the case of (1)?

This is why I mentioned in the beginning they should be fixed at the same time.(for keystone) but sorry my explanation was not enough.
I remember that (1) didn't work. keystone command returned error.
But I'll double check it today and share the result here.

1. 400 error
2. connection aborted

Test

1. deploy keystone and mysql charm
2. openstack endpoint list ( works fine )
3. a2enmod remoteip
4. change configuration files and restart them

Thanks so much.

[xtrusia]

@freyes I'm wondering if you might check the latest comment and have some time to review this issue. Thanks for your review.

[xtrusia]

re-added configuration option since we discussed it in sprint.

[lathiat]

1. I think it would be nice to have a draft PR up for charm-keystone incorporating these changes, so we can review/test the implementation before this is merged. As we may find we later need changes and have to do a second PR. Could you upload such a change in draft mode for review?

It would also be great to explicitly document the following

2. Do both apache2 and haproxy support these exact same modules/configuration options on focal (for focal-yoga) as well as jammy + noble. The requesting environment is Jammy-Yoga so the intent is to backport this feature to at least Yoga. But the same charm has to support both Jammy-Yoga and Focal-Yoga.

3. Do we (now) intend for the config option to be disabled or enabled by default. While avoiding an unexpected change on existing environments is ideal, I can't really think of scenarios where people would NOT want this option enabled - if they are using both haproxy and apache2. I agree it would be ideal to have an option to turn it off in case of a problem. If we are going to go ahead with having it disabled by default, can we document exactly why - is it only abundance of caution, or otherwise what specific scenarios may be broken without it.

4. What happens during either charm upgrades (or just charm config change) when only 1 or 2 units have finished upgrading their config file and restarting. (since you do 1 unit at a time, which hosts both) - in cases where the config option is both off and on. Will it still work, or fail during that time until all units have updated. This may be an issue for managed upgrades where daemon restarts are not being done. When enabled, RemoteIPProxyProtocol requires that clients connect with PROXYv2 by default, so I suspect this won't work.

Lastly, some review notes

5. When deployed in a non-HA configuration, that this is not enabled. I am not sure this is the case currently? Perhaps by default we would only activate this option when haproxy is in use for a HA deployment with the hacluster relation, but the config option could force it on otherwise.

6. When enabled, RemoteIPProxyProtocol requires that clients connect with PROXYv2. We should check whether any charms have NRPE checks or other reasons the port is conneted to with a non-PROXYv2 client that will be broken by this. It may also frustrate us at debug time, when you try to directly query the apache2 server when you are trying to debug problems for example. Perhaps it's desirable to configure mod_remoteip to -only- request PROXYv2 from the specific list of haproxy IP addresses. Seems this may be possible with RemoteIPProxyProtocolExceptions.

7. That other clients cannot speak Proxy V2 and spoof the client source IP. Currently, we don't firewall off apache2 access and many clients of the customer's network can often connect directly to the apache2 port, bypassing haproxy. So we need to prevent this by configuring either a firewall, or using configuration options like RemoteIPTrustedProxy or RemoteIPInternalProxy to limit this access to certain IP addresses - e.g. the IP addresses of the other units haproxy instance.

8. Looking at the apache2 documentation, it seems it ignores RFC1918/Intranet IP addresses unless a specific RemoteIPInternalProxy is configured. I think in practice, most Charmed OpenStack customers will be using such addresses, and so we may need to configure this option specifically otherwise the config option may be mostly useless.

[xtrusia]

It seems that haproxy and apache2 are tightly coupled so other components connect via haproxy only, they don't have to access to apache2 directly by proxyv2.

Testing Notes

1. Test keystone charm

   • master: https://review.opendev.org/c/openstack/charm-keystone/+/955042
   • 2024.1: https://review.opendev.org/c/openstack/charm-keystone/+/962637
   • yoga: https://review.opendev.org/c/openstack/charm-keystone/+/962639

2. RemoteIPProxyProtocolExceptions

   • Adding RemoteIPProxyProtocolExceptions 127.0.0.1 is sufficient, since the charm itself checks connectivity via localhost during deployment.
   • Tested several components with the updated Keystone and no issues observed so far.
   • Plan: test different combinations (focal-yoga, jammy-yoga, etc.) with both older and new config changes.

3. Directive availability

   • Supported since Apache 2.4.31.
   • Focal ships Apache 2.4.41 → supported.
   • Ref: https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html

4. Default value

   • Should be false for compatibility.

5. NRPE tests

   • Focal-yoga with Nagios web UI monitoring
     • Config changes tested (on/off) → no issues.
   • Jammy-yoga with Nagios web UI monitoring
     • Config changes tested (on/off) → no issues.

6. zaza test

   • master
     • Ran zaza test with the patch above (although it doesn’t include a conf test, it at least confirms default conf works).
       • Test result(master): https://pastebin.ubuntu.com/p/zTVXzkRtyX/
     • With test https://github.com/openstack-charmers/zaza-openstack-tests/pull/1328/files
     • Test result(master): https://pastebin.ubuntu.com/p/j2VC75X3qf/
   • 2024.1
     • Test result: https://pastebin.ubuntu.com/p/MW5w8bD4VV/
   • yoga
     • Test result: https://pastebin.ubuntu.com/p/T9J9ZT3Kqy/

7. RemoteIPTrustedProxy 127.0.0.1 doesn't block connection from somewhere else. like below picture. it just checks X-F-F header

RemoteIPTrustedProxy 127.0.0.1 checks only x-forwarded-for. if I use below script(injecting fake client ip), it just pass.
We do need to setup firewall for apache2 port(in this case 4980) as proxyv2 doesn't have any

As above structure in HA, I think security issue should be done by new ticket.
if we secure them strictly, may set up firewall instead of setting up apache2.

#!/usr/bin/env python3
"""
Send an HTTP request with a PROXY protocol v2 header (IPv4/TCP).
Usage:
  ./send_proxy_v2.py <target_ip> <target_port> <fake_client_ip> <fake_client_port>
"""

import argparse
import socket
import struct


def build_proxy_v2_ipv4(fake_src_ip: str, dst_ip: str, fake_src_port: int, dst_port: int) -> bytes:
    # PROXY v2 signature + version/cmd + family/proto + addr length(IPv4=12)
    sig = b"\r\n\r\n\x00\r\nQUIT\n"
    ver_cmd = b"\x21"   # v2 + PROXY
    fam_proto = b"\x11" # INET + STREAM(TCP)
    addr_len = struct.pack("!H", 12)

    src_addr = socket.inet_aton(fake_src_ip)
    dst_addr = socket.inet_aton(dst_ip)
    src_port_b = struct.pack("!H", fake_src_port)
    dst_port_b = struct.pack("!H", dst_port)

    return sig + ver_cmd + fam_proto + addr_len + src_addr + dst_addr + src_port_b + dst_port_b


def main() -> int:
    p = argparse.ArgumentParser(add_help=True)
    p.add_argument("target_ip")
    p.add_argument("target_port", type=int)
    p.add_argument("fake_client_ip")
    p.add_argument("fake_client_port", type=int)
    p.add_argument("--path", default="/", help="HTTP path (default: /)")
    p.add_argument("--host", default=None, help="Host header (default: target_ip)")
    args = p.parse_args()

    host_header = args.host or args.target_ip
    proxy_hdr = build_proxy_v2_ipv4(args.fake_client_ip, args.target_ip, args.fake_client_port, args.target_port)
    http_req = f"GET {args.path} HTTP/1.1\r\nHost: {host_header}\r\nConnection: close\r\n\r\n".encode("ascii")

    payload = proxy_hdr + http_req

    try:
        with socket.create_connection((args.target_ip, args.target_port), timeout=5) as s:
            s.sendall(payload)
            buf = bytearray()
            while True:
                chunk = s.recv(4096)
                if not chunk:
                    break
                buf += chunk
    except Exception as e:
        print(f"Error: {e}")
        return 1

    print(buf.decode("utf-8", errors="replace"))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

[Raven-182]

The switch to mode http in this PR is breaking HTTPS. Previously, HAProxy was explicitly changed to mode tcp in this commit: 30f7b62 to allow TLS passthrough.

HAProxy was using TCP mode to forward encrypted traffic to Apache, where TLS termination is configured.

I verified this against a deployment with this PR applied to the Keystone charm and enabling https:

• Apache is configured with TLS enabled
• HTTPS via HAProxy fails
• HAProxy is no longer passing TLS through
• TLS is not being terminated at HAProxy either

As a result, encrypted traffic is being sent to a frontend expecting plain HTTP which causes HTTPS to fail.
After discussing with @freyes , we want to retain TLS termination at the Apache layer. To preserve the existing architecture, HAProxy should remain in mode tcp

[xtrusia]

It seems that haproxy and apache2 are tightly coupled so other components connect via haproxy only, they don't have to access to apache2 directly by proxyv2.

Testing Notes

1. Test keystone charm

   • master: https://review.opendev.org/c/openstack/charm-keystone/+/955042
   • 2024.1: https://review.opendev.org/c/openstack/charm-keystone/+/962637
   • yoga: https://review.opendev.org/c/openstack/charm-keystone/+/962639

2. RemoteIPProxyProtocolExceptions

   • Adding RemoteIPProxyProtocolExceptions 127.0.0.1 is sufficient, since the charm itself checks connectivity via localhost during deployment.
   • Tested several components with the updated Keystone and no issues observed so far.
   • Plan: test different combinations (focal-yoga, jammy-yoga, etc.) with both older and new config changes.

3. Directive availability

   • Supported since Apache 2.4.31.
   • Focal ships Apache 2.4.41 → supported.
   • Ref: https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html

4. Default value

   • Should be false for compatibility.

5. NRPE tests

   • Focal-yoga with Nagios web UI monitoring

     • Config changes tested (on/off) → no issues.

   • Jammy-yoga with Nagios web UI monitoring

     • Config changes tested (on/off) → no issues.

6. zaza test

   • master

     • Ran zaza test with the patch above (although it doesn’t include a conf test, it at least confirms default conf works).

       • Test result(master): https://pastebin.ubuntu.com/p/zTVXzkRtyX/

     • With test https://github.com/openstack-charmers/zaza-openstack-tests/pull/1328/files

     • Test result(master): https://pastebin.ubuntu.com/p/j2VC75X3qf/

   • 2024.1

     • Test result: https://pastebin.ubuntu.com/p/MW5w8bD4VV/

   • yoga

     • Test result: https://pastebin.ubuntu.com/p/T9J9ZT3Kqy/

7. RemoteIPTrustedProxy 127.0.0.1 doesn't block connection from somewhere else. like below picture. it just checks X-F-F header

RemoteIPTrustedProxy 127.0.0.1 checks only x-forwarded-for. if I use below script(injecting fake client ip), it just pass. We do need to setup firewall for apache2 port(in this case 4980) as proxyv2 doesn't have any As above structure in HA, I think security issue should be done by new ticket. if we secure them strictly, may set up firewall instead of setting up apache2.

#!/usr/bin/env python3
"""
Send an HTTP request with a PROXY protocol v2 header (IPv4/TCP).
Usage:
  ./send_proxy_v2.py <target_ip> <target_port> <fake_client_ip> <fake_client_port>
"""

import argparse
import socket
import struct


def build_proxy_v2_ipv4(fake_src_ip: str, dst_ip: str, fake_src_port: int, dst_port: int) -> bytes:
    # PROXY v2 signature + version/cmd + family/proto + addr length(IPv4=12)
    sig = b"\r\n\r\n\x00\r\nQUIT\n"
    ver_cmd = b"\x21"   # v2 + PROXY
    fam_proto = b"\x11" # INET + STREAM(TCP)
    addr_len = struct.pack("!H", 12)

    src_addr = socket.inet_aton(fake_src_ip)
    dst_addr = socket.inet_aton(dst_ip)
    src_port_b = struct.pack("!H", fake_src_port)
    dst_port_b = struct.pack("!H", dst_port)

    return sig + ver_cmd + fam_proto + addr_len + src_addr + dst_addr + src_port_b + dst_port_b


def main() -> int:
    p = argparse.ArgumentParser(add_help=True)
    p.add_argument("target_ip")
    p.add_argument("target_port", type=int)
    p.add_argument("fake_client_ip")
    p.add_argument("fake_client_port", type=int)
    p.add_argument("--path", default="/", help="HTTP path (default: /)")
    p.add_argument("--host", default=None, help="Host header (default: target_ip)")
    args = p.parse_args()

    host_header = args.host or args.target_ip
    proxy_hdr = build_proxy_v2_ipv4(args.fake_client_ip, args.target_ip, args.fake_client_port, args.target_port)
    http_req = f"GET {args.path} HTTP/1.1\r\nHost: {host_header}\r\nConnection: close\r\n\r\n".encode("ascii")

    payload = proxy_hdr + http_req

    try:
        with socket.create_connection((args.target_ip, args.target_port), timeout=5) as s:
            s.sendall(payload)
            buf = bytearray()
            while True:
                chunk = s.recv(4096)
                if not chunk:
                    break
                buf += chunk
    except Exception as e:
        print(f"Error: {e}")
        return 1

    print(buf.decode("utf-8", errors="replace"))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The switch to mode http in this PR is breaking HTTPS. Previously, HAProxy was explicitly changed to mode tcp in this commit: 30f7b62 to allow TLS passthrough.

HAProxy was using TCP mode to forward encrypted traffic to Apache, where TLS termination is configured.

I verified this against a deployment with this PR applied to the Keystone charm and enabling https:

• Apache is configured with TLS enabled
• HTTPS via HAProxy fails
• HAProxy is no longer passing TLS through
• TLS is not being terminated at HAProxy either

As a result, encrypted traffic is being sent to a frontend expecting plain HTTP which causes HTTPS to fail. After discussing with @freyes , we want to retain TLS termination at the Apache layer. To preserve the existing architecture, HAProxy should remain in mode tcp

@Raven-182 Thanks for the review. I originally went with PROXY Protocol v2 for client IP forwarding, but switched direction after noticing that a rolling upgrade in HA setups could cause a brief window where upgraded and non-upgraded units can't work as expected — since both ends(haproxy and apache in different units) need to speak the same protocol. Would love to hear your thoughts on that if we can fix the issue by using proxyv2. Thanks.

7d887dc

[freyes]

@xtrusia , thanks for putting al this together, and @Raven-182 for reviewing and testing this.

So this is my take is:

1. New config option that defaults to False, to turn on/off send-proxy-v2, this will prevent that during upgrades there is a blip, since this is something we want to backport, it's important we mitigate the risk.
2. Since send-proxy-v2 support tcp mode, the ssl terminate will still happen at Apache and not in haproxy.
3. when the user does juju config <app> enable-send-proxy-v2=True, the right config is rendered and haproxy/apache2 get restarted.

This should address all the concerns and unblock this PR(?)

[Raven-182]

Hi @xtrusia, I’ve been testing this and verified that it works with the neutron-api charm when using TCP mode with PROXY protocol.

I’m currently seeing some failing health checks, so I’m investigating the cause. I’ll propose a follow-up change for this once I have a fix.

[xtrusia]

Hi @xtrusia, I’ve been testing this and verified that it works with the neutron-api charm when using TCP mode with PROXY protocol.

I’m currently seeing some failing health checks, so I’m investigating the cause. I’ll propose a follow-up change for this once I have a fix.

hello. Thanks! could you please share your test bundle if you have? i would like to check them as well. i tested it with keystone(also had its own templates) as the customer pointed out keystone. thanks

[Raven-182]

Hi @xtrusia, I’ve been testing this and verified that it works with the neutron-api charm when using TCP mode with PROXY protocol.
I’m currently seeing some failing health checks, so I’m investigating the cause. I’ll propose a follow-up change for this once I have a fix.

hello. Thanks! could you please share your test bundle if you have? i would like to check them as well. i tested it with keystone(also had its own templates) as the customer pointed out keystone. thanks

The bundle I've been using for testing: neutron-api-ha.yaml
Note that TLS needs to be enabled for Apache to translate the proxy header. With plain http, Apache is not involved and requests go directly to neutron-server

[Raven-182]

@xtrusia I tested this with the neutron-api charm with TLS enabled. Without the check keyword on the server lines, healthchecks were being skipped. After adding check, the healthchecks started, but they are failing.

Switching send-proxy-v2 (PROXYv2) to send-proxy (PROXYv1) resolves the issue. I haven’t been able to pinpoint the exact cause, do you know if there’s a specific reason PROXYv2 was chosen? Functionally, the two seem equivalent in this context.

[xtrusia]

@xtrusia I tested this with the neutron-api charm with TLS enabled. Without the check keyword on the server lines, healthchecks were being skipped. After adding check, the healthchecks started, but they are failing.

Switching send-proxy-v2 (PROXYv2) to send-proxy (PROXYv1) resolves the issue. I haven’t been able to pinpoint the exact cause, do you know if there’s a specific reason PROXYv2 was chosen? Functionally, the two seem equivalent in this context.

This is a really good point. I haven't checked v1 yet.

I think we may need to add some tests based on your findings.

I haven't had a chance to run your bundle yet, but I plan to do so soon.
Could you share the steps you used to set up TLS with your bundle?

If you can provide more detailed steps, I can quickly follow your setup
and help validate and support your testing.

I'm curious if health check with v2 show "unsupported command 20" ?

Thank you so much!

[Raven-182]

Yes, with PROXYv2 enabled the logs did show "unsupported command 20" for healthchecks.
Here are the steps you can use to reproduce:

1. After applying your patch to neutron-api charm and deploying the bundle, set up the VIP config option on neutron-api
   juju config neutron-api VIP=<VIP>
2. Use OpenSSL to generate self-signed certificate and key
   openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes -subj "/CN=<VIP>"
3. Set the certificate and key via charm config
   juju config neutron-api ssl_cert="$(base64 -w 0 cert.pem)"
juju config neutron-api ssl_key="$(base64 -w 0 key.pem)"
4. Wait for services to settle and verify https is enabled
   curl -k https://<VIP>:9696/ should return a valid response
5. enable haproxy-enable-proxyv2 config option

You will notice that healthchecks are not being performed, and you will need to add the check keyword on the server lines.

[xtrusia]

Yes, with PROXYv2 enabled the logs did show "unsupported command 20" for healthchecks. Here are the steps you can use to reproduce:

1. After applying your patch to neutron-api charm and deploying the bundle, set up the VIP config option on neutron-api
   juju config neutron-api VIP=<VIP>
2. Use OpenSSL to generate self-signed certificate and key
   openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes -subj "/CN=<VIP>"
3. Set the certificate and key via charm config
   juju config neutron-api ssl_cert="$(base64 -w 0 cert.pem)"
juju config neutron-api ssl_key="$(base64 -w 0 key.pem)"
4. Wait for services to settle and verify https is enabled
   curl -k https://<VIP>:9696/ should return a valid response
5. enable haproxy-enable-proxyv2 config option

You will notice that healthchecks are not being performed, and you will need to add the check keyword on the server lines.

Thanks, I uploaded patch with proxyv1 instead of proxyv2,
I researched remoteip a bit more and found out actually remoteip doesn't handle proxyv2 check(0x20, although spec has it) so the issue is there.
Thank you again for pointing out it.

[Raven-182]

Thanks @xtrusia . With this change, the proposed fix looks good to me.

[freyes]

minor change needed

[freyes]

we need a patch like this

diff --git a/constraints.txt b/constraints.txt
new file mode 100644
index 00000000..78c6a369
--- /dev/null
+++ b/constraints.txt
@@ -0,0 +1 @@
+setuptools<82.0
diff --git a/tox.ini b/tox.ini
index 87d736b5..75319c2c 100644
--- a/tox.ini
+++ b/tox.ini
@@ -15,31 +15,46 @@ setenv = VIRTUAL_ENV={envdir}
          PYTHONHASHSEED=0
 passenv = HOME,TERM
 commands = nose2 {posargs} --with-coverage -s tests/
-deps = -r{toxinidir}/test-requirements.txt
+deps =
+    -c{toxinidir}/constraints.txt
+    -r{toxinidir}/test-requirements.txt
+
 
 [testenv:py3]
 basepython = python3
-deps = -r{toxinidir}/test-requirements.txt
+deps =
+    -c{toxinidir}/constraints.txt
+    -r{toxinidir}/test-requirements.txt
 
 [testenv:py38]
 basepython = python3.8
-deps = -r{toxinidir}/test-requirements.txt
+deps =
+    -c{toxinidir}/constraints.txt
+    -r{toxinidir}/test-requirements.txt
 
 [testenv:py310]
 basepython = python3.10
-deps = -r{toxinidir}/test-requirements.txt
+deps =
+    -c{toxinidir}/constraints.txt
+    -r{toxinidir}/test-requirements.txt
 
 [testenv:py311]
 basepython = python3.11
-deps = -r{toxinidir}/test-requirements.txt
+deps =
+    -c{toxinidir}/constraints.txt
+    -r{toxinidir}/test-requirements.txt
 
 [testenv:py312]
 basepython = python3.12
-deps = -r{toxinidir}/test-requirements.txt
+deps =
+    -c{toxinidir}/constraints.txt
+    -r{toxinidir}/test-requirements.txt
 
 [testenv:pep8]
 basepython = python3
-deps = -r{toxinidir}/test-requirements.txt
+deps =
+    -c{toxinidir}/constraints.txt
+    -r{toxinidir}/test-requirements.txt
 commands = flake8 -v {posargs} charmhelpers tests tools
 
 [flake8]

[freyes]

LGTM. Waiting on the CI