Enhancement Proposal: GET linked invoice/ACK metadata

[monkeypants]

Further to conversation about #5, suggest we consider enhancement to 1.0 version of the protocol that allows access to data about the state of an invoice/receipt by utilising gateway-generated GUIDs and GET verbs.

As an Australian Business
I need the ability to automatically monitor the status of my invoices
so that I have timely, low cost information about my accounts receivable

For example, if successful submission (response code 200) of POST /invoices/ returned data containing a gateway submission GUID, then GET /invoices/<GUID>/ (at the same gateway) could return data about the state of the submission. (subsequent analysis required - what is the state transition model of a submission? it's it more complicated than "unacknowledged" or "acknowledged"?)

As an Australian Business
I need the ability to link to the status of my invoices
So that I can chose to share access to that information with Interested Parties

and also:

As an Interested Party
I need the ability to monitor the status of 3rd party invoices
So that I can inform the management of my counter-party risk

This implies that the data returned from POST /invoices/ has two properties:

• Acknowledgement metadata (might be a link to the receipt acknowledgement URL)
• Some kind of signature that can be used to verify claims about the invoice that was submitted

For example, an Australian Business might tell their Financier (Interested Party) "I sent this invoice, and here is the gateway endpoint for the submission", and the financier would be able to verify that claim (of submission - same date, amount, etc) and monitor it's status (hmm, queried/disputed). Financiers making use of this information may be able to provide more price-competitive credit through a combination of lower operating costs (automation) and more precise risk management.

Another example might be a debt collection service that is sent verifiable invoices / status endpoints, and who uses them (in combination with payment data) to drive an automated debt recovery protocol.

As an Australian Business
I need the ability to prove an invoice payload corresponds to an invoice GUID
So that interested parties can verify invoice status information that I have shared with them

Is this all-or-nothing (one signature for the whole invoice), or would an Australian Business want to share some but not all data from a particular invoice with a third party who then needs to verify it?

Similarly, a successful POST /responses/ (or whatever the noun is, it's still not specified in the spec) should return a data containing a GUID, such that GET /responses/<GUID>/ will describe the state of the acknowledgement. This may or may not be a duplication of the data available in the GET /invoices/<GUID> endpoint. My preference is that the /invoice/ endpoint contains a link to the relevant /responses/ endpoint (for better cache performance of GET /invoices/ subsequent to transitioning out of null-acknowledgement state), but either could work I think.

Again, modelling the state machine of acknowledgements would be an interesting exercise - are there different stages that could be mapped to standard accounting practice? For example, is an invoice first mechanically acknowledged (valid submission received by gateway, forwarded to business), then some kind of human acknowledgement (received by business but no comment about liability), then some kind of affirmation (or dispute - indicate intention to pay/not pay)? Perhaps even a "the cheque's in the mail" assertion...

Note: I'm assuming GUID is essentially random and that /invoice/ and /response/ are different values of GUID, even if the response corresponds to the invoice.

If an invoice has been acknowledged, then the GET /invoices/<GUID> should contain a URL for the GET /responses/<GUID>. In other words, an invoice should link to it's response (if any) but not necessarily the other way round.

[monkeypants]

How should ack status get updated?

The financial system of the Australian Business must be authenticated/authorised to the Gateway where acks get posted. So that means we could accept PUT /responses/<GUID> too.

Alternatives depend on the design of the ACK state machine. If there is a reasonably linear, irreversible process where states can be skipped, then there are other options to PUT. For example, clobbering state with POST /responses/ could work, however this would create new GUIDs and require an update in two places (old_ack superceded-by new_ack, plus update the invoice acknowledged-by ack link). POST chain would preserve history but fowel caches. PUT chain would protect caches (and save on writes) but not necessarily preserve history.

Suggest PUT updates to the ACK with an embedded state-change journal that is returned in the body of the GET response.

[monkeypants]

About signature verification of invoice parameters: Maybe GET /invoices/<GUID>/ should return:

• URL of the ack status endpoint (if present - as above)
• signature of the ACK url (if present) by the gateway
• some random value
• SHA hash of every field in the invoice, salted by the random value
• signature of (random value, collection of hashes) by the gateway.

This way, the invoice data is only POSTED once (by the seller) and PUT once (transition out of null-acknowledgement state). And PUT is tightly constrained to the null fields, signed hashes are imutable.

[monkeypants]

...Maybe GET /invoices// should return:

• ...
• some random value
• SHA hash of every field in the invoice, salted by the random value

sorry, that's silly. Salt with the GUID.

[monkeypants]

As an Australian Business
I need the ability to prove an invoice payload corresponds to an invoice GUID
So that interested parties can verify invoice status information that I have shared with them

Is this all-or-nothing (one signature for the whole invoice), or would an Australian Business want to share some but not all data from a particular invoice with a third party who then needs to verify it?

Maybe we should chunk it up in a way that is congruent with the UBL objects/collections.
OrderReference, AccountingSupplierParty, AccountingCustomerParty etc.

What I mean is GET /invoices/<GUID> returns:

• response URL
• url + hash of OrderReference,
• url + hash of AccountingSupplierParty
• ...etc, for each object/collection chunk submitted (POST /invoices/)

Where url of each object is optional (but not hash - if object present then must have hash), and if a url is present then presumably it requires authentication and appropriate authority to access. So, if an Australian Business provided the whole invoice to an interested party, they could verify each chunk in turn. Or the business might provide only relevant chunks to interested parties, who could verify those objects individually.

This might seem overcomplicated, but mapping to UBL semantics does not mean we have to map to their document granularity. HATEOS style approach should be simpler and more versatile in the long run, and shouldn't add real difficulty to maintaining a UBL/REST adapter.

[onthebreeze]

Phew! What a lot of ideas. I like the idea of a HATEOAS style collection of URL actions in response to a GET/invoices/{GUID}. Good questions about the state lifecycle of invoice - and something that will for sure provoke a lot of community discussion. There are several possible states and so that implies there are logically several response messages for a given invoice. There are a few implementation options. All start with a POST /invoices to the receiver gateway and get a GUID response that is the key to that specific invoice. After that there are several ways we could manage responses.

1. GET /invoices/{GUID} to the receiver gateway returns a UBL response structure with a complete history of status updates.
2. GET /invoices/{GUID} to the receiver gateway returns a HATEOAS style list of URLs of the form GET /responses/{GUID}, each of which point to a separate response document

Since I think we would also want to allow the recipient to POST responses back to the sender whenever there is a status change, option 2 fits more neatly with that model. So the recipient gateway (of an invoice) will POST responses back to the invoice sender (assuming the sender SMP specifies that capability) and it would also host the response for any third parties (eg debtor financiers) to GET/response/{GUID}. This discrete responses model is probably also a bit more compatible with a future blockchain style shared state model.

If you make a pull request along these lines, I'll be pleased to accept.

[asmith1024]

Can we get higher-entropy resource identifiers please? Even if there's no easy exploit, a recognizable UID structure may encourage hostile experimentation.

[onthebreeze]

Excellent point. I'm not an expert on algorithms for cryptographically strong GUIDs (as opposed to just unique but still maybe guessable GUIDs). Any suggestions?

[asmith1024]

Yes. The bonus is it's based on GUIDs under-the-bonnet, so the generating access point can still work with those. I have to produce a "vanilla" implementation and get permission to publish. Won't take too long.

[monkeypants]

UUID4 has 100 bits of entropy. It didn't occur to me that it wasn't enough, it's a good point.

https://tools.ietf.org/html/rfc4122.html#section-6 actually makes it too:

Distributed applications generating UUIDs at a variety of hosts must
be willing to rely on the random number source at all hosts.

We should be using payload encryption where we need it. It's fair to assume anyone who wants a complete copy of the public data can have one. What would be the problem if someone could magically guess the id's?

[asmith1024]

1. Our users and their business partners need payload encryption and signing. Senders encrypt with the recipient's public key and sign using their own private key. No one reads the contents except the intended recipient. A business does not have to supply its own PKI. It can delegate this to its Access Point provider.
2. The real danger is not threats you can think of, but ones you can't. It is counter-productive to mandate a threat vector. Even if 99% of Access Point providers implement UIDs in a safe, non-enumerable manner, it's only a matter of time before the vulnerable implementation is exploited. Another reason for 1.

[monkeypants]

No argument about payload encryption and signing, we are on the same page there.

I'm not saying "I can't think of a specific threat, therefor there isn't one". I'm saying "any sufficiently determined/lucky attacker can get access to (some or all) invoice and acknowledgement URLs". The URLs are not guarded secrets, they are bandied about to whoever needs them. Some parties will be malevolent, foolish, and/or unlucky; It's unavoidable.

So, when (not if) an attacker gets access to invoice and acknowledgement URLs, what's the damage?
If we make a concerted effort, then I think we can make it harmless by ensuring that knowledge of the URL (alone) only gives you access to safe public data (signatures, GUIDs, maybe cypher-text but I'm not even sure that's necessary).

In other words we might POST a signed and encrypted invoice/response, but we can only GET safe public data (e.g. the signature of the plaintext). And only store safe public data too.

So if I provide an plaintext invoice + URL to an interested party and assert that I sent it, the interested party can compare a locally computed hash of the plaintext with the signature retrieved by GET URL, and verify/falsify my assertion. Assuming we don't make a mistake with the crypro, I think that publishing the hash does not leak any commercially sensitive information. Is that right?

The difficult thing is to prevent commercially sensitive information leaking out through historic traffic analysis (assuming attacker has access to a high proportion of URLs, or even all of them). I think that means we need to ensure the public information is non-identifying. Assuming the attacker also has the entire NAPTR record, the only identifying information about the URL is the Access Point / Gateway (AP/GW, nomenclature?) that the URL belongs to, which translates to a collection of ABN through the NAPTR DB. That's an argument for using a popular AP/GW, there will be a size of AP/GW that's to small (identity of URL could be guessed). A sufficiently large one will provide identity-safety-in-numbers.

[AlistairSkippr]

I am not technically experienced to comment here but from a commercial perspective for debtor finance, this HATEOAS style collection of URLs to provide up-to-date statuses for invoices will be invaluable for a financier. We will be able to calibrate our API with the accounting platform who sends outs the invoice to dynamically track these statuses so if there were any red flag events, the appropriate actions can be pursued immediately.

[asmith1024]

@AlistairSkippr the only argument we have with HATEOAS is the acronym itself: it sucks. What we're saying here is when we specify an identifier in a URL we don't use a database ID or a UID or any recognizable/enumerable data type, but rather as purely random (and therefore meaningless, except as an ID) a sequence of characters as possible.

[monkeypants]

the only argument we have with HATEOAS is the acronym itself: it sucks.

Do you prefer "RESTafarian", or should we just say "no session-state, locking mechanisms or any of that rubbish".