0037 XLS-37d: Improved Concise Transaction Identifier (CTID)

[interc0der]

An updated version of this spec can be found here: XLS-37: Concise Transaction Identifier CTID.

The earlier version can be found by looking at the edit history.

[dangell7]

Can you give an example of how you see an explorer, a developer and a user using this standard?

How would we use this? Just some examples might help.

[interc0der]

Sure. You can think of these Improved Concise Transaction Identifiers (CTIM) as a replacement for transaction hashes. In most situations, where you are querying information about a validated transaction, you can use this identifier over a hash.

Maybe you are a front-end developer and you want to cache an array of validated transactions for a given wallet. You could store an unfiltered array of concise identifiers for any and all networks using the XRP Ledger Protocol Chain. Each identifier gives information about the validating network (chain) and ledger sequence. This would not be possible with transaction hashes.

Maybe you are an explorer, and you want to set up a map of ledger indexes with nested transactions. Your transaction query would be far most efficient if you know the ledger index. Sure, you may do this with a transaction hash ( decode hash and get sequence number ), but since these concise identifiers are not encoded using the contents of the transaction, they take up a quarter of the space as compared to the hashes.

Just some ideas. Hope this helps.

[dangell7]

Thank you,

This is only for the transaction hash right? Not the following hashs?

• Channel
• Escrow
• Offer
• NFTokenOffer
• NFTokenID
• URITokenID
• Address

[interc0der]

CTIMs would be valid for all transaction types such as NFTokenCreateOffer, PaymentChannelCreate, etc. However, concise transaction identifiers would not apply to things like Addresses, NFtokenId, ChannelId, etc.

Sure, the networkId could be added to the base transaction and I am not directly opposed to that. However, having a slim identifier holding only the ledger sequence, transaction id, and network id, seems like a valuable asset to all developers building on the ledger, especially as more sidechains and parallel networks are operating in tandem.

Whether this identifier sits on the ledger is still up for vote. ( I have not seen Richard's PR, but rather than something that sits on the ledger, its probably going to be a "calculated" value injected in API responses )

[interc0der]

I should also note:
A standard proposal (and PR) pertaining to networkId's is in the works. I would love to get your feedback about that standard proposal and how networkId's could be structured to enhance interoperability

[mvadari]

What does the M stand for in CTIM?

[RichardAH]

Improved like in VIM. ICTI didn't really roll off the tongue

[RichardAH]

Updated the name from CTIM -> CTID on @mvadari recommendation

[pkcs8]

All other artifacts (r-address, hashes, ledger index etc) in XRPL are network agnostic. Should CTIDs be network aware?

[AlexanderBuzz]

I am currently evaluating this CTID thing. Use case is transaction syncing in E-Commerce, where it may likely save some DB resources. Let's say someone uses the plugin in test mode and decides to switch to live, there would be transactions from Testnet and Mainnet in the table. So the requirement "get me the last ledger index and update from there" would result in two last ledger indexes in that case. Plus that it allows to use sidechains ad libitum.

[interc0der]

Exactly!
If you are using the ledger index ( sequence number ) and transaction index to identify transactions (rather than the transaction hash), there will be an overlap between parallel chains. For this reason, the CTID needs to be network aware.

If using a transaction hash, the network ID would be included in the hashed content, therefore making it unique and identifiable across parallel networks.

[AlexanderBuzz]

Hey, what about putting the network id at the first position? It would allow for indexing in databases (performance) and even fit the acronym, now being "CanonicalTransaction Identifier" - hehehe

[interc0der]

Could you give us an example of how you would index your database if the network_id was in the first position?
I would be curious to know what the performance benefits would be if indexed by network_id compared to ledger_index.

[AlexanderBuzz]

Ok, so the hands-on use case is someone paying with XRP in an online shop. The payment page polls the shop periodically to check wether the transaction is already settled. The Shop in turn fetches the latest transactions from the ledger. For performance reasons, and maybe they are just academic, it is desirable to get the $lastLedgerIndex like "snip". Currently I would have to save the network and the ledger index and index both. Would be something like SELECT MAX(ledger_index) from xrpl_tx WHERE network_id = 'testnet'. Having it all in one column might increase performance, but I would have to coroborrate that by tests. Is there a tool to create boarloads of XRPL testdata? If not, I create one, I need one in the future anyways.

For those that don't know, in relational databases like MySQL columns can get indexed, which means that internally those colums are stored in a pre-sorted way so that searching and joining besomes much faster that search operations in randomly distributed datasets

[cjcobb23]

Why not just add a chain id to the transaction itself, which would cause transactions on different chains to always have different hashes? This is what the various EVM chains do to prevent replaying transactions on different chains and to keep hashes unique.

[mvadari]

That's already a separate proposal: XRPLF/rippled#4370

[interc0der]

@dangell7 had a similar idea. See #91 (reply in thread) was my response.

Long story short, CTIDs need to be network-aware since they are not derived from the contents of the transaction, but rather the sequential order. Adding the Network ID to all transaction objects is independent of CTIDs

[dangell7]

@interc0der It was already in there when I said that 😂 lol

I did make this the other day though.

Its a global xrpl id. If you paste this into a search bar, you would be able to parse go directly to the tx or ledger object/entry. Kinda cool, but not sure anyone would use it lol

its long right now but when it uses the ctid it could be pretty short.

[AlexanderBuzz]

Stupid question perhaps, but how to I transfer a NetworkId like "testnet" to a int/hex value?

[AlexanderBuzz]

Um, not really yet. I could arbitrarily assign NetworkIds like I do now, but when they will get standardized in the future and those mappings are not correct, bunk goes the CTID. Perhaps this proposal should also include a mapping of the currently existing networks.

Which would take some consideration, as I could see there might be reserved spaces like in networking ports. For example, I will need to test at some point with a local rippled, and there might me test cases where more than one instance is needed, so a reserved space for 10-100 values to be used for local testing would be nice. all the possible XLS0815d's would also require some space, as they migh involve the creation of a new network for testing purposes etc.

[interc0der]

"In XRPL Protocol Chains the Network ID should match the chosen peer port. Thus the natural limitation on Network ID is that of the TCP port (65536)."

[AlexanderBuzz]

When I look here devnet / testnet /amm-devnet all have the same port :(

[mvadari]

First of all, that link is referring to the HTTPS and WebSocket ports, not the peer port. Second of all, you can run server_info on any network to find out its network ID - mainnet is 0, testnet is 1, and devnet is 2 I believe.

[AlexanderBuzz]

Thanks for clarifying this, I cobbled together peer port and network port and'm probably not the only one.

[nbougalis]

CTIDs are composed of 16 uppercase hex nibbles, and always begin with a C.
The identifier is divided into three fields and a single lead-in nibble:

C [ XXXXXXX ] [ YYYY ] [ ZZZZ ]

Here, X represents, Y represents and Z represents is the ledger sequence number, Y is the transaction ID and Z is the network ID.

I'd recommend changing the format to either ZZZZ-XXXXXXXX-YYYY which makes identifiers more "natural" for humans, or ZZZZ-YYYY-XXXXXXX which is likely to mismatch sooner, making distinguishing two separate CTIDs easier. I feel that human readability is an important consideration.

[AlexanderBuzz]

CanonicalTransactionIDentifier :)

[sappenin]

@RichardAH Do you mind pushing this spec along through new CONTRIBUTING process that was recently merged?
In particular, I think the next step here is to publish this spec as a DRAFT via a PR to this repo (read more about how to do that here).

One intention with the new process is that all specs would move through the process in a more overt fashion before code is merged into rippled develop branch, so I think this will have bearing on #4418.

Of course, this is a somewhat new process, so feedback welcome as we try it out in this repo (possibly as a PR to the CONTRIBUTING.md if you want).

Just today, I initiated this process for the CFT spec -- see it in action here.

[dangell7]

@sappenin The standard spec has been added to the PR's. Thank you

[tequdev]

[dangell7]

Thats documentation drift. We will get that updated. No flag

[tequdev]

I made a mistake in the comment place. sorry.

[dangell7]

Hello group: per the new XLS Contributing process, it is my opinion that we have reach a "well-refined standard."

As such, I propose that we move this discussion to a file (via #111) and work on final changes using additional PRs, for better change-tracking.

Please comment here if you would like to object to moving this spec/discussion forward in the process into a DRAFT spec.

The official PR in the rippled repository is XRPLF/rippled#4418.

(Note that per the Contributing guidelines, moving a spec into the DRAFT state does not mean any kind of endorsement, nor does it mean that this specification will become adopted. It is solely meant as a mechanism to enable better change tracking using PRs.)

[intelliot]

Closing this discussion in order to direct future comments to these PRs:

• Specification PR: XLS-37d: Improved Concise Transaction Identifier (CTID) #111
• Implementation PR: feat: support Concise Transaction Identifier (CTID) (XLS-37) rippled#4418