From d786e9eda859867539cbba25f9c1d8d39cf7b7cf Mon Sep 17 00:00:00 2001 From: Han Liu <667070+dixia@users.noreply.github.com> Date: Tue, 11 Feb 2020 19:13:37 +0800 Subject: [PATCH 1/5] check-in the raw MD file check-in the raw MD file --- docs/03_tutorials/how-to-monitor-state.md | 90 ++++++++ .../monitoring-with-3rd-party-plugins.md | 157 ++++++++++++++ .../monitoring-with-state-history-plugin.md | 196 ++++++++++++++++++ 3 files changed, 443 insertions(+) create mode 100644 docs/03_tutorials/how-to-monitor-state.md create mode 100644 docs/03_tutorials/monitoring-with-3rd-party-plugins.md create mode 100644 docs/03_tutorials/monitoring-with-state-history-plugin.md diff --git a/docs/03_tutorials/how-to-monitor-state.md b/docs/03_tutorials/how-to-monitor-state.md new file mode 100644 index 00000000..3b2849d6 --- /dev/null +++ b/docs/03_tutorials/how-to-monitor-state.md @@ -0,0 +1,90 @@ + +[block:callout] +{ + "type": "info", + "body": "This article assumes you have installed the EOSIO software and are familiar with using the EOSIO nodeos and cleos tools. It is recommended that you have completed [the Getting Started section](https://developers.eos.io/eosio-home/docs)" +} +[/block] +# Introduction + +This tutorial is a guide for monitoring the status of transactions on the blockchain. + +As an example we will show how an application can monitor EOS token transfers, show how a transaction state changes, how this affects the blockchain, and show when a transaction is irreversible. The example will show the transfer of tokens using standard-conforming EOSIO token contracts. The eosio.token contract conforms to this standard. + + +# Running Nodeos + +It is recommended that a local instance of nodeos is used to monitor activity. This local instance will connect to the blockchain, and when run with the state history plugin, collect the history of actions on the blockchain. +[block:callout] +{ + "type": "info", + "body": "The local instance in this tutorial will be run as a non-producing node see [Non-Producing Node](https://developers.eos.io/eosio-nodeos/docs/environment-non-producing-node) for more details." +} +[/block] +Nodeos can be run in three modes: + + 1. **Speculative** In "speculative" mode you can see transactions and state changes up to the current head block plus changes in the pending block. The state changes in the pending block are also included in the chain state file. + + 2. **Head** In "head" mode you can see transactions and state changes up to the current head block. This node will produce a pending block, though the pending block is not included in the chain state file. + + 3. **Read only** In "read-only" mode you can see transactions and state changes up to the current head block. This node will NOT produce a pending block. + + +- The default mode is speculative. +- Chain state is a memory mapped file containing the state of the blockchain for each block. A record of chain state is kept for each block going back to the last irreversible block. +- The pending block is the block currently being built by a producing node, transactions are added to the pending block as they are received and processed. The pending block becomes the head block once it is written to the blockchain. Pending blocks are discarded when in Head mode. Pending blocks are not produced when in Read Only mode. + +Nodes used for monitoring transactions should be run in: + - Speculative mode to see transactions as they arrive (Confirmed and unconfirmed) + - Read-Only mode to see transactions once they are recorded in the blockchain (Confirmed only) + +For more information see [Read Modes](https://developers.eos.io/eosio-nodeos/docs/read-modes) +[block:callout] +{ + "type": "warning", + "body": "A transaction is only complete once it has a status of executed and the block containing the transaction is irreversible.\nA transaction is confirmed once it has been written to a block on the blockchain.\nA transaction is unconfirmed if it has been received by nodeos but has not yet been written to a block on the blockchain.\nA transaction may also fail or expire." +} +[/block] + ## Transaction States + +Transactions can have the following states: + +- **executed**. The transaction has succeeded, no error handler executed +- **soft_fail**. The transaction has objectively failed (not executed) and the error handler executed + successfully (onerror) +- **hard_fail**. The transaction has objectively failed and error handler objectively failed thus no state change occurred (onerror also failed) +- **delayed**. The transaction is delayed/deferred/scheduled for future execution +- **expired**. The transaction is expired and the storage space refunded to the user + +A transaction is not executed until the status is executed, and the transaction can be cancelled up until it is executed, failed or expired, i.e. while it is waiting to be executed. + +Transactions can be delayed for **up to** 45 days, these are known as `deferred transactions`, and the delay can **only** be set when the transaction is generated. A `deferred transaction` can be cancelled at any time before it is executed. For more information about deferred transactions please see [Communication Model](doc:communication-model) +[block:callout] +{ + "type": "warning", + "body": "A transaction may be executed, but only when the block containing the transaction is irreversible can you be sure that the operation is final and complete. A block is irreversible if the block number is less than the block number of the last irreversible block (LIB). The last irreversible block (LIB) is the most recent block which has been acknowledged by 2/3 of the block producers." +} +[/block] +## Tracking YOUR transaction + +It is always possible that transactions similar to your transactions are processed while you are monitoring the public main net. It is important to ensure that you use ALL identifying fields to ensure you are looking at the right transaction. + +Some of the fields you should use to identify a transaction are: transaction_id, account, name, data, to, receiver, memo, transaction_id. + +## Ensuring a Transaction is complete and final + +When the block containing a transaction has a `"block_num"` <= `"last_irreversible_block"`, then the transaction is also irreversible. + +Once the last irreversible block has moved past the expiration time of a transaction, you can safely mark a transaction as failed. + +## Handling Errors + +Sometimes network issues will cause a transaction to fail and never be included in a block. Your internal database will need to know when this has happened so that it can inform the user and/or try again. If you do not get an immediate error when you submit your local transfer, then you must wait for the transaction to expire. Every transaction has an "expiration", after which the transaction can never be applied. Once the last irreversible block has moved past the expiration time, you can safely mark your attempted withdrawal as failed and not worry about it "floating around the ether" to be applied when you least expect. + +## Monitoring Options + +Now that we understand the various states that a transaction can be in, and how this relates to blockchain and the last irreversible block (LIB) we can look at the various options for monitoring transactions and their current state. + +In part II [Monitoring with State History Plugin](doc:monitoring-with-state-history) we will discuss monitoring state using the state history plugin. + +In part III [Monitoring with 3rd Party plugins](doc:monitoring-with-3rd-party) we will discuss 3rd party plugins which may also be used for monitoring state. \ No newline at end of file diff --git a/docs/03_tutorials/monitoring-with-3rd-party-plugins.md b/docs/03_tutorials/monitoring-with-3rd-party-plugins.md new file mode 100644 index 00000000..73d788fd --- /dev/null +++ b/docs/03_tutorials/monitoring-with-3rd-party-plugins.md @@ -0,0 +1,157 @@ + +[block:callout] +{ + "type": "warning", + "body": "The resources listed below are developed, offered, and maintained by third-parties and not by block.one. Providing information, material, or commentaries about such third-party resources does not mean we endorse or recommend any of these resources. We are not responsible, and disclaim any responsibility or liability, for your use of or reliance on any of these resources. Third-party resources may be updated, changed or terminated at any time, so the information below may be out of date or inaccurate. USAGE AND RELIANCE IS ENTIRELY AT YOUR OWN RISK." +} +[/block] +In part I [How to monitor state](doc:how-to-monitor-state) we discussed the states of a transaction. + +In part II [Monitoring with State History Plugin](doc:monitoring-with-state-history) we discussed monitoring state using the state history plugin. + +In this article we will take a brief look at some of the 3rd party options which are available for monitoring the state of the blockchain. There may be other solutions for monitoring state not present in this document so please research what is currently the best option for you. + +This information comes from `https://github.com/EOSIO/eos/blob/master/plugins/COMMUNITY.md`. These are plugins developed by the community, in no particular order, which can be added to Nodeos by compiling Nodeos with those plugins included. In this article we only show the plugins related to monitoring transactions. + +| Description | URL | +| ----------- | --- | +| ElasticSearch | https://github.com/EOSLaoMao/elasticsearch_plugin | +| Kafka | https://github.com/TP-Lab/kafka_plugin | +| Eosio MySQLdb Plugin | https://github.com/eosBLACK/eosio_mysqldb_plugin | +| SQL | https://github.com/asiniscalchi/eosio_sql_plugin | +| EOSIO Watcher - Watch for specific actions and send them to an HTTP URL | https://github.com/eosauthority/eosio-watcher-plugin | +| ZMQ / history | https://github.com/cc32d9/eos_zmq_plugin | +| ZMQ Light History API | https://github.com/cc32d9/eos_zmq_light_api | +| Mongo History API | https://github.com/CryptoLions/EOS-mongo-history-API | +| Chintai ZMQ Watcher | https://github.com/acoutts/chintai-zeromq-watcher-plugin | +| State History API | https://github.com/acoutts/EOS-state-history-API | +| Chronicle | https://github.com/EOSChronicleProject/eos-chronicle | +| Hyperion History | https://github.com/eosrio/Hyperion-History-API | +| eos zmq plugin receiver | https://github.com/cc32d9/eos_zmq_plugin_receiver | +| dfuse (commercial) | https://www.dfuse.io/en/technology | +[block:callout] +{ + "type": "warning", + "body": "These are in no particular order.\nDescriptions of each tool have been provided by the tool authors." +} +[/block] + +[block:api-header] +{ + "title": "Elastic Search Plugin" +} +[/block] +The Elasticsearch plugin is a nodeos plugin for archiving blockchain data into Elasticsearch. It is designed to transfer nodeos data structures to json documents, and to post these documents to an Elasticsearch cluster, using bulk requests. It speeds up the processing progress by using a thread pool to handle the serialisation and the bulk request jobs. + +Please refer to https://github.com/EOSLaoMao/elasticsearch_plugin for more information. +[block:api-header] +{ + "title": "Kafka Plugin" +} +[/block] +EOSIO Kafka Plugin is used to receive the transaction data from the blockchain and to send out the transaction through a kafka producer. Developers can receive the transaction data through a kafka consumer in the back-end application. + +Please refer to https://github.com/TP-Lab/kafka_plugin for more information. +[block:api-header] +{ + "title": "Eosio MySQLdb Plugin" +} +[/block] +The mysqldb_plugin is a plug-in to store block, transaction and action data in a MySQL database. + +Please refer to https://github.com/eosBLACK/eosio_mysqldb_plugin for more information. +[block:api-header] +{ + "title": "SQL" +} +[/block] +Please refer to https://github.com/asiniscalchi/eosio_sql_plugin for more information. +[block:api-header] +{ + "title": "EOSIO Watcher" +} +[/block] +EOSIO Watcher plugin is a simple plugin to watch for specific actions on chain and send them as HTTP POSTs to a url in real-time. + +Please refer to https://github.com/eosauthority/eosio-watcher-plugin for more information. +[block:api-header] +{ + "title": "ZMQ / history" +} +[/block] +The ZMQ / history plugin collects EOS action traces, converting them into JSON format, and exporting them via a ZeroMQ socket. It may export all action traces, or only those relevant to a number of whitelisted accounts. A receiver for this data can be implemented in any modern programming language that supports the ZeroMQ library. + +Please refer to https://github.com/cc32d9/eos_zmq_plugin for more information. +[block:api-header] +{ + "title": "ZMQ Light History API" +} +[/block] +The ZMQ Light History API receives data from the `nodeos` ZMQ / history plugin. This stores the latest state of each account and account token balances. It is called Light because does not store historic data. + +Please refer to https://github.com/cc32d9/eos_zmq_light_api for more information. + +[block:api-header] +{ + "title": "Mongo History API" +} +[/block] +The Mongo History API Plugin was developed as a gateway to the MongoDB Plugin which may be used in place of the history plugin and state history plugin. The API requests are the same, however, there are some additional API requests. + + Please refer to https://github.com/CryptoLions/EOS-mongo-history-API for more information and to https://history.cryptolions.io/ for documentation. +[block:api-header] +{ + "title": "Chintai ZMQ Watcher Plugin" +} +[/block] +This plugin is designed to allow a blockchain application developer to easily monitor actions from the blockchain. It is originally forked from EOSIO Watcher but modified to use ZeroMQ instead of HTTP requests, as using ZMQ guarantees ordering. When building a blockchain application that requires an exact ordering of block events and a guarantee that no blocks will be (easily) lost during a replay, a message queue like ZeroMQ is a great solution. + +- While this plugin does allow for the monitoring of on-chain events, it does not include a ZeroMQ receiver component which would receive the events from the plugin. The receiver is generally proprietary to the specific implementation of your blockchain application's web backend, however, there are ZeroMQ libraries available for most programming languages. This plugin puts blockchain events onto a ZeroMQ messaging queue, and any language using the ZeroMQ library can connect to the queue and process events from it. +- This plugin listens to head block signals and is therefore subject to periodic microforks in the network. The handling of these forks must be done in your ZMQ receiver component. If you prefer, you can modify the plugin to use irreversible blocks, however, there would be a 2-3 minute delay in receiving transactions. +- The plugin is currently setup to filter on actions specific to the Chintai smart contract. The filtering function should be updated to your dApps's specific implementation. + +Please refer to https://github.com/acoutts/chintai-zeromq-watcher-plugin for more information +[block:api-header] +{ + "title": "State History API" +} +[/block] +The State History API is intended as a public API interface which matches the history plugin api, but uses the data stored by the state history plugin in a Postgresql database, by using fill-postgresql, please see [Monitoring with State History Plugin](doc:monitoring-with-state-history) + +Please refer to https://github.com/acoutts/EOS-state-history-API for more information. +[block:api-header] +{ + "title": "Chronicle" +} +[/block] +The chronicle project aims to build a toolset for EOSIO history databases. Currently Chronicle Receiver is released, and this tool will read the data from the state history plugin, decoding it into JSON format, and exporting this json to downstream consumers. + +Please refer to https://github.com/EOSChronicleProject/eos-chronicle for more information. +[block:api-header] +{ + "title": "Hyperion History" +} +[/block] +Hyperion History uses a multi-threaded indexer to extract data from the state history plugin, storing data in an Elasticsearch cluster. Optimising the data structures used to store data has reduced the amount of storage required by up to 85%, and has meant lower response times when responding to queries whilst also reducing data transfer overhead. + +Access to this data is provided by a javascript library, the Hyperion HTTP API, and next steps include implementing a WebSocket API for action streaming. + +Please refer to https://medium.com/@eosriobrazil/presenting-hyperion-history-api-solution-f8a8fda5865b for more information. +The source code can be located here https://github.com/eosrio/Hyperion-History-API +For more information about the Hyperion HTTP API Javascript library please refer to https://eos.hyperion.eosrio.io/v2/docs/index.html and the source code for this library can be found here https://github.com/eoscafe/hyperion-api +[block:api-header] +{ + "title": "eos zmq plugin receiver" +} +[/block] +This is a set of tools written in Perl for receiving and processing the EOSIO network events exported by the ZMQ / history plugin (see above). They can be useful for dispatching and troubleshooting message flow, and there is a ready-made solution for storing action data in MySQL database. + +Please refer to https://github.com/cc32d9/eos_zmq_plugin_receiver for more information. +[block:api-header] +{ + "title": "dfuse" +} +[/block] +dfuse offers powerful streaming APIs, GraphQL queries and subscriptions, server-side navigation of forks, and some of the deepest insight of on-chain data, like action-level database operations. dfuse also allows querying of the blockchain state (contract tables) at any past block height, providing a consistent view of tables even when they hold millions of rows. dfuse Search allows you to find transactions through the full blockchain history in under a second, with granular search criterias. dfuse Events allows you to index special fields directly from your smart contracts, making them available instantaneously through dfuse Search. + +dfuse is a commercial offering by the team at EOS Canada. See overview of dfuse technology at https://www.dfuse.io/en/technology . Product documentation is available at https://docs.dfuse.io/ \ No newline at end of file diff --git a/docs/03_tutorials/monitoring-with-state-history-plugin.md b/docs/03_tutorials/monitoring-with-state-history-plugin.md new file mode 100644 index 00000000..f915ddbe --- /dev/null +++ b/docs/03_tutorials/monitoring-with-state-history-plugin.md @@ -0,0 +1,196 @@ +In part I [How to monitor state](doc:how-to-monitor-state) we discussed the states of a transaction. In this article we will take a look at monitoring state using the state history plugin. + +[block:api-header] +{ + "title": "State History Plugin" +} +[/block] +The state history plugin replaces the deprecated History Plugin. It caches data in file, rather than in memory, reducing the running costs of node. + +To use the state history plugin run nodeos with the "--plugin eosio::state_history_plugin" option. See the [state_history_plugin](doc:state_history_plugin) +[block:code] +{ + "codes": [ + { + "code": "nodeos -e -p eosio --plugin eosio::producer_plugin --plugin eosio::chain_api_plugin --plugin eosio::http_plugin --plugin eosio::state_history_plugin --data-dir /Users/mydir/eosio/data --config-dir /Users/mydir/eosio/config --access-control-allow-origin='*' --contracts-console --http-validate-host=false --state-history-dir /shpdata --trace-history --chain-state-history --verbose-http-errors --filter-on='*' --disable-replay-opts >> nodeos.log 2>&1 &", + "language": "shell" + } + ] +} +[/block] +The nodeos command above adds the state history plugin ``--plugin eosio::state_history_plugin`` storing the chain data in a directory called /shpdata under the --data-dir ``--state-history-dir /shpdata`` storing both trace history and chain-state-history ``--trace-history --chain-state-history``. To run the state history plugin you are required to ``--disable-replay-opts ``. + +Once nodeos is running the state history plugin, you need to run fill-postgresql (or an alternative connector) to store the chain data in a postgresql database for querying. +[block:callout] +{ + "type": "warning", + "body": "This assumes you have installed [postgesql](https://www.postgresql.org/download/) and have some knowledge of how to use psql", + "title": "postgresql is Required" +} +[/block] +## fill-postgresql + +See [fill-postgresql repo](https://github.com/EOSIO/fill-postgresql) + +Configuration options: + +| Option | Default | Description | +| ------ | ------ | +| --host | localhost | state-history-plugin host to connect to | +| --port | 8080 | state-history-plugin port to connect to | +| --schema | chain | Database schema to fill | +| --trim | | Trim history before irreversible + +When running it for the first time, use the --create option to create the schema and tables. +[block:code] +{ + "codes": [ + { + "code": "./fill-postgresql --create", + "language": "shell" + } + ] +} +[/block] + +[block:code] +{ + "codes": [ + { + "code": "$ ./fill-postgresql --create\ninfo 2019-01-22T06:25:43.039 thread-0 main.cpp:84 main ] fill-postgresql version not-a-release-1-gaf49f6d\ninfo 2019-01-22T06:25:43.039 thread-0 main.cpp:85 main ] fill-postgresql using configuration file /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/config/config.ini\ninfo 2019-01-22T06:25:43.039 thread-0 main.cpp:86 main ] fill-postgresql data directory is /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/data\ninfo 2019-01-22T06:25:43.039 thread-0 fill_postgresql_plugin:748 session ] connect to postgresql\ninfo 2019-01-22T06:25:43.044 thread-0 fill_postgresql_plugin:762 start ] connect to localhost:8080\ninfo 2019-01-22T06:25:43.053 thread-0 fill_postgresql_plugin:870 create_tables ] create schema \"chain\"\ninfo 2019-01-22T06:25:43.294 thread-0 fill_postgresql_plugin:1207 close_streams ] block 1 - 199\ninfo 2019-01-22T06:25:43.424 thread-0 fill_postgresql_plugin:1207 close_streams ] block 200 - 399\ninfo 2019-01-22T06:25:43.502 thread-0 fill_postgresql_plugin:1207 close_streams ] block 400 - 478\ninfo 2019-01-22T06:25:43.502 thread-0 fill_postgresql_plugin:1148 receive_result ] block 479\n", + "language": "text" + } + ] +} +[/block] +To wipe the schema and start over, run with --drop --create. +[block:code] +{ + "codes": [ + { + "code": "./fill-postgresql --drop --create", + "language": "shell" + } + ] +} +[/block] + +[block:code] +{ + "codes": [ + { + "code": "$ ./fill-postgresql --drop --create\ninfo 2019-01-24T09:48:04.324 thread-0 main.cpp:84 main ] fill-postgresql version not-a-release-1-gaf49f6d\ninfo 2019-01-24T09:48:04.324 thread-0 main.cpp:85 main ] fill-postgresql using configuration file /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/config/config.ini\ninfo 2019-01-24T09:48:04.324 thread-0 main.cpp:86 main ] fill-postgresql data directory is /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/data\ninfo 2019-01-24T09:48:04.325 thread-0 fill_postgresql_plugin:748 session ] connect to postgresql\ninfo 2019-01-24T09:48:04.332 thread-0 fill_postgresql_plugin:757 start ] drop schema \"chain\"\ninfo 2019-01-24T09:48:04.416 thread-0 fill_postgresql_plugin:762 start ] connect to localhost:8080\ninfo 2019-01-24T09:48:04.425 thread-0 fill_postgresql_plugin:870 create_tables ] create schema \"chain\"\ninfo 2019-01-24T09:48:04.654 thread-0 fill_postgresql_plugin:1207 close_streams ] block 1 - 199\ninfo 2019-01-24T09:48:04.785 thread-0 fill_postgresql_plugin:1207 close_streams ] block 200 - 399\ninfo 2019-01-24T09:48:04.917 thread-0 fill_postgresql_plugin:1207 close_streams ] block 400 - 599\ninfo 2019-01-24T09:48:05.063 thread-0 fill_postgresql_plugin:1207 close_streams ] block 600 - 799\n", + "language": "text" + } + ] +} +[/block] +To run normally +[block:code] +{ + "codes": [ + { + "code": "./fill-postgresql", + "language": "shell" + } + ] +} +[/block] + +[block:code] +{ + "codes": [ + { + "code": "$ ./fill-postgresql\ninfo 2019-01-24T09:48:48.664 thread-0 main.cpp:84 main ] fill-postgresql version not-a-release-1-gaf49f6d\ninfo 2019-01-24T09:48:48.664 thread-0 main.cpp:85 main ] fill-postgresql using configuration file /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/config/config.ini\ninfo 2019-01-24T09:48:48.664 thread-0 main.cpp:86 main ] fill-postgresql data directory is /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/data\ninfo 2019-01-24T09:48:48.664 thread-0 fill_postgresql_plugin:748 session ] connect to postgresql\ninfo 2019-01-24T09:48:48.675 thread-0 fill_postgresql_plugin:762 start ] connect to localhost:8080\ninfo 2019-01-24T09:48:48.831 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4200 - 4399\ninfo 2019-01-24T09:48:48.967 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4400 - 4599\ninfo 2019-01-24T09:48:49.099 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4600 - 4799\ninfo 2019-01-24T09:48:49.231 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4800 - 4999\ninfo 2019-01-24T09:48:49.361 thread-0 fill_postgresql_plugin:1207 close_streams ] block 5000 - 5199\n", + "language": "text" + } + ] +} +[/block] +This connects to the State History Plugin and populates a Postgresql database with historical data such as action traces, transaction traces, block states, and other interesting things such as contract table changes at every block. It will track real-time updates from nodeos. + +## Postgres queries, tables and results + +To identify the Last Irreversible Block +[block:code] +{ + "codes": [ + { + "code": "eos=> select head, irreversible from chain.fill_status;\n\n\n head | irreversible \n--------+--------------\n 134112 | 134111\n(1 row)", + "language": "sql" + } + ] +} +[/block] + +[block:api-header] +{ + "title": "Examples" +} +[/block] +### Prerequisites + +It is assumed that you have a local `nodeos` server connected to an EOSIO blockchain; that this nodeos is using the state history plugin; that an account has been created for `eosio.token`; and that the `eosio.token` contract has been deployed. Completing the [Deploy, Issue, and Transfer Tokens](https://developers.eos.io/eosio-home/docs/token-contract) in the 'Getting Started' section prior to this tutorial will have satisfied these prerequisites. + +This tutorial uses the transfer action of the `eosio.token` contract. This could be considered as a deposit in one account and a withdrawal in another account. + +Transfer A to B +[block:code] +{ + "codes": [ + { + "code": "cleos -u http://127.0.0.1:8889/ push action eosio.token issue '[ \"alice\", \"100.0000 SYS\", \"memo\" ]' -p eosio@active", + "language": "shell" + } + ] +} +[/block] +Transfer A to B results in chain.action_trace +[block:code] +{ + "codes": [ + { + "code": "select block_index, transaction_id, action_index, parent_action_index, transaction_status, receipt_receiver, receipt_global_sequence, account, name from chain.action_trace where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff');\n\n\n\n block_index | transaction_id | action_index | parent_action_index | transaction_status | receipt_receiver | receipt_global_sequence | account | name \n-------------+------------------------------------------------------------------+--------------+---------------------+--------------------+------------------+-------------------------+-------------+----------\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 1 | 0 | executed | eosio.token | 2877 | eosio.token | transfer\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 2 | 1 | executed | alice | 2878 | eosio.token | transfer\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 3 | 1 | executed | bob | 2879 | eosio.token | transfer\n(3 rows)\n", + "language": "sql" + } + ] +} +[/block] +Transfer A to B also results in chain.transaction_trace +[block:code] +{ + "codes": [ + { + "code": "select block_index, transaction_id, status from chain.transaction_trace where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff');\n\n\n\n block_index | transaction_id | status \n-------------+------------------------------------------------------------------+----------\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | executed\n", + "language": "sql" + } + ] +} +[/block] +Transfer A to B also results in chain.action_trace_auth_sequence +[block:code] +{ + "codes": [ + { + "code": "select * from chain.action_trace_auth_sequence where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff');\n\n\n\n block_index | transaction_id | action_index | index | transaction_status | account | sequence \n-------------+------------------------------------------------------------------+--------------+-------+--------------------+---------+----------\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 2 | 1 | executed | alice | 5\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 3 | 1 | executed | alice | 6\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 1 | 1 | executed | alice | 4\n(3 rows)\n", + "language": "sql" + } + ] +} +[/block] +Transfer A to B also results in chain.action_trace_authorization +[block:code] +{ + "codes": [ + { + "code": "select * from chain.action_trace_authorization where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff');\n\n\n block_index | transaction_id | action_index | index | transaction_status | actor | permission \n-------------+------------------------------------------------------------------+--------------+-------+--------------------+-------+------------\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 2 | 1 | executed | alice | active\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 3 | 1 | executed | alice | active\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 1 | 1 | executed | alice | active\n(3 rows)\n", + "language": "sql" + } + ] +} +[/block] + +# Alternatives +[Demux](https://github.com/EOSIO/demux-js) is an alternative that piggybacks off the RPC API. + +In part III [Monitoring with 3rd Party plugins](doc:monitoring-with-3rd-party) we will discuss 3rd party plugins which may also be used for monitoring state. \ No newline at end of file From db219de221d41d909103796051905b55588f321d Mon Sep 17 00:00:00 2001 From: bobgt Date: Wed, 12 Feb 2020 03:01:54 +0800 Subject: [PATCH 2/5] editorial changes per requested by Han --- docs/03_tutorials/how-to-monitor-state.md | 50 +++++++++++------------ 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/docs/03_tutorials/how-to-monitor-state.md b/docs/03_tutorials/how-to-monitor-state.md index 3b2849d6..bcc8a61a 100644 --- a/docs/03_tutorials/how-to-monitor-state.md +++ b/docs/03_tutorials/how-to-monitor-state.md @@ -9,12 +9,12 @@ This tutorial is a guide for monitoring the status of transactions on the blockchain. -As an example we will show how an application can monitor EOS token transfers, show how a transaction state changes, how this affects the blockchain, and show when a transaction is irreversible. The example will show the transfer of tokens using standard-conforming EOSIO token contracts. The eosio.token contract conforms to this standard. +As an example, we will show how an application can monitor EOS token transfers, how a transaction state changes, how this affects the blockchain, and when a transaction is deemed irreversible. The example will show the transfer of tokens using standard-conforming EOSIO token contracts. The eosio.token contract conforms to this standard. # Running Nodeos -It is recommended that a local instance of nodeos is used to monitor activity. This local instance will connect to the blockchain, and when run with the state history plugin, collect the history of actions on the blockchain. +It is recommended that a local instance of nodeos is used to monitor activity. This local instance will connect to the blockchain, and when run with the state history plugin, it will collect the history of actions on the blockchain. [block:callout] { "type": "info", @@ -23,20 +23,20 @@ It is recommended that a local instance of nodeos is used to monitor activity. T [/block] Nodeos can be run in three modes: - 1. **Speculative** In "speculative" mode you can see transactions and state changes up to the current head block plus changes in the pending block. The state changes in the pending block are also included in the chain state file. + 1. **Speculative**: In this mode, you can see transactions and state changes up to the current head block along with changes in the pending block. The state changes in the pending block are also included in the chain state file. - 2. **Head** In "head" mode you can see transactions and state changes up to the current head block. This node will produce a pending block, though the pending block is not included in the chain state file. + 2. **Head**: In this mode, you can see transactions and state changes up to the current head block. This node will produce a pending block, though the pending block is not included in the chain state file. - 3. **Read only** In "read-only" mode you can see transactions and state changes up to the current head block. This node will NOT produce a pending block. + 3. **Read-only**: In this mode, mode you can see transactions and state changes up to the current head block. This node will not produce a pending block. - -- The default mode is speculative. -- Chain state is a memory mapped file containing the state of the blockchain for each block. A record of chain state is kept for each block going back to the last irreversible block. -- The pending block is the block currently being built by a producing node, transactions are added to the pending block as they are received and processed. The pending block becomes the head block once it is written to the blockchain. Pending blocks are discarded when in Head mode. Pending blocks are not produced when in Read Only mode. +[[info | Info]] +| * The **Speculative** mode is the default mode. +| * The chain state file is a memory mapped file containing the state of the blockchain for each block. A record of chain state is kept for each block going back to the last irreversible block. +| * The pending block is the block currently being built by a producing node, transactions are added to the pending block as they are received and processed. The pending block becomes the head block once it is written to the blockchain. Pending blocks are discarded when in **Head** mode. Pending blocks are not produced when in **Read-only** mode. Nodes used for monitoring transactions should be run in: - - Speculative mode to see transactions as they arrive (Confirmed and unconfirmed) - - Read-Only mode to see transactions once they are recorded in the blockchain (Confirmed only) + - **Speculative** mode to see transactions as they arrive (Confirmed and unconfirmed) + - **Read-only** mode to see transactions once they are recorded in the blockchain (Confirmed only) For more information see [Read Modes](https://developers.eos.io/eosio-nodeos/docs/read-modes) [block:callout] @@ -49,25 +49,25 @@ For more information see [Read Modes](https://developers.eos.io/eosio-nodeos/doc Transactions can have the following states: -- **executed**. The transaction has succeeded, no error handler executed -- **soft_fail**. The transaction has objectively failed (not executed) and the error handler executed +- **executed**: The transaction has succeeded, no error handler executed +- **soft_fail**: The transaction has objectively failed (not executed) and the error handler executed successfully (onerror) -- **hard_fail**. The transaction has objectively failed and error handler objectively failed thus no state change occurred (onerror also failed) -- **delayed**. The transaction is delayed/deferred/scheduled for future execution -- **expired**. The transaction is expired and the storage space refunded to the user +- **hard_fail**: The transaction has objectively failed and error handler objectively failed, thus, no state change occurred (onerror also failed) +- **delayed**: The transaction is delayed/deferred/scheduled for future execution +- **expired**: The transaction is expired and the storage space refunded to the user -A transaction is not executed until the status is executed, and the transaction can be cancelled up until it is executed, failed or expired, i.e. while it is waiting to be executed. +A transaction is not executed until the status is executed, and the transaction can be cancelled up until it is executed, failed or expired, i.e., while it is waiting to be executed. -Transactions can be delayed for **up to** 45 days, these are known as `deferred transactions`, and the delay can **only** be set when the transaction is generated. A `deferred transaction` can be cancelled at any time before it is executed. For more information about deferred transactions please see [Communication Model](doc:communication-model) +Transactions can be delayed for **up to** 45 days, these are known as `deferred transactions`, and the delay can **only** be set when the transaction is generated. A `deferred transaction` can be cancelled at any time before it is executed. For more information about deferred transactions, see [Communication Model](doc:communication-model) [block:callout] { "type": "warning", "body": "A transaction may be executed, but only when the block containing the transaction is irreversible can you be sure that the operation is final and complete. A block is irreversible if the block number is less than the block number of the last irreversible block (LIB). The last irreversible block (LIB) is the most recent block which has been acknowledged by 2/3 of the block producers." } [/block] -## Tracking YOUR transaction +## Tracking your transaction -It is always possible that transactions similar to your transactions are processed while you are monitoring the public main net. It is important to ensure that you use ALL identifying fields to ensure you are looking at the right transaction. +It is always possible that transactions similar to your transactions are processed while you are monitoring the public mainnet. Ensure that you use ALL identifying fields to look at the right transaction. Some of the fields you should use to identify a transaction are: transaction_id, account, name, data, to, receiver, memo, transaction_id. @@ -75,16 +75,16 @@ Some of the fields you should use to identify a transaction are: transaction_id, When the block containing a transaction has a `"block_num"` <= `"last_irreversible_block"`, then the transaction is also irreversible. -Once the last irreversible block has moved past the expiration time of a transaction, you can safely mark a transaction as failed. +Once the last irreversible block has moved past the expiration time of a transaction, you can safely mark a transaction as **failed**. ## Handling Errors -Sometimes network issues will cause a transaction to fail and never be included in a block. Your internal database will need to know when this has happened so that it can inform the user and/or try again. If you do not get an immediate error when you submit your local transfer, then you must wait for the transaction to expire. Every transaction has an "expiration", after which the transaction can never be applied. Once the last irreversible block has moved past the expiration time, you can safely mark your attempted withdrawal as failed and not worry about it "floating around the ether" to be applied when you least expect. +Sometimes, network issues will cause a transaction to fail and never be included in a block. Your internal database will need to know when this has happened so that it can inform the user and/or try again. If you do not get an immediate error when you submit your local transfer, then you must wait for the transaction to expire. Every transaction has an "expiration", after which the transaction can never be applied. Once the last irreversible block has moved past the expiration time, you can safely mark your attempted withdrawal as failed and not worry about it "floating around the ether" to be applied when you least expect. ## Monitoring Options -Now that we understand the various states that a transaction can be in, and how this relates to blockchain and the last irreversible block (LIB) we can look at the various options for monitoring transactions and their current state. +Now that we understand the various states that a transaction can be in, and how this relates to blockchain and the last irreversible block (LIB), we can look at the various options for monitoring transactions and their current state. -In part II [Monitoring with State History Plugin](doc:monitoring-with-state-history) we will discuss monitoring state using the state history plugin. +In part II, [Monitoring with State History Plugin,](doc:monitoring-with-state-history) we will discuss monitoring state using the state history plugin. -In part III [Monitoring with 3rd Party plugins](doc:monitoring-with-3rd-party) we will discuss 3rd party plugins which may also be used for monitoring state. \ No newline at end of file +In part III, [Monitoring with 3rd Party plugins,](doc:monitoring-with-3rd-party) we will discuss 3rd party plugins which may also be used for monitoring state. \ No newline at end of file From 823c78a465bac418ac36663aba685af8b040766f Mon Sep 17 00:00:00 2001 From: Han Liu <667070+dixia@users.noreply.github.com> Date: Wed, 19 Feb 2020 16:01:35 +0800 Subject: [PATCH 3/5] remove old syntax remove old syntax --- docs/03_tutorials/how-to-monitor-state.md | 39 +-- .../monitoring-with-state-history-plugin.md | 277 ++++++++---------- 2 files changed, 141 insertions(+), 175 deletions(-) diff --git a/docs/03_tutorials/how-to-monitor-state.md b/docs/03_tutorials/how-to-monitor-state.md index bcc8a61a..494d25ba 100644 --- a/docs/03_tutorials/how-to-monitor-state.md +++ b/docs/03_tutorials/how-to-monitor-state.md @@ -1,10 +1,7 @@ -[block:callout] -{ - "type": "info", - "body": "This article assumes you have installed the EOSIO software and are familiar with using the EOSIO nodeos and cleos tools. It is recommended that you have completed [the Getting Started section](https://developers.eos.io/eosio-home/docs)" -} -[/block] +[[info]] +| +This article assumes you have installed the EOSIO software and are familiar with using the EOSIO nodeos and cleos tools. It is recommended that you have completed [the Getting Started section](https://developers.eos.io/eosio-home/docs) # Introduction This tutorial is a guide for monitoring the status of transactions on the blockchain. @@ -15,12 +12,9 @@ As an example, we will show how an application can monitor EOS token transfers, # Running Nodeos It is recommended that a local instance of nodeos is used to monitor activity. This local instance will connect to the blockchain, and when run with the state history plugin, it will collect the history of actions on the blockchain. -[block:callout] -{ - "type": "info", - "body": "The local instance in this tutorial will be run as a non-producing node see [Non-Producing Node](https://developers.eos.io/eosio-nodeos/docs/environment-non-producing-node) for more details." -} -[/block] +[[info]] +| +The local instance in this tutorial will be run as a non-producing node see [Non-Producing Node](https://developers.eos.io/eosio-nodeos/docs/environment-non-producing-node) for more details. Nodeos can be run in three modes: 1. **Speculative**: In this mode, you can see transactions and state changes up to the current head block along with changes in the pending block. The state changes in the pending block are also included in the chain state file. @@ -39,12 +33,12 @@ Nodes used for monitoring transactions should be run in: - **Read-only** mode to see transactions once they are recorded in the blockchain (Confirmed only) For more information see [Read Modes](https://developers.eos.io/eosio-nodeos/docs/read-modes) -[block:callout] -{ - "type": "warning", - "body": "A transaction is only complete once it has a status of executed and the block containing the transaction is irreversible.\nA transaction is confirmed once it has been written to a block on the blockchain.\nA transaction is unconfirmed if it has been received by nodeos but has not yet been written to a block on the blockchain.\nA transaction may also fail or expire." -} -[/block] +[[warning]] +| +A transaction is only complete once it has a status of executed and the block containing the transaction is irreversible. +A transaction is confirmed once it has been written to a block on the blockchain. +A transaction is unconfirmed if it has been received by nodeos but has not yet been written to a block on the blockchain. +A transaction may also fail or expire. ## Transaction States Transactions can have the following states: @@ -59,12 +53,9 @@ Transactions can have the following states: A transaction is not executed until the status is executed, and the transaction can be cancelled up until it is executed, failed or expired, i.e., while it is waiting to be executed. Transactions can be delayed for **up to** 45 days, these are known as `deferred transactions`, and the delay can **only** be set when the transaction is generated. A `deferred transaction` can be cancelled at any time before it is executed. For more information about deferred transactions, see [Communication Model](doc:communication-model) -[block:callout] -{ - "type": "warning", - "body": "A transaction may be executed, but only when the block containing the transaction is irreversible can you be sure that the operation is final and complete. A block is irreversible if the block number is less than the block number of the last irreversible block (LIB). The last irreversible block (LIB) is the most recent block which has been acknowledged by 2/3 of the block producers." -} -[/block] +[[warning]] +| +A transaction may be executed, but only when the block containing the transaction is irreversible can you be sure that the operation is final and complete. A block is irreversible if the block number is less than the block number of the last irreversible block (LIB). The last irreversible block (LIB) is the most recent block which has been acknowledged by 2/3 of the block producers. ## Tracking your transaction It is always possible that transactions similar to your transactions are processed while you are monitoring the public mainnet. Ensure that you use ALL identifying fields to look at the right transaction. diff --git a/docs/03_tutorials/monitoring-with-state-history-plugin.md b/docs/03_tutorials/monitoring-with-state-history-plugin.md index f915ddbe..61810dc8 100644 --- a/docs/03_tutorials/monitoring-with-state-history-plugin.md +++ b/docs/03_tutorials/monitoring-with-state-history-plugin.md @@ -1,33 +1,18 @@ In part I [How to monitor state](doc:how-to-monitor-state) we discussed the states of a transaction. In this article we will take a look at monitoring state using the state history plugin. -[block:api-header] -{ - "title": "State History Plugin" -} -[/block] +## State History Plugin The state history plugin replaces the deprecated History Plugin. It caches data in file, rather than in memory, reducing the running costs of node. To use the state history plugin run nodeos with the "--plugin eosio::state_history_plugin" option. See the [state_history_plugin](doc:state_history_plugin) -[block:code] -{ - "codes": [ - { - "code": "nodeos -e -p eosio --plugin eosio::producer_plugin --plugin eosio::chain_api_plugin --plugin eosio::http_plugin --plugin eosio::state_history_plugin --data-dir /Users/mydir/eosio/data --config-dir /Users/mydir/eosio/config --access-control-allow-origin='*' --contracts-console --http-validate-host=false --state-history-dir /shpdata --trace-history --chain-state-history --verbose-http-errors --filter-on='*' --disable-replay-opts >> nodeos.log 2>&1 &", - "language": "shell" - } - ] -} -[/block] +```shell +nodeos -e -p eosio --plugin eosio::producer_plugin --plugin eosio::chain_api_plugin --plugin eosio::http_plugin --plugin eosio::state_history_plugin --data-dir /Users/mydir/eosio/data --config-dir /Users/mydir/eosio/config --access-control-allow-origin='*' --contracts-console --http-validate-host=false --state-history-dir /shpdata --trace-history --chain-state-history --verbose-http-errors --filter-on='*' --disable-replay-opts >> nodeos.log 2>&1 & +``` The nodeos command above adds the state history plugin ``--plugin eosio::state_history_plugin`` storing the chain data in a directory called /shpdata under the --data-dir ``--state-history-dir /shpdata`` storing both trace history and chain-state-history ``--trace-history --chain-state-history``. To run the state history plugin you are required to ``--disable-replay-opts ``. Once nodeos is running the state history plugin, you need to run fill-postgresql (or an alternative connector) to store the chain data in a postgresql database for querying. -[block:callout] -{ - "type": "warning", - "body": "This assumes you have installed [postgesql](https://www.postgresql.org/download/) and have some knowledge of how to use psql", - "title": "postgresql is Required" -} -[/block] +[[warning]] +|postgresql is Required +This assumes you have installed [postgesql](https://www.postgresql.org/download/) and have some knowledge of how to use psql ## fill-postgresql See [fill-postgresql repo](https://github.com/EOSIO/fill-postgresql) @@ -42,92 +27,79 @@ Configuration options: | --trim | | Trim history before irreversible When running it for the first time, use the --create option to create the schema and tables. -[block:code] -{ - "codes": [ - { - "code": "./fill-postgresql --create", - "language": "shell" - } - ] -} -[/block] - -[block:code] -{ - "codes": [ - { - "code": "$ ./fill-postgresql --create\ninfo 2019-01-22T06:25:43.039 thread-0 main.cpp:84 main ] fill-postgresql version not-a-release-1-gaf49f6d\ninfo 2019-01-22T06:25:43.039 thread-0 main.cpp:85 main ] fill-postgresql using configuration file /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/config/config.ini\ninfo 2019-01-22T06:25:43.039 thread-0 main.cpp:86 main ] fill-postgresql data directory is /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/data\ninfo 2019-01-22T06:25:43.039 thread-0 fill_postgresql_plugin:748 session ] connect to postgresql\ninfo 2019-01-22T06:25:43.044 thread-0 fill_postgresql_plugin:762 start ] connect to localhost:8080\ninfo 2019-01-22T06:25:43.053 thread-0 fill_postgresql_plugin:870 create_tables ] create schema \"chain\"\ninfo 2019-01-22T06:25:43.294 thread-0 fill_postgresql_plugin:1207 close_streams ] block 1 - 199\ninfo 2019-01-22T06:25:43.424 thread-0 fill_postgresql_plugin:1207 close_streams ] block 200 - 399\ninfo 2019-01-22T06:25:43.502 thread-0 fill_postgresql_plugin:1207 close_streams ] block 400 - 478\ninfo 2019-01-22T06:25:43.502 thread-0 fill_postgresql_plugin:1148 receive_result ] block 479\n", - "language": "text" - } - ] -} -[/block] +```shell +./fill-postgresql --create +``` + +```text +$ ./fill-postgresql --create +info 2019-01-22T06:25:43.039 thread-0 main.cpp:84 main ] fill-postgresql version not-a-release-1-gaf49f6d +info 2019-01-22T06:25:43.039 thread-0 main.cpp:85 main ] fill-postgresql using configuration file /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/config/config.ini +info 2019-01-22T06:25:43.039 thread-0 main.cpp:86 main ] fill-postgresql data directory is /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/data +info 2019-01-22T06:25:43.039 thread-0 fill_postgresql_plugin:748 session ] connect to postgresql +info 2019-01-22T06:25:43.044 thread-0 fill_postgresql_plugin:762 start ] connect to localhost:8080 +info 2019-01-22T06:25:43.053 thread-0 fill_postgresql_plugin:870 create_tables ] create schema "chain" +info 2019-01-22T06:25:43.294 thread-0 fill_postgresql_plugin:1207 close_streams ] block 1 - 199 +info 2019-01-22T06:25:43.424 thread-0 fill_postgresql_plugin:1207 close_streams ] block 200 - 399 +info 2019-01-22T06:25:43.502 thread-0 fill_postgresql_plugin:1207 close_streams ] block 400 - 478 +info 2019-01-22T06:25:43.502 thread-0 fill_postgresql_plugin:1148 receive_result ] block 479 + +``` To wipe the schema and start over, run with --drop --create. -[block:code] -{ - "codes": [ - { - "code": "./fill-postgresql --drop --create", - "language": "shell" - } - ] -} -[/block] - -[block:code] -{ - "codes": [ - { - "code": "$ ./fill-postgresql --drop --create\ninfo 2019-01-24T09:48:04.324 thread-0 main.cpp:84 main ] fill-postgresql version not-a-release-1-gaf49f6d\ninfo 2019-01-24T09:48:04.324 thread-0 main.cpp:85 main ] fill-postgresql using configuration file /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/config/config.ini\ninfo 2019-01-24T09:48:04.324 thread-0 main.cpp:86 main ] fill-postgresql data directory is /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/data\ninfo 2019-01-24T09:48:04.325 thread-0 fill_postgresql_plugin:748 session ] connect to postgresql\ninfo 2019-01-24T09:48:04.332 thread-0 fill_postgresql_plugin:757 start ] drop schema \"chain\"\ninfo 2019-01-24T09:48:04.416 thread-0 fill_postgresql_plugin:762 start ] connect to localhost:8080\ninfo 2019-01-24T09:48:04.425 thread-0 fill_postgresql_plugin:870 create_tables ] create schema \"chain\"\ninfo 2019-01-24T09:48:04.654 thread-0 fill_postgresql_plugin:1207 close_streams ] block 1 - 199\ninfo 2019-01-24T09:48:04.785 thread-0 fill_postgresql_plugin:1207 close_streams ] block 200 - 399\ninfo 2019-01-24T09:48:04.917 thread-0 fill_postgresql_plugin:1207 close_streams ] block 400 - 599\ninfo 2019-01-24T09:48:05.063 thread-0 fill_postgresql_plugin:1207 close_streams ] block 600 - 799\n", - "language": "text" - } - ] -} -[/block] +```shell +./fill-postgresql --drop --create +``` + +```text +$ ./fill-postgresql --drop --create +info 2019-01-24T09:48:04.324 thread-0 main.cpp:84 main ] fill-postgresql version not-a-release-1-gaf49f6d +info 2019-01-24T09:48:04.324 thread-0 main.cpp:85 main ] fill-postgresql using configuration file /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/config/config.ini +info 2019-01-24T09:48:04.324 thread-0 main.cpp:86 main ] fill-postgresql data directory is /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/data +info 2019-01-24T09:48:04.325 thread-0 fill_postgresql_plugin:748 session ] connect to postgresql +info 2019-01-24T09:48:04.332 thread-0 fill_postgresql_plugin:757 start ] drop schema "chain" +info 2019-01-24T09:48:04.416 thread-0 fill_postgresql_plugin:762 start ] connect to localhost:8080 +info 2019-01-24T09:48:04.425 thread-0 fill_postgresql_plugin:870 create_tables ] create schema "chain" +info 2019-01-24T09:48:04.654 thread-0 fill_postgresql_plugin:1207 close_streams ] block 1 - 199 +info 2019-01-24T09:48:04.785 thread-0 fill_postgresql_plugin:1207 close_streams ] block 200 - 399 +info 2019-01-24T09:48:04.917 thread-0 fill_postgresql_plugin:1207 close_streams ] block 400 - 599 +info 2019-01-24T09:48:05.063 thread-0 fill_postgresql_plugin:1207 close_streams ] block 600 - 799 + +``` To run normally -[block:code] -{ - "codes": [ - { - "code": "./fill-postgresql", - "language": "shell" - } - ] -} -[/block] - -[block:code] -{ - "codes": [ - { - "code": "$ ./fill-postgresql\ninfo 2019-01-24T09:48:48.664 thread-0 main.cpp:84 main ] fill-postgresql version not-a-release-1-gaf49f6d\ninfo 2019-01-24T09:48:48.664 thread-0 main.cpp:85 main ] fill-postgresql using configuration file /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/config/config.ini\ninfo 2019-01-24T09:48:48.664 thread-0 main.cpp:86 main ] fill-postgresql data directory is /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/data\ninfo 2019-01-24T09:48:48.664 thread-0 fill_postgresql_plugin:748 session ] connect to postgresql\ninfo 2019-01-24T09:48:48.675 thread-0 fill_postgresql_plugin:762 start ] connect to localhost:8080\ninfo 2019-01-24T09:48:48.831 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4200 - 4399\ninfo 2019-01-24T09:48:48.967 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4400 - 4599\ninfo 2019-01-24T09:48:49.099 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4600 - 4799\ninfo 2019-01-24T09:48:49.231 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4800 - 4999\ninfo 2019-01-24T09:48:49.361 thread-0 fill_postgresql_plugin:1207 close_streams ] block 5000 - 5199\n", - "language": "text" - } - ] -} -[/block] +```shell +./fill-postgresql +``` + +```text +$ ./fill-postgresql +info 2019-01-24T09:48:48.664 thread-0 main.cpp:84 main ] fill-postgresql version not-a-release-1-gaf49f6d +info 2019-01-24T09:48:48.664 thread-0 main.cpp:85 main ] fill-postgresql using configuration file /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/config/config.ini +info 2019-01-24T09:48:48.664 thread-0 main.cpp:86 main ] fill-postgresql data directory is /Users/philip.halsall/Library/Application Support/eosio/fill-postgresql/data +info 2019-01-24T09:48:48.664 thread-0 fill_postgresql_plugin:748 session ] connect to postgresql +info 2019-01-24T09:48:48.675 thread-0 fill_postgresql_plugin:762 start ] connect to localhost:8080 +info 2019-01-24T09:48:48.831 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4200 - 4399 +info 2019-01-24T09:48:48.967 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4400 - 4599 +info 2019-01-24T09:48:49.099 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4600 - 4799 +info 2019-01-24T09:48:49.231 thread-0 fill_postgresql_plugin:1207 close_streams ] block 4800 - 4999 +info 2019-01-24T09:48:49.361 thread-0 fill_postgresql_plugin:1207 close_streams ] block 5000 - 5199 + +``` This connects to the State History Plugin and populates a Postgresql database with historical data such as action traces, transaction traces, block states, and other interesting things such as contract table changes at every block. It will track real-time updates from nodeos. ## Postgres queries, tables and results To identify the Last Irreversible Block -[block:code] -{ - "codes": [ - { - "code": "eos=> select head, irreversible from chain.fill_status;\n\n\n head | irreversible \n--------+--------------\n 134112 | 134111\n(1 row)", - "language": "sql" - } - ] -} -[/block] - -[block:api-header] -{ - "title": "Examples" -} -[/block] +```sql +eos=> select head, irreversible from chain.fill_status; + + + head | irreversible +--------+-------------- + 134112 | 134111 +(1 row) +``` + +## Examples ### Prerequisites It is assumed that you have a local `nodeos` server connected to an EOSIO blockchain; that this nodeos is using the state history plugin; that an account has been created for `eosio.token`; and that the `eosio.token` contract has been deployed. Completing the [Deploy, Issue, and Transfer Tokens](https://developers.eos.io/eosio-home/docs/token-contract) in the 'Getting Started' section prior to this tutorial will have satisfied these prerequisites. @@ -135,62 +107,65 @@ It is assumed that you have a local `nodeos` server connected to an EOSIO blockc This tutorial uses the transfer action of the `eosio.token` contract. This could be considered as a deposit in one account and a withdrawal in another account. Transfer A to B -[block:code] -{ - "codes": [ - { - "code": "cleos -u http://127.0.0.1:8889/ push action eosio.token issue '[ \"alice\", \"100.0000 SYS\", \"memo\" ]' -p eosio@active", - "language": "shell" - } - ] -} -[/block] +```shell +cleos -u http://127.0.0.1:8889/ push action eosio.token issue '[ "alice", "100.0000 SYS", "memo" ]' -p eosio@active +``` Transfer A to B results in chain.action_trace -[block:code] -{ - "codes": [ - { - "code": "select block_index, transaction_id, action_index, parent_action_index, transaction_status, receipt_receiver, receipt_global_sequence, account, name from chain.action_trace where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff');\n\n\n\n block_index | transaction_id | action_index | parent_action_index | transaction_status | receipt_receiver | receipt_global_sequence | account | name \n-------------+------------------------------------------------------------------+--------------+---------------------+--------------------+------------------+-------------------------+-------------+----------\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 1 | 0 | executed | eosio.token | 2877 | eosio.token | transfer\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 2 | 1 | executed | alice | 2878 | eosio.token | transfer\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 3 | 1 | executed | bob | 2879 | eosio.token | transfer\n(3 rows)\n", - "language": "sql" - } - ] -} -[/block] +```sql +select block_index, transaction_id, action_index, parent_action_index, transaction_status, receipt_receiver, receipt_global_sequence, account, name from chain.action_trace where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff'); + + + + block_index | transaction_id | action_index | parent_action_index | transaction_status | receipt_receiver | receipt_global_sequence | account | name +-------------+------------------------------------------------------------------+--------------+---------------------+--------------------+------------------+-------------------------+-------------+---------- + 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 1 | 0 | executed | eosio.token | 2877 | eosio.token | transfer + 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 2 | 1 | executed | alice | 2878 | eosio.token | transfer + 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 3 | 1 | executed | bob | 2879 | eosio.token | transfer +(3 rows) + +``` Transfer A to B also results in chain.transaction_trace -[block:code] -{ - "codes": [ - { - "code": "select block_index, transaction_id, status from chain.transaction_trace where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff');\n\n\n\n block_index | transaction_id | status \n-------------+------------------------------------------------------------------+----------\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | executed\n", - "language": "sql" - } - ] -} -[/block] + +```sql +select block_index, transaction_id, status from chain.transaction_trace where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff'); + + + + block_index | transaction_id | status +-------------+------------------------------------------------------------------+---------- + 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | executed + +``` Transfer A to B also results in chain.action_trace_auth_sequence -[block:code] -{ - "codes": [ - { - "code": "select * from chain.action_trace_auth_sequence where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff');\n\n\n\n block_index | transaction_id | action_index | index | transaction_status | account | sequence \n-------------+------------------------------------------------------------------+--------------+-------+--------------------+---------+----------\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 2 | 1 | executed | alice | 5\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 3 | 1 | executed | alice | 6\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 1 | 1 | executed | alice | 4\n(3 rows)\n", - "language": "sql" - } - ] -} -[/block] +```sql +select * from chain.action_trace_auth_sequence where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff'); + + + + block_index | transaction_id | action_index | index | transaction_status | account | sequence +-------------+------------------------------------------------------------------+--------------+-------+--------------------+---------+---------- + 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 2 | 1 | executed | alice | 5 + 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 3 | 1 | executed | alice | 6 + 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 1 | 1 | executed | alice | 4 +(3 rows) + +``` Transfer A to B also results in chain.action_trace_authorization -[block:code] -{ - "codes": [ - { - "code": "select * from chain.action_trace_authorization where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff');\n\n\n block_index | transaction_id | action_index | index | transaction_status | actor | permission \n-------------+------------------------------------------------------------------+--------------+-------+--------------------+-------+------------\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 2 | 1 | executed | alice | active\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 3 | 1 | executed | alice | active\n 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 1 | 1 | executed | alice | active\n(3 rows)\n", - "language": "sql" - } - ] -} -[/block] +```sql +select * from chain.action_trace_authorization where transaction_id = UPPER('ba42b4b6c7a41e58c29a7cd90bb3dc3b554d2ce8ed9733dcf16bf80ad1d729ff'); + + + block_index | transaction_id | action_index | index | transaction_status | actor | permission +-------------+------------------------------------------------------------------+--------------+-------+--------------------+-------+------------ + 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 2 | 1 | executed | alice | active + 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 3 | 1 | executed | alice | active + 2864 | BA42B4B6C7A41E58C29A7CD90BB3DC3B554D2CE8ED9733DCF16BF80AD1D729FF | 1 | 1 | executed | alice | active +(3 rows) + +``` # Alternatives + [Demux](https://github.com/EOSIO/demux-js) is an alternative that piggybacks off the RPC API. In part III [Monitoring with 3rd Party plugins](doc:monitoring-with-3rd-party) we will discuss 3rd party plugins which may also be used for monitoring state. \ No newline at end of file From 432a020e07652f42a4d42dd5008891097d7e3736 Mon Sep 17 00:00:00 2001 From: Rakesh Ghatvisave Date: Thu, 20 Feb 2020 20:06:20 +0800 Subject: [PATCH 4/5] Editorial and peer review --- .../monitoring-with-3rd-party-plugins.md | 97 +++++++++++-------- 1 file changed, 55 insertions(+), 42 deletions(-) diff --git a/docs/03_tutorials/monitoring-with-3rd-party-plugins.md b/docs/03_tutorials/monitoring-with-3rd-party-plugins.md index 73d788fd..dd7f0c09 100644 --- a/docs/03_tutorials/monitoring-with-3rd-party-plugins.md +++ b/docs/03_tutorials/monitoring-with-3rd-party-plugins.md @@ -5,13 +5,13 @@ "body": "The resources listed below are developed, offered, and maintained by third-parties and not by block.one. Providing information, material, or commentaries about such third-party resources does not mean we endorse or recommend any of these resources. We are not responsible, and disclaim any responsibility or liability, for your use of or reliance on any of these resources. Third-party resources may be updated, changed or terminated at any time, so the information below may be out of date or inaccurate. USAGE AND RELIANCE IS ENTIRELY AT YOUR OWN RISK." } [/block] -In part I [How to monitor state](doc:how-to-monitor-state) we discussed the states of a transaction. +In part I, [How to monitor state](doc:how-to-monitor-state), we discussed the states of a transaction. -In part II [Monitoring with State History Plugin](doc:monitoring-with-state-history) we discussed monitoring state using the state history plugin. +In part II, [Monitoring with State History Plugin](doc:monitoring-with-state-history), we discussed monitoring state using the state history plugin. -In this article we will take a brief look at some of the 3rd party options which are available for monitoring the state of the blockchain. There may be other solutions for monitoring state not present in this document so please research what is currently the best option for you. +In this article, we will take a brief look at some of the 3rd party options which are available for monitoring the state of the blockchain. There may be other solutions for monitoring state not present in this document, so please research what is currently the best option for you. -This information comes from `https://github.com/EOSIO/eos/blob/master/plugins/COMMUNITY.md`. These are plugins developed by the community, in no particular order, which can be added to Nodeos by compiling Nodeos with those plugins included. In this article we only show the plugins related to monitoring transactions. +This information here is extractd from the [Community Plugin List](https://github.com/EOSIO/eos/blob/master/plugins/COMMUNITY.md) page. These are plugins developed by the community, in no particular order, which can be added to Nodeos by compiling Nodeos with those plugins included. In this article, we only show the plugins related to monitoring transactions. | Description | URL | | ----------- | --- | @@ -32,7 +32,7 @@ This information comes from `https://github.com/EOSIO/eos/blob/master/plugins/CO [block:callout] { "type": "warning", - "body": "These are in no particular order.\nDescriptions of each tool have been provided by the tool authors." + "body": "These are in no particular order. Descriptions of each tool have been provided by the tool authors." } [/block] @@ -41,76 +41,80 @@ This information comes from `https://github.com/EOSIO/eos/blob/master/plugins/CO "title": "Elastic Search Plugin" } [/block] -The Elasticsearch plugin is a nodeos plugin for archiving blockchain data into Elasticsearch. It is designed to transfer nodeos data structures to json documents, and to post these documents to an Elasticsearch cluster, using bulk requests. It speeds up the processing progress by using a thread pool to handle the serialisation and the bulk request jobs. +The Elasticsearch plugin is a nodeos plugin for archiving blockchain data into Elasticsearch. It is designed to transfer nodeos data structures to JSON documents, and to post these documents to an Elasticsearch cluster using bulk requests. It speeds up the processing progress by using a thread pool to handle the serialization and the bulk request jobs. + +For more information, see the [Elastic Search Plugin](https://github.com/EOSLaoMao/elasticsearch_plugin) page on Github. -Please refer to https://github.com/EOSLaoMao/elasticsearch_plugin for more information. [block:api-header] { "title": "Kafka Plugin" } [/block] -EOSIO Kafka Plugin is used to receive the transaction data from the blockchain and to send out the transaction through a kafka producer. Developers can receive the transaction data through a kafka consumer in the back-end application. +The EOSIO Kafka Plugin is used to receive the transaction data from the blockchain and to send out the transaction through a kafka producer. Developers can receive the transaction data through a kafka consumer in the back-end application. + +For more information, see the [Kafka Plugin](https://github.com/TP-Lab/kafka_plugin) page on Github. -Please refer to https://github.com/TP-Lab/kafka_plugin for more information. [block:api-header] { - "title": "Eosio MySQLdb Plugin" + "title": "EOSIO MySQLdb Plugin" } [/block] The mysqldb_plugin is a plug-in to store block, transaction and action data in a MySQL database. -Please refer to https://github.com/eosBLACK/eosio_mysqldb_plugin for more information. +For more information, see the [EOSIO MySQL Plugin](https://github.com/eosBLACK/eosio_mysqldb_plugin) page on Github. + + [block:api-header] { - "title": "SQL" + "title": "SQL Plugin" } [/block] -Please refer to https://github.com/asiniscalchi/eosio_sql_plugin for more information. +For more information, see the [SQL Plugin](https://github.com/asiniscalchi/eosio_sql_plugin) page on Github. + + + [block:api-header] { - "title": "EOSIO Watcher" + "title": "EOSIO Watcher Plugin" } [/block] -EOSIO Watcher plugin is a simple plugin to watch for specific actions on chain and send them as HTTP POSTs to a url in real-time. +The EOSIO Watcher plugin is a simple plugin to watch for specific actions on chain and send them as HTTP POSTs to a url in real-time. + +For more information, see the [EOSIO Watcher Plugin](https://github.com/eosauthority/eosio-watcher-plugin) page on Github. + -Please refer to https://github.com/eosauthority/eosio-watcher-plugin for more information. [block:api-header] { - "title": "ZMQ / history" + "title": "ZMQ Plugin" } [/block] The ZMQ / history plugin collects EOS action traces, converting them into JSON format, and exporting them via a ZeroMQ socket. It may export all action traces, or only those relevant to a number of whitelisted accounts. A receiver for this data can be implemented in any modern programming language that supports the ZeroMQ library. -Please refer to https://github.com/cc32d9/eos_zmq_plugin for more information. +For more information, see the [ZMQ Plugin](https://github.com/cc32d9/eos_zmq_plugin) page on Github. + [block:api-header] { "title": "ZMQ Light History API" } [/block] -The ZMQ Light History API receives data from the `nodeos` ZMQ / history plugin. This stores the latest state of each account and account token balances. It is called Light because does not store historic data. +The ZMQ Light History API receives data from the `nodeos` ZMQ / history plugin. This stores the latest state of each account and account token balances. It is called **Light** because it does not store historic data. -Please refer to https://github.com/cc32d9/eos_zmq_light_api for more information. -[block:api-header] -{ - "title": "Mongo History API" -} -[/block] -The Mongo History API Plugin was developed as a gateway to the MongoDB Plugin which may be used in place of the history plugin and state history plugin. The API requests are the same, however, there are some additional API requests. - - Please refer to https://github.com/CryptoLions/EOS-mongo-history-API for more information and to https://history.cryptolions.io/ for documentation. +For more information, see the [ZMQ Light History API](https://github.com/cc32d9/eosio_light_api) page on Github. + + [block:api-header] { "title": "Chintai ZMQ Watcher Plugin" } [/block] -This plugin is designed to allow a blockchain application developer to easily monitor actions from the blockchain. It is originally forked from EOSIO Watcher but modified to use ZeroMQ instead of HTTP requests, as using ZMQ guarantees ordering. When building a blockchain application that requires an exact ordering of block events and a guarantee that no blocks will be (easily) lost during a replay, a message queue like ZeroMQ is a great solution. +The Chintai ZMQ Watcher plugin is designed to allow a blockchain application developer to easily monitor actions from the blockchain. It is originally forked from EOSIO Watcher, but modified to use ZeroMQ instead of HTTP requests, as using ZMQ guarantees ordering. When building a blockchain application that requires an exact ordering of block events and a guarantee that no blocks will be (easily) lost during a replay, a message queue like ZeroMQ is a great solution. - While this plugin does allow for the monitoring of on-chain events, it does not include a ZeroMQ receiver component which would receive the events from the plugin. The receiver is generally proprietary to the specific implementation of your blockchain application's web backend, however, there are ZeroMQ libraries available for most programming languages. This plugin puts blockchain events onto a ZeroMQ messaging queue, and any language using the ZeroMQ library can connect to the queue and process events from it. - This plugin listens to head block signals and is therefore subject to periodic microforks in the network. The handling of these forks must be done in your ZMQ receiver component. If you prefer, you can modify the plugin to use irreversible blocks, however, there would be a 2-3 minute delay in receiving transactions. - The plugin is currently setup to filter on actions specific to the Chintai smart contract. The filtering function should be updated to your dApps's specific implementation. -Please refer to https://github.com/acoutts/chintai-zeromq-watcher-plugin for more information +For more information, see the [Chintai ZMQ Watcher Plugin](https://github.com/acoutts/chintai-zeromq-watcher-plugin) page on Github. [block:api-header] { "title": "State History API" @@ -118,35 +122,43 @@ Please refer to https://github.com/acoutts/chintai-zeromq-watcher-plugin for mor [/block] The State History API is intended as a public API interface which matches the history plugin api, but uses the data stored by the state history plugin in a Postgresql database, by using fill-postgresql, please see [Monitoring with State History Plugin](doc:monitoring-with-state-history) -Please refer to https://github.com/acoutts/EOS-state-history-API for more information. +For more information, see the [State History API](https://github.com/acoutts/EOS-state-history-API) page on Github. [block:api-header] { - "title": "Chronicle" + "title": "Chronicle Project" } [/block] -The chronicle project aims to build a toolset for EOSIO history databases. Currently Chronicle Receiver is released, and this tool will read the data from the state history plugin, decoding it into JSON format, and exporting this json to downstream consumers. +The Chronicle project aims to build a toolset for EOSIO history databases. Currently, Chronicle Receiver is released, and this tool will read the data from the state history plugin, decoding it into JSON format, and exporting this JSON to downstream consumers. + +For more information, see the [Chronicle Project](https://github.com/EOSChronicleProject/eos-chronicle) page on Github. -Please refer to https://github.com/EOSChronicleProject/eos-chronicle for more information. [block:api-header] { - "title": "Hyperion History" + "title": "Hyperion History API" } [/block] -Hyperion History uses a multi-threaded indexer to extract data from the state history plugin, storing data in an Elasticsearch cluster. Optimising the data structures used to store data has reduced the amount of storage required by up to 85%, and has meant lower response times when responding to queries whilst also reducing data transfer overhead. +The Hyperion History API uses a multi-threaded indexer to extract data from the state history plugin, storing data in an Elasticsearch cluster. Optimizing the data structures used to store data has reduced the amount of storage required by up to 85%, and has meant lower response times when responding to queries whilst also reducing data transfer overhead. Access to this data is provided by a javascript library, the Hyperion HTTP API, and next steps include implementing a WebSocket API for action streaming. -Please refer to https://medium.com/@eosriobrazil/presenting-hyperion-history-api-solution-f8a8fda5865b for more information. -The source code can be located here https://github.com/eosrio/Hyperion-History-API -For more information about the Hyperion HTTP API Javascript library please refer to https://eos.hyperion.eosrio.io/v2/docs/index.html and the source code for this library can be found here https://github.com/eoscafe/hyperion-api +For more information on **Hyperion History**, see the [Hyperion History](https://github.com/eosrio/Hyperion-History-API) page on Github. + +For more information on the **Hyperion HTTP API Javascript library**, see the [Hyperion HTTP API](https://github.com/eoscafe/hyperion-api) page on Github. + +For additional information, see this [Medium](https://medium.com/@eosriobrazil/presenting-hyperion-history-api-solution-f8a8fda5865b) article. + + + + [block:api-header] { - "title": "eos zmq plugin receiver" + "title": "EOS ZMP plugin receiver" } [/block] This is a set of tools written in Perl for receiving and processing the EOSIO network events exported by the ZMQ / history plugin (see above). They can be useful for dispatching and troubleshooting message flow, and there is a ready-made solution for storing action data in MySQL database. -Please refer to https://github.com/cc32d9/eos_zmq_plugin_receiver for more information. +For more information, see the [EOS ZMQ Plugin](https://github.com/cc32d9/eos_zmq_plugin_receiver) page on Github. + [block:api-header] { "title": "dfuse" @@ -154,4 +166,5 @@ Please refer to https://github.com/cc32d9/eos_zmq_plugin_receiver for more infor [/block] dfuse offers powerful streaming APIs, GraphQL queries and subscriptions, server-side navigation of forks, and some of the deepest insight of on-chain data, like action-level database operations. dfuse also allows querying of the blockchain state (contract tables) at any past block height, providing a consistent view of tables even when they hold millions of rows. dfuse Search allows you to find transactions through the full blockchain history in under a second, with granular search criterias. dfuse Events allows you to index special fields directly from your smart contracts, making them available instantaneously through dfuse Search. -dfuse is a commercial offering by the team at EOS Canada. See overview of dfuse technology at https://www.dfuse.io/en/technology . Product documentation is available at https://docs.dfuse.io/ \ No newline at end of file +dfuse is a commercial offering by the team at EOS Canada. For product overview, see the [dfuse technology] +(https://www.dfuse.io/en/technology) page and for product documentation, see [dfuse API Documentation](https://docs.dfuse.io/). From 3e1209d123ee38da535046d7a356c6bb9ff4af45 Mon Sep 17 00:00:00 2001 From: Rakesh Ghatvisave Date: Mon, 2 Mar 2020 14:08:56 +0800 Subject: [PATCH 5/5] Update monitoring-with-3rd-party-plugins.md --- .../monitoring-with-3rd-party-plugins.md | 109 ++++-------------- 1 file changed, 24 insertions(+), 85 deletions(-) diff --git a/docs/03_tutorials/monitoring-with-3rd-party-plugins.md b/docs/03_tutorials/monitoring-with-3rd-party-plugins.md index dd7f0c09..5df8d532 100644 --- a/docs/03_tutorials/monitoring-with-3rd-party-plugins.md +++ b/docs/03_tutorials/monitoring-with-3rd-party-plugins.md @@ -1,10 +1,6 @@ +[[info | Warning]] +| The resources listed below are developed, offered, and maintained by third-parties and not by block.one. Providing information, material, or commentaries about such third-party resources does not mean we endorse or recommend any of these resources. We are not responsible, and disclaim any responsibility or liability, for your use of or reliance on any of these resources. Third-party resources may be updated, changed or terminated at any time, so the information below may be out of date or inaccurate. USAGE AND RELIANCE IS ENTIRELY AT YOUR OWN RISK. -[block:callout] -{ - "type": "warning", - "body": "The resources listed below are developed, offered, and maintained by third-parties and not by block.one. Providing information, material, or commentaries about such third-party resources does not mean we endorse or recommend any of these resources. We are not responsible, and disclaim any responsibility or liability, for your use of or reliance on any of these resources. Third-party resources may be updated, changed or terminated at any time, so the information below may be out of date or inaccurate. USAGE AND RELIANCE IS ENTIRELY AT YOUR OWN RISK." -} -[/block] In part I, [How to monitor state](doc:how-to-monitor-state), we discussed the states of a transaction. In part II, [Monitoring with State History Plugin](doc:monitoring-with-state-history), we discussed monitoring state using the state history plugin. @@ -29,85 +25,48 @@ This information here is extractd from the [Community Plugin List](https://githu | Hyperion History | https://github.com/eosrio/Hyperion-History-API | | eos zmq plugin receiver | https://github.com/cc32d9/eos_zmq_plugin_receiver | | dfuse (commercial) | https://www.dfuse.io/en/technology | -[block:callout] -{ - "type": "warning", - "body": "These are in no particular order. Descriptions of each tool have been provided by the tool authors." -} -[/block] - -[block:api-header] -{ - "title": "Elastic Search Plugin" -} -[/block] + +[[info | Warning]] +| These are in no particular order. Descriptions of each tool have been provided by the tool authors + +## Elastic Search Plugin The Elasticsearch plugin is a nodeos plugin for archiving blockchain data into Elasticsearch. It is designed to transfer nodeos data structures to JSON documents, and to post these documents to an Elasticsearch cluster using bulk requests. It speeds up the processing progress by using a thread pool to handle the serialization and the bulk request jobs. For more information, see the [Elastic Search Plugin](https://github.com/EOSLaoMao/elasticsearch_plugin) page on Github. -[block:api-header] -{ - "title": "Kafka Plugin" -} -[/block] +## Kafka Plugin The EOSIO Kafka Plugin is used to receive the transaction data from the blockchain and to send out the transaction through a kafka producer. Developers can receive the transaction data through a kafka consumer in the back-end application. For more information, see the [Kafka Plugin](https://github.com/TP-Lab/kafka_plugin) page on Github. -[block:api-header] -{ - "title": "EOSIO MySQLdb Plugin" -} -[/block] + +## EOSIO MySQLdb Plugin The mysqldb_plugin is a plug-in to store block, transaction and action data in a MySQL database. For more information, see the [EOSIO MySQL Plugin](https://github.com/eosBLACK/eosio_mysqldb_plugin) page on Github. -[block:api-header] -{ - "title": "SQL Plugin" -} -[/block] +## SQL Plugin For more information, see the [SQL Plugin](https://github.com/asiniscalchi/eosio_sql_plugin) page on Github. - -[block:api-header] -{ - "title": "EOSIO Watcher Plugin" -} -[/block] +## EOSIO Watcher Plugin The EOSIO Watcher plugin is a simple plugin to watch for specific actions on chain and send them as HTTP POSTs to a url in real-time. For more information, see the [EOSIO Watcher Plugin](https://github.com/eosauthority/eosio-watcher-plugin) page on Github. - -[block:api-header] -{ - "title": "ZMQ Plugin" -} -[/block] +## ZMQ Plugin The ZMQ / history plugin collects EOS action traces, converting them into JSON format, and exporting them via a ZeroMQ socket. It may export all action traces, or only those relevant to a number of whitelisted accounts. A receiver for this data can be implemented in any modern programming language that supports the ZeroMQ library. For more information, see the [ZMQ Plugin](https://github.com/cc32d9/eos_zmq_plugin) page on Github. -[block:api-header] -{ - "title": "ZMQ Light History API" -} -[/block] +## ZMQ Light History API The ZMQ Light History API receives data from the `nodeos` ZMQ / history plugin. This stores the latest state of each account and account token balances. It is called **Light** because it does not store historic data. - For more information, see the [ZMQ Light History API](https://github.com/cc32d9/eosio_light_api) page on Github. - - -[block:api-header] -{ - "title": "Chintai ZMQ Watcher Plugin" -} -[/block] + + +## Chintai ZMQ Watcher Plugin The Chintai ZMQ Watcher plugin is designed to allow a blockchain application developer to easily monitor actions from the blockchain. It is originally forked from EOSIO Watcher, but modified to use ZeroMQ instead of HTTP requests, as using ZMQ guarantees ordering. When building a blockchain application that requires an exact ordering of block events and a guarantee that no blocks will be (easily) lost during a replay, a message queue like ZeroMQ is a great solution. - While this plugin does allow for the monitoring of on-chain events, it does not include a ZeroMQ receiver component which would receive the events from the plugin. The receiver is generally proprietary to the specific implementation of your blockchain application's web backend, however, there are ZeroMQ libraries available for most programming languages. This plugin puts blockchain events onto a ZeroMQ messaging queue, and any language using the ZeroMQ library can connect to the queue and process events from it. @@ -115,28 +74,18 @@ The Chintai ZMQ Watcher plugin is designed to allow a blockchain application dev - The plugin is currently setup to filter on actions specific to the Chintai smart contract. The filtering function should be updated to your dApps's specific implementation. For more information, see the [Chintai ZMQ Watcher Plugin](https://github.com/acoutts/chintai-zeromq-watcher-plugin) page on Github. -[block:api-header] -{ - "title": "State History API" -} -[/block] + +## State History API The State History API is intended as a public API interface which matches the history plugin api, but uses the data stored by the state history plugin in a Postgresql database, by using fill-postgresql, please see [Monitoring with State History Plugin](doc:monitoring-with-state-history) For more information, see the [State History API](https://github.com/acoutts/EOS-state-history-API) page on Github. -[block:api-header] -{ - "title": "Chronicle Project" -} -[/block] + +## Chronicle Project The Chronicle project aims to build a toolset for EOSIO history databases. Currently, Chronicle Receiver is released, and this tool will read the data from the state history plugin, decoding it into JSON format, and exporting this JSON to downstream consumers. For more information, see the [Chronicle Project](https://github.com/EOSChronicleProject/eos-chronicle) page on Github. -[block:api-header] -{ - "title": "Hyperion History API" -} -[/block] +## Hyperion History API The Hyperion History API uses a multi-threaded indexer to extract data from the state history plugin, storing data in an Elasticsearch cluster. Optimizing the data structures used to store data has reduced the amount of storage required by up to 85%, and has meant lower response times when responding to queries whilst also reducing data transfer overhead. Access to this data is provided by a javascript library, the Hyperion HTTP API, and next steps include implementing a WebSocket API for action streaming. @@ -148,22 +97,12 @@ For more information on the **Hyperion HTTP API Javascript library**, see the [H For additional information, see this [Medium](https://medium.com/@eosriobrazil/presenting-hyperion-history-api-solution-f8a8fda5865b) article. - - -[block:api-header] -{ - "title": "EOS ZMP plugin receiver" -} -[/block] +## EOS ZMP plugin receiver This is a set of tools written in Perl for receiving and processing the EOSIO network events exported by the ZMQ / history plugin (see above). They can be useful for dispatching and troubleshooting message flow, and there is a ready-made solution for storing action data in MySQL database. For more information, see the [EOS ZMQ Plugin](https://github.com/cc32d9/eos_zmq_plugin_receiver) page on Github. -[block:api-header] -{ - "title": "dfuse" -} -[/block] +## dfuse dfuse offers powerful streaming APIs, GraphQL queries and subscriptions, server-side navigation of forks, and some of the deepest insight of on-chain data, like action-level database operations. dfuse also allows querying of the blockchain state (contract tables) at any past block height, providing a consistent view of tables even when they hold millions of rows. dfuse Search allows you to find transactions through the full blockchain history in under a second, with granular search criterias. dfuse Events allows you to index special fields directly from your smart contracts, making them available instantaneously through dfuse Search. dfuse is a commercial offering by the team at EOS Canada. For product overview, see the [dfuse technology]