Spy on your Ethereum node events
Retrieving Ethereum node events has never been this fast.
TL;DR
If you’re reading this post because you want to retrieve all the past and future events from your Ethereum node you must check out EthLogSpy on Github.
Events retrieval bottleneck and a simple solution
Retrieving events from the Ethereum Blockchain is painfully slow due to the amount of data you need to query and the sequential nature of the chain (even more so if you have an archive node instead of just a «mainnet-synced» one).
This is a bottleneck in developing dApps because we would like to interact with our Smart Contracts events as fast as possibile, especially if we need to retrieve historical data about them (eg. all the transactions events that our Smart Contract emitted).
This problem was brought up to me by a colleague that was searching for a open source solution that could allow him and his complex dApp to overcome the events retrieval slowness, pointing at how a service provider like Infura does not suffer from this problem because they implemented “Ferryman”, which is a “caching layer” for all the Ethereum events.
Since no more info has been given to us about this layer (the information above is from April 2019) I decided to help my friend by developing EthLogSpy.
How does it work?
EthLogSpy is a Golang reverse proxy for Ethereum nodes that stores logs information for a faster retrieval. Basically it just exposes the given node (setup in the configuration file) to the world by just reverse proxying every JSON RPC call made to it, except for all the log related ones.
In fact, when it receives a eth_getLogs call, instead of calling the node it searches for the requested logs in the database (MongoDB) or in the cache (Redis) and returns the result in the same format that the Ethereum node would use, basically faking its response.
When the spy starts, it starts syncing all the logs (yes, some time it’s still required but as you need to sync your node might as well wait some time to sync your logs too) and then subscribes to the Blockchain head so that every new log that gets emitted by our node is instantly stored on the database.
It has full support for both HTTP/HTTPS and WS/WSS connections towards your node.
Web3 full compatibility
By using this approach EthLogSpy is 100% compatible with the web3 library; this means that you can retrieve everything from your node by simply pointing to your spy instance instead of the node one:
The above example uses web3js, but you can obviously use other web3 libraries such as web3py.
Install with Docker Compose
The simples way to install EthLogSpy is by using the docker-compose command (for all the other installation methods please refer to the GitHub repository).
I will not cover how to install docker and docker-compose, you can easily find some tutorial that will explain it to you better than I could.
These are the commands you need to run in order to start a fresh EthLogSpy default installation (if you need to override the default configuration, before running the build command edit the config.yml file in the ethlogspy/configs directory):
git clone https://github.com/PaoloRollo/ethlogspy
cd ethlogspy
docker-compose build
nohup docker-compose up &This will spin up for you the EthLogSpy, MongoDB and Redis containers ready to use.
If you want to help me support more installation methods, please contribute to the repository (all the help is accepted)!
Security considerations
As described above, EthLogSpy requires a running Ethereum node, a MongoDB database and a Redis instance. If you’re running everything on the same machine (eg. node on 8545, MongoDB on 27017, Redis on 6379 and EthLogSpy on 8080) please make sure to not expose all the ports to the public and just leave the EthLogSpy one open or if you do, edit the configuration files properly to setup some authentication to both the db and the cache.
By using EthLogSpy, the database becomes the source of truth for all the logs that can be retrieved through your node, so make sure that you do not expose its port publicly without any security mechanism, do not delete the persistent volume where the db is stored or the db itself if you don’t want to perform a new log sync and that you do not manually edit any of the logs inside the database: this could cause some inconsistency between the real Ethereum Blockchain state and your EthLogSpy instance and make your dApp users question about what’s going on.
Thanks for reading!
Thanks for reading this, if you want to contribute please follow the link for the GitHub repo:
