Monthly data update — April 04
As we started to work on signull.io, we discovered how complex working with crypto data is, in reality. This blog will explain some basic methodology behind signull.io and (hopefully) will be published monthly. In case anything is unexplained or unclear, join our Discord.
Issues
As the end of Q1 approached, we faced some major issues with our app. Most of them were caused by the poor performance of our database. We need to spend some time optimizing it because if the number of tokens we have will increase, we might face this issue more often. For now, we just increased the HW resources as a temporal fix until some optimization is done.
Some small outages were caused by issues on the data provider side. This also led to a discussion on how to prevent this kind of issue in the future, however, there is no “simple” solution for this so we have planned a few sessions about the data source in late Q2.
Major Changes
Last month, we deployed some major improvements to the data ingress. The biggest (and most promising) change was including BTC and ETH into the list of supported quote currencies. It sounds easy, but under the hood, we have to create a system that stores the historical exchange rate between USD, BTC or ETH and the required token. For BTC and ETH the job was quite simple — we are using BTC and ETH for charting so no matter what we always store prices in USD, BTC and ETH. But our internal analysis shows, we need to go beyond their three currencies (and “USD or ETH-like” stable coins) and this step will require additional work to store the exchange rates so we will always know which currency was used and what was the exact exchange rate. An increasing the number of quote currencies is planned for Q2.
As a side effect of adding BTC and ETH as quote currencies, some tokens changed their primary quote currency to BTC/ETH based on the volume impact %. This leads to a more accurate charting experience.
The second biggest improvement was the Data Healer. Initially, we wanted to fix weird candles (not connected candles, extreme wicks and more) using another service that will scan our database and locate potentially weird candles and try to replace them with corrected values loaded from the external data source. Unfortunately, this approach results in a heavy data load — scanning ~73 500 000 candles took too long even though we used 10 concurrent scans. We did not want to go higher on the concurrency, because we did not want to affect the performance of the database. The v0 data healer was rewritten a few times. The approach has changed a lot but ended with insufficient results. So we have decided to get rid of this component and use the data ingress for this instead.
Right now, the data ingress is wiping all one-minute candles periodically and replacing them with new ones. This functionality will be expanded to all time frames during Q2.
Numbers
As usual, I would like to share some numbers and comment on them.
The reason for our increase of tokens with Marketcap is unknown to me. The increase in tokens probably does not cause the increase in tokens with the market cap. This is one of our key metrics and will be improved in the future, I believe we can hit 50 % in Q2.
The increase of tokens without Marketcap from 6.9k to 8.5k is caused by adding BTC, ETH and some other USD-like stable coins into our “supported quote currencies” list. I believe we can get approx. 3/4k more tokens in Q2, without major changes of the data ingress, which will enable >80 % ratio of Tokens with Charts.
Also, we have added a new metric — Percentage of tokens with complete history — it’s used when we decide to wipe out all data we have in our database (e.g. due to some glitch or unknown errors). This should always stay over 99 % and tells how many tokens have the whole history of candles loaded in our database.
Future
The main focus during Q2 will be on improving the data quality (healing all timeframes) and also expanding the number of tokens with charts.
Also, as mentioned, we would like to spend some time tweaking the database performance, so we will be ready to handle more tokens without any huge performance impacts.
At the end of the last data update, I promised we should know more about improving market caps, however, this activity has been put on hold for a few weeks and should be finished next month.
As I mentioned above, in case of any questions about our methodology, join our Discord or just try our app :)
