Commit ae26e32a authored by squbs's avatar squbs

electrumx 1.4.4p2

parents
*/__pycache__/
.cache/
tests/*/__pycache__/
.pytest_cache/
*/*~
*.#*
*#
*~
docs/_build
/build
/dist
/electrumx.egg-info
sudo: required
dist: trusty
language: python
before_install:
- sudo add-apt-repository -y ppa:giskou/librocksdb
- sudo apt-get -qq update
- sudo apt-get install -yq libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev
- sudo apt-get install -yq --allow-unauthenticated librocksdb
- wget https://launchpad.net/ubuntu/+archive/primary/+files/leveldb_1.20.orig.tar.gz
- tar -xzvf leveldb_1.20.orig.tar.gz
- pushd leveldb-1.20 && make && sudo mv out-shared/libleveldb.* /usr/local/lib && sudo cp -R include/leveldb /usr/local/include && sudo ldconfig && popd
python:
- "3.6"
- "nightly"
# command to install dependencies
install:
- pip install aiohttp
- pip install aiorpcX
- pip install ecdsa
- pip install plyvel
- pip install pycodestyle
- pip install pylru
- pip install pyrocksdb
- pip install pytest-cov
- pip install Sphinx
# hashes
- pip install tribus-hash
- pip install blake256
- pip install x11_hash
- pip install xevan_hash
- pip install quark_hash
- pip install git+https://github.com/goacoincore/neoscrypt
# command to run tests
script:
- pytest --cov=server --cov=lib --cov=wallet
- pycodestyle server/*.py lib/*.py
- sh -c "cd docs && make html"
# Dont report coverage from nightly
after_success:
- if [[ $(python3 -V 2>&1) == *"Python 3.6"* ]]; then
pip install python-coveralls;
coveralls;
fi
Copyright (c) 2016-2017, Neil Booth
All rights reserved.
The MIT License (MIT)
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
.. image:: https://travis-ci.org/kyuupichan/electrumx.svg?branch=master
:target: https://travis-ci.org/kyuupichan/electrumx
.. image:: https://coveralls.io/repos/github/kyuupichan/electrumx/badge.svg
:target: https://coveralls.io/github/kyuupichan/electrumx
===============================================
ElectrumX - Reimplementation of electrum-server
===============================================
For a future network with bigger blocks.
:Licence: MIT
:Language: Python (>= 3.6)
:Author: Neil Booth
Documentation
=============
See `readthedocs <https://electrumx.readthedocs.io/>`_.
#!/usr/bin/env python3
#
# Copyright (c) 2017, Neil Booth
#
# All rights reserved.
#
# See the file "LICENCE" for information about the copyright
# and warranty status of this software.
'''Script to compact the history database. This should save space and
will reset the flush counter to a low number, avoiding overflow when
the flush count reaches 65,536.
This needs to lock the database so ElectrumX must not be running -
shut it down cleanly first.
It is recommended you run this script with the same environment as
ElectrumX. However it is intended to be runnable with just
DB_DIRECTORY and COIN set (COIN defaults as for ElectrumX).
If you use daemon tools, you might run this script like so:
envdir /path/to/the/environment/directory ./compact_history.py
Depending on your hardware this script may take up to 6 hours to
complete; it logs progress regularly.
Compaction can be interrupted and restarted harmlessly and will pick
up where it left off. However, if you restart ElectrumX without
running the compaction to completion, it will not benefit and
subsequent compactions will restart from the beginning.
'''
import logging
import sys
import traceback
from os import environ
from server.env import Env
from server.db import DB
def compact_history():
if sys.version_info < (3, 6):
raise RuntimeError('Python >= 3.6 is required to run ElectrumX')
environ['DAEMON_URL'] = '' # Avoid Env erroring out
env = Env()
db = DB(env)
assert not db.first_sync
history = db.history
# Continue where we left off, if interrupted
if history.comp_cursor == -1:
history.comp_cursor = 0
history.comp_flush_count = max(history.comp_flush_count, 1)
limit = 8 * 1000 * 1000
while history.comp_cursor != -1:
history._compact_history(limit)
# When completed also update the UTXO flush count
db.set_flush_count(history.flush_count)
def main():
logging.basicConfig(level=logging.INFO)
logging.info('Starting history compaction...')
try:
compact_history()
except Exception:
traceback.print_exc()
logging.critical('History compaction terminated abnormally')
else:
logging.info('History compaction complete')
if __name__ == '__main__':
main()
______________________________ _____ ____ __. _________
/ _____/\__ ___/\______ \ / _ \ | |/ _| / _____/
\_____ \ | | | _/ / /_\ \| < \_____ \
/ \ | | | | \/ | \ | \ / \
/_______ / |____| |____|_ /\____|__ /____|__ \/_______ /
\/ \/ \/ \/ \/
STRAKS Electrum Server
----------------------
$SERVER_SUBVERSION
$DAEMON_SUBVERSION
----------------------
Website: https://straks.tech
Block Explorer: https://straks.info
Discord Forum: https://discord.straks.tech
STRAKS Forum: https://straks.co
STRAKS ANN: https://bitcointalk.org/index.php?topic=2433318.0
STRAKS GitHub: https://github.com/straks/straks
STRAKS Insight: https://straks.tech/insight/
STRAKS Telegram: https://telegram.straks.tech
STRAKS Twitter: https://twitter.straks.tech
STRAKS Facebook: https://facebook.straks.tech
STRAKS Core Wallet: https://github.com/straks/straks/releases
STRAKS Electrum Wallet: https://github.com/straks/electrum-stak/releases
STRAKS Android Wallet: https://straks.tech/#wallets
STRAKS Point-of-Sale: https://github.com/straks/strakspos
STRAKS Paper Wallet: https://paper.straks.tech
CoinMarketCap: https://coinmarketcap.com/currencies/straks/
Masternodes Online: https://masternodes.online/currencies/STAK/
Masternodes Pro: https://masternodes.pro/stats/stak
CoinGecko: https://www.coingecko.com/en/coins/straks
Coinlib: https://coinlib.io/coin/STAK/Straks
Crypto-Coinz: https://www.crypto-coinz.net/coin-info/?100-Straks-STAK-Lyra2v2-calculator/
WhatToMine: https://whattomine.com/coins/240-stak-lyra2rev2
----------------------
COIN = STRAKS
DB_DIRECTORY = /home/squbs/electrumdb_STRAKS
DAEMON_URL = http://ustraks:randompassword@127.0.0.1:7574/
RPC_HOST = 127.0.0.1
RPC_PORT = 8000
HOST =
TCP_PORT = 50001
SSL_PORT = 50002
SSL_CERTFILE =
SSL_KEYFILE =
EVENT_LOOP_POLICY = uvloop
IRC = 1
REPORT_HOST = ex001-stak.qxu.io
NET = mainnet
BANNER_FILE = /etc/electrumx.banner
BANDWIDTH_LIMIT=10000000
[Unit]
Description=Electrumx_STAK
After=network.target
[Service]
EnvironmentFile=/etc/electrumx_STAK.conf
ExecStart=/usr/local/bin/electrumx_server.py
User=squbs
LimitNOFILE=8192
TimeoutStopSec=30min
[Install]
WantedBy=multi-user.target
/path/to/electrumx_server.py
#!/bin/sh
exec multilog s500000 n10 /path/to/log/dir
#!/bin/sh
echo "Launching ElectrumX server..."
USERNAME=$(envdir ./env printenv USERNAME)
ELECTRUMX=$(envdir ./env printenv ELECTRUMX)
ulimit -n 4000 && exec 2>&1 envdir ./env envuidgid $USERNAME python3 $ELECTRUMX
#!/bin/sh
###########################
#Installation of Python 3.6
###########################
sudo add-apt-repository ppa:jonathonf/python-3.6
sudo apt-get update && sudo apt-get install python3.6 python3.6-dev
cd ~
git clone https://github.com/kyuupichan/electrumx.git
cd electrumx
sudo python3.6 setup.py install
#!/bin/sh
###################
# install electrumx
###################
# Remove "raspi-copies-and-fills" as it breaks the upgrade process
sudo apt-get purge raspi-copies-and-fills
# upgrade raspbian to 'stretch' distribution
sudo echo 'deb http://mirrordirector.raspbian.org/raspbian/ testing main contrib non-free rpi' > /etc/apt/sources.list.d/stretch.list
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get autoremove
# install electrumx dependencies
sudo apt-get install python3-pip
sudo apt-get install build-essential libc6-dev
sudo apt-get install libncurses5-dev libncursesw5-dev
sudo apt install libreadline6-dev/stable libreadline6/stable
sudo apt-get install libleveldb-dev
sudo apt-get install git
sudo pip3 install plyvel
# install electrumx
git clone https://github.com/kyuupichan/electrumx.git
cd electrumx
sudo python3 setup.py install
#!/bin/sh
###############
# run_electrumx
###############
# configure electrumx
export COIN=BitcoinSegwit
export DAEMON_URL=http://rpcuser:[email protected]
export NET=mainnet
export CACHE_MB=400
export DB_DIRECTORY=/home/username/.electrumx/db
export SSL_CERTFILE=/home/username/.electrumx/certfile.crt
export SSL_KEYFILE=/home/username/.electrumx/keyfile.key
export BANNER_FILE=/home/username/.electrumx/banner
export DONATION_ADDRESS=your-donation-address
# connectivity
export HOST=
export TCP_PORT=50001
export SSL_PORT=50002
# visibility
export REPORT_HOST=hostname.com
export RPC_PORT=8000
# run electrumx
ulimit -n 10000
/usr/local/bin/electrumx_server.py 2>> /home/username/.electrumx/electrumx.log >> /home/username/.electrumx/electrumx.log &
######################
# auto-start electrumx
######################
# add this line to crontab -e
# @reboot /path/to/run_electrumx.sh
# default /etc/electrumx.conf for systemd
# REQUIRED
DB_DIRECTORY = /db
# Bitcoin Node RPC Credentials
DAEMON_URL = http://username:password@hostname:port/
# See http://electrumx.readthedocs.io/en/latest/environment.html for
# information about other configuration settings you probably want to consider.
[Unit]
Description=Electrumx
After=network.target
[Service]
EnvironmentFile=/etc/electrumx.conf
ExecStart=/usr/local/bin/electrumx_server.py
User=electrumx
LimitNOFILE=8192
TimeoutStopSec=30min
[Install]
WantedBy=multi-user.target
Thanks to Thomas Voegtlin for creating the Electrum software and
infrastructure and for maintaining it so diligently. Electrum is
probably the best desktop Bitcoin wallet solution for most users. My
faith in it is such that I use Electrum software to store most of my
Bitcoins.
Whilst the vast majority of the code here is my own original work and
includes some new ideas, it is very clear that the general structure
and concept are those of Electrum. Some parts of the code and ideas
of Electrum, some of which it itself took from other projects such as
Abe and pywallet, remain. Thanks to the authors of all the software
this is derived from.
I'd like to thank bauerj, hsmiths and JWU42 of #electrum for their
help and support in testing the versions of ElectrumX prior to 1.0.
Thanks to Daniel Bernstein for daemontools and other software, and to
Matthew Dillon for DragonFlyBSD. They are both deeply inspirational
people.
And of course, thanks to Satoshi for the wonderful creation that is
Bitcoin.
\ No newline at end of file
This diff is collapsed.
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS = -W
SPHINXBUILD = sphinx-build
SPHINXPROJ = ElectrumX
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M [email protected] "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
Just some notes on performance with Python 3.5. We are taking these into
account in the code.
- 60% faster to create lists with [] list comprehensions than tuples
or lists with tuple(), list(). Of those list is 10% faster than
tuple.
- however when not initializing from a generator, a fixed-length tuple
is at least 80% faster than a list.
- an implicit default argument is ~5% faster than passing the default
explicitly
- using a local variable x rather than self.x in loops and list
comprehensions is over 50% faster
- struct.pack, struct.unpack are over 60% faster than int.to_bytes and
int.from_bytes. They are faster little endian (presumably because
it matches the host) than big endian regardless of length. Furthermore,
using stored packing and unpacking methods from Struct classes is faster
than using the flexible-format struct.[un]pack equivalents.
After storing the Struct('<Q').unpack_from function as unpack_uint64_from,
later calls to unpack_uint64_from(b, 0) are about 30% faster than calls to
unpack_from('<Q', b, 0).
- single-item list and tuple unpacking. Suppose b = (1, )
a, = b is a about 0.4% faster than (a,) = b
and about 45% faster than a = b[0]
- multiple assignment is faster using tuples only for 3 or more items
- retrieving a previously stored length of a bytes object can be over 200%
faster than a new call to len(b)
\ No newline at end of file
Architecture
============
.. image:: https://docs.google.com/drawings/d/1Su_DR2c8__-4phm12hAzV65fL2tNm_1IhKr4XivkW6Q/pub?w=720&h=540
:target: https://docs.google.com/drawings/d/1Su_DR2c8__-4phm12hAzV65fL2tNm_1IhKr4XivkW6Q/pub?w=960&h=720
Env
---
Holds configuration taken from the environment, with apprioriate
defaulting appropriately. Generally passed to the constructor of
other components which take their settings from it.
Controller
----------
The central part of the server process initialising and coordinating
all the others. Manages resource usage.
LocalRPC
--------
Handles local JSON RPC connections querying ElectrumX server state.
Started when the ElectrumX process starts.
ElectrumX
---------
Handles JSON Electrum client connections over TCP or SSL. One
instance per client session. Should be the only component concerned
with the details of the Electrum wire protocol.
Not started until the Block Processor has caught up with bitcoind.
Daemon
------
Encapsulates the RPC wire protocol with bitcoind for the whole server.
Transparently handles temporary bitcoind connection errors, and fails
over if necessary.
Notifies the Mempool when the list of mempool transaction hashes is
updated.
Block Processor
---------------
Responsible for managing block chain state (UTXO set, history,
transaction and undo information) and for handling block chain
reorganisations.
When caught up, processes new blocks as they are found, and flushes
the updates to the Database immediately.
When syncing uses caches for in-memory state updates since the prior
flush. Occasionally flushes state to the storage layer when caches
get large.
Prefetcher
----------
Cooperates with the Block Processor to asynchronously prefetch blocks
from bitcoind. Once it has caught up it additionally asks the Daemon
to refresh its view of bitcoind's mempool transaction hashes. Serves
blocks to the Block Processor via a queue.
Mempool
-------
Handles all the details of maintaining a representation of bitcoind's
mempool state. Obtains the list of current mempool transaction hashes
from the Daemon when notified by the Prefetcher.
Notifies the Controller that addresses have been touched when the
mempool refreshes (or implicitly when a new block is found).
Database
--------
The underlying data store, made up of the DB backend (such as
`leveldb`) and the host filesystem.
.. _Authors:
Authors
=======
* Neil Booth
Creator and maintainer.
* Johann Bauer
Backend DB abstraction.
* John Jegutanis
Alt-chain integrations.
ChangeLog
=========
Version 1.4.3
-------------
* Fix `#442`_.
Version 1.4.2
-------------
* proxy remote IP reported properly if :envvar:`FORCE_PROXY` is set.
Fixes `#301`_.
* requires aiorpcx 0.5.5
Version 1.4.1
-------------
* minor bugfixes - cleaner shutdown; group handling
* set PROTOCOL_MIN to 1.0; this will prevent 2.9.x clients from connecting
and encourage upgrades to more recent clients without the security hole
* requires aiorpcx 0.5.4
Version 1.4
-----------
* switch to `aiorpcX <https://github.com/kyuupichan/aiorpcX>`_ for all
networking, ``JSON RPC`` and proxy handling
* proxy detection improvements
* `documentation <https://electrumx.readthedocs.io/>`_ rewrite
* new environment variable :envvar:`LOG_FORMAT` to control logging format
* new environment variable :envvar:`DROP_CLIENT` to cut off unsupported
client software
* coin updates: Litecoin (pooler), bitbayd (kongeo), BTG (wilsonmeier),
danny91, wakiyamap, snowgem, Dash (theLazier), fujicoin
* new coins: Decred (cipherzzz), axe (-k),
* typo fixes (dax, romanz)
.. note:: the Dash-specific undocumented ``masternode.subscribe()``
RPC call was not following the JSON RPC spec; this was shown up by
the switch to aiorpcX. I had to modify the code but it may break
Dash clients.
The Decred implementation doesn't work on mainnet; I will remove it
if this remains unfixed.
Version 1.3
-----------
* Switch to :ref:`version 1.2` of the protocol.
:func:`mempool.get_fee_histogram` implementation contributed by ecdsa,
verbose mode of :func:`blockchain.transaction.get` by gdassori.
* :func:`blockchain.scripthash.listunspent` now takes into account mempool
spends and receipts.
* Improved client notification handling.
* Wait for mempool to fully sync before serving.
* Documentation moved to `readthedocs.io
<https://electrumx.readthedocs.io/>`_. Rewritten and improved
protocol documentation.
* new/updated coins: Chips (cipig), Feathercoin (lclc), Zclassic(heyrhett),
Dash (thelazier), NYC (xarakas), Koto (wo01), BitcoinZ (cipig), BitCore
(cipig), Fujicoin (fujicoin), Bitcoin Atom (erasmospunk), Deanrius (carsenk),
SNG (blackjok3rtt).
* Minor fixes and improvements: duckartes, blin00, bauerj,
erasmospunk, SomberNight, romanz.
Version 1.2.1
-------------
- remove IRC support. Most coins had empty IRC channels. Those that
don't have peers populated.
- use estimatesmartfee RPC call if available (SomberNight)
- new/updated coins: Emercoin (Sergii Vakula), Bitcoin Gold (erasmospunk),
Monacoin testnet (Wakiyama P), sibcoin (53r63rn4r), Komodo and Monaize
(cipig), Hush (Duke Leto)
- doc updates (fr3aker)
- issues fixed: `#302`_
Version 1.2
-----------
.. note:: version 1.2 changes script hash indexing in the database, so
you will need to rebuild your databases from scratch. Running this
version will refuse to open the DB and not corrupt it, so you can
revert to 1.1.x if you wish. The initial synchronisation process
should be around 10-15% faster than 1.1, owing to this change and
Justin Arthur's optimisations from 1.1.1.
- separate P2PKH from P2PK entries in the history and UTXO databases.
These were previously amalgamated by address as that is what
electrum-server used to do. However Electrum didn't handle P2PK
spends correctly and now the protocol admits subscriptions by script
hash there is no need to have these merged any more.
For Bitcoin (BitcoinSegwit/mainnet) you can download a leveldb database
synced up to block 490153 using this bittorrent magnet
`link (~24GB) <magnet:?xt=urn:btih:caa804f48a319b061be3884ac011656c27121a6f&dn=electrumx_1.2_btc_leveldb_490153>`_.
Version 1.1.2
-------------
- PEER_DISCOVERY environment variable is now tri-state (fixes
`#287`_). Please check your setting as its meaning has changed
slightly.
- fix listunspent protocol methods to remove in-mempool spends (fixes
`#277`_).
- improved environment variable handling
- EMC2 update (cipig), Monacoin update (cryptocoin-junkey),
Canada Ecoin (koad)
- typo fixes, Bitcoin testnet peers updates (SomberNight)
Version 1.1.1
-------------
- various refactorings, improvement of env var handling
- update docs to match
- various optimizations mainly affecting initial sync (Justin Arthur)
- Dash fixes (cipig)
- Add ALLOW_ROOT option (Luke Childs)
- Add BitZeny support, update Monacoin (cryptocoin-junkey)
Version 1.1
-----------
See the changelogs below for recent changes. The most important is
that for mainnet bitcoin **NET** must now be *mainnet* and you must
choose a **COIN** from *BitcoinCash* and *BitcoinSegwit*. Similarly
for testnets. These coins will likely diverge further in future so
it's best they become separate coins now.
- no longer persist peers, rediscover on restart
- onion peers only reported if can connect; hard-coded exception removed
- small fix for blockchain.transaction.broadcast
**Neil Booth** [email protected] https://github.com/kyuupichan
1BWwXJH3q6PRsizBkSGm2Uw4Sz1urZ5sCj
LKaFk4KkVpw9pYoUpbckQSKKgCVC4oj78b
.. _#277: https://github.com/kyuupichan/electrumx/issues/277
.. _#287: https://github.com/kyuupichan/electrumx/issues/287
.. _#301: https://github.com/kyuupichan/electrumx/issues/301
.. _#302: https://github.com/kyuupichan/electrumx/issues/302
.. _#442: https://github.com/kyuupichan/electrumx/issues/442
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#