ハイアベイラビリティクラスタ

Neo4j Wiki から

目次

[edit] はじめに

The Neo4j High Availability (HA) project has the following two goals:

  1. Provide a fault-tolerant database architecture, where several Neo4j slave databases can be configured to be exact replicas of a single Neo4j master database. This allows the end-user system to be fully functional and both read and write to the database in the event of hardware failure.
  2. Provide a horizontally scaling read-mostly architecture that enables the system to handle much more read load than a single Neo4j database.

Neo4j HA uses a single master and multiple slaves. Both the master and the slaves can accept write requests. A slave handles a write by synchronizing with the master to preserve consistency. Updates to slaves are asynchronous so a write from one slave is not immediately visible on all other slaves. This is the only difference between HA and single node operation (all other ACID characteristics are the same).

[edit] Setting up a local development and testing environment

[edit] Coordinator クラスタのセットアップ

The members of the HA cluster use a Coordinator cluster to manage themselves and coordinate lifecycle activity like electing a master. When running an embedded HA cluster, a Zookeeper cluster can be used for coordination. That must be installed and configured before working with the HA instances.

Go to www.apache.org/dyn/closer.cgi/hadoop/zookeeper/, select a mirror and grab the 3.3.2 release.

Unpack somewhere and create three config files called server1.cfg, server2.cfg and server3.cfg in the conf directory:

#server1.cfg
tickTime=2000
initLimit=10
syncLimit=5

dataDir=data/zookeeper1
clientPort=2181

server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890

The other two config files will have a different dataDir and clientPort set but the other parameters identical to the first one:

#server2.cfg
#...
dataDir=data/zookeeper2
clientPort=2182
#...

#server3.cfg
dataDir=data/zookeeper3
clientPort=2183

Create the data dirs:

zookeeper-3.3.2$ mkdir -p data/zookeeper1 data/zookeeper2 data/zookeeper3

Next we need to create a file in each data directory called "myid" that contains an id for each server equal to the number in "server.1" "server.2" and "server.3" from the configuration files.

zookeeper-3.3.2$ echo '1' > data/zookeeper1/myid
zookeeper-3.3.2$ echo '2' > data/zookeeper2/myid
zookeeper-3.3.2$ echo '3' > data/zookeeper3/myid

We are now ready to start the ZooKeeper instances:

zookeeper-3.3.2$ java -cp lib/log4j-1.2.15.jar:zookeeper-3.3.2.jar org.apache.zookeeper.server.quorum.QuorumPeerMain conf/server1.cfg &
zookeeper-3.3.2$ java -cp lib/log4j-1.2.15.jar:zookeeper-3.3.2.jar org.apache.zookeeper.server.quorum.QuorumPeerMain conf/server2.cfg &
zookeeper-3.3.2$ java -cp lib/log4j-1.2.15.jar:zookeeper-3.3.2.jar org.apache.zookeeper.server.quorum.QuorumPeerMain conf/server3.cfg &

ZooKeeper の詳細については、hadoop.apache.org/zookeeper/docs/r3.3.2/ を参照してください。

[edit] ダウンロード

If you are using maven, simply add the following dependency to your project:

      <dependency>
         <groupId>org.neo4j</groupId>
         <artifactId>neo4j-ha</artifactId>
         <version>1.3</version>
      </dependency>

If you prefer to download the jar files manually, they are included in the Neo4j distribution.

[edit] Start a highly available graph database service

The difference in code when using Neo4j-HA is the creation of the graph database service.

GraphDatabaseService db = new HighlyAvailableGraphDatabase( path, config );

The configuration can contain the standard configuration parameters but will also have to contain:

#unique machine id for this graph database
#can not be negative id and must be unique
ha.machine_id = 1

#ip and port for this instance to bind to
ha.server = localhost:6001

#connection information to zookeeper
ha.zoo_keeper_servers = localhost:2181,localhost:2182,localhost:2183

Start a project that depends on the neo4j-ha-0.5-1.2.jar library. You may have to manually add zookeeper-dev.jar and zookeeper-3.3.2.jar in a directory called lib (the jars can be found where you unpacked zookeeper). Here is a full dependency list required to run neo4j-ha:

 org.neo4j:neo4j-ha:jar:0.5-1.2
 +- org.neo4j:neo4j-kernel:jar:1.2-1.2
 |  \- org.apache.geronimo.specs:geronimo-jta_1.1_spec:jar:1.1.1
 +- org.neo4j:neo4j-management:jar:1.2-1.2
 +- org.neo4j:neo4j-shell:jar:1.2-1.2
 |  \- org.apache.servicemix.bundles:org.apache.servicemix.bundles.jline:jar:0.9.94_1
 +- org.neo4j:neo4j-lucene-index:jar:0.2-1.2
 |  \- org.apache.servicemix.bundles:org.apache.servicemix.bundles.lucene:jar:3.0.1_2
 +- org.apache.zookeeper:zookeeper:jar:3.3.2
 +- org.jboss.netty:netty:jar:3.2.1.Final
 +- log4j:log4j:jar:1.2.16

First we need to create a database that can be used for replication. This is easiest done by just starting a normal embedded graph database, pointing out a path and shutdown.

new EmbeddedGraphDatabase( "empty-db" ).shutdown();

The next step would be to create a configuration file and a Main class to start a highly available graph database.

#ha-server1.conf
ha.machine_id = 1
ha.server = localhost:6001
ha.zoo_keeper_servers = localhost:2181,localhost:2182,localhost:2183

enable_remote_shell = port=1331

The Main class:

import java.util.Map;

import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.kernel.HighlyAvailableGraphDatabase;

public class StartLocalHaDb
{
    public static void main( String[] args )
    {
        String path = args[0];
        String configFile = args[1];
        Map<String,String> config = HighlyAvailableGraphDatabase.loadConfigurations( configFile );
        GraphDatabaseService db = new HighlyAvailableGraphDatabase( path, config );
    }
}

We created a config file with machine id=1 and enabled remote shell. The main method will expect the path to the db as first parameter and the configuration file as the second parameter. Copy the empty db to a path and try to execute:

$ cp -R empty-db hadb1
$ java -cp <all jars and StartLocalHaDb class> StartLocalHaDb hadb1 ha-server1.conf

It should now be possible to connect to the instance using the Neo4j Shell:

$./bin/neo4j-shell -port 1331
NOTE: Remote Neo4j graph database service 'shell' at port 1331
Welcome to the Neo4j Shell!Enter 'help' for a list of commands

neo4j-sh (0)$ hainfo
I'm currently master
Connected slaves:
neo4j-sh (0)$ mkrel -ct TEST
neo4j-sh (0)$ ls
(me) --<TEST>-> (1)
neo4j-sh (0)$ cd 1
neo4j-sh (1)$ set name test1
neo4j-sh (1,test1)$ ls
*name =[test1]
(me) <-<TEST>-- (0)

Since it is the first instance to join the cluster it is elected master. Starting another instance would require a second configuration and another path to the db.

#ha-server2.conf
ha.machine_id = 2
ha.server = localhost:6002
ha.zoo_keeper_servers = localhost:2181,localhost:2182,localhost:2183

enable_remote_shell = port=1332

We need to change machine id, shell port and listen port of the ha server, then copy the db to a new path and startup:

$ cp -R empty-db hadb2
$ java -cp <all jars and StartLocalHaDb class> StartLocalHaDb hadb2 ha-server2.conf

Now start the shell connecting to port 1332:

$./start-neo4j-shell-port 1332
NOTE: Remote Neo4j graph database service 'shell' at port 1332
Welcome to the Neo4j Shell!Enter 'help' for a list of commands

neo4j-sh (0)$ hainfo
I'm currently slave
neo4j-sh (0)$ mkrel -ct TEST
neo4j-sh (0)$ ls
(me) --<TEST>-> (1,test1)
(me) --<TEST>-> (2)
neo4j-sh (0)$ cd 2
neo4j-sh (2)$ set name test2
neo4j-sh (2,test2)$ cd ..
neo4j-sh (0)$ ls
(me) --<TEST>-> (1,test1)
(me) --<TEST>-> (2,test2)
neo4j-sh (0)$  

[edit] Setup a production environment or multi machine testing environment

[edit] Porting your application to run on Neo4j-HA

As mentioned earlier the only thing that needs to change is the creation of the graph database service. In single machine operations the EmbeddedGraphDatabase class should be used while in multi machine HA environment the HighlyAvailableGraphDatabase class should be used. Depending on application type this either means modifying the code or the configuration of the container running the application.

[edit] Multi machine cluster setup

We have already covered how to setup a ZooKeeper service locally. Setting up a multi machine environment together with Neo4j-HA can be done in different ways depending on requirements. First, reading the ZooKeeper administration guide on how to deploy and maintain a ZooKeeper cluster is recommended.

Depending on the requirements such as load, fault tolerance and available hardware Neo4j-HA can be configured in different ways. Here we will outline different setups and describe their characteristics.

[edit] Small

Requirements:

  • 3 physical (or virtual) machines
  • 1ZooKeeper instance running on each machine
  • 1Neo4j-HA instance running on each machine

This setup is conservative in use of hardware while being able to handle moderate read load. It can only operate when at least 2 of the ZooKeeper instances are running. Since the ZooKeeper service and Neo4j-HA are running together on each machine this will in most scenarios mean that only one server is allowed to go down (theoretically it can handle 1 ZooKeeper instance crash and 2 Neo4j-HA instances going down).

[edit] Medium

Requirements:

  • 5-7+ machines
  • ZooKeeper running on 3, 5 or 7 machines
  • Neo4j-HA can run on 5+ machines

The drawback with this setup is that on some machines both a Neo4j-HA instance and ZooKeeper instance are running while on some machines only a Neo4j-HA instance is running. This means that two different machine setups have to be managed.

Fault tolerance will depend on how many machines are running ZooKeeper. With 3 ZooKeeper instances the cluster can survive one ZooKeeper going down, with 5 it can survive 2 and with 7 it can handle 3 ZooKeeper instances failing. The number of Neo4j-HA instances that can fail is theoretically all but 1.

[edit] Large

Requirements:

  • 8+ machines
  • ZooKeeper running on dedicated machines.

In this setup all ZooKeeper instances are running on separate machines as a dedicated ZooKeeper service. The dedicated ZooKeeper cluster of 5 machines can handle 2 machines failing, 7 machines 3 and so on. The Neo4j-HA cluster will be able to operate from X machines down to a single machine. Adding more Neo4j-HA instances will be very easy in this setup since one does not have to be concerned with ZooKeeper (hosted as a dedicated service elsewhere) thus, the only thing that needs to be configured for each new Neo4j-HA is machine id and host.

[edit] Configuration, administration and operations

Except for standard Neo4j kernel configuration and the ha.machine_id, ha.server, and ha.zoo_keeper_servers settings, there is at the moment one additional configuration parameter for Neo4j-HA called ha.pull_interval. This parameter tells how often (in seconds) a slave should check for updates on the master when the slave is in idle mode (or just serving read requests).

When adding more instances to an existing Neo4j-HA cluster it is very important to make sure the ha.machine_id setting gets assigned a unique id. Start by creating the configuration file with unique machine id and other required configuration parameters. Take a fairly up to date copy of the database running in the Neo4j-HA cluster and deploy it together with your new configuration and application. Finally start the application.

To take an instance down for maintenance:

  1. Invoke the shutdown method on the highly available graph database (or if the application is running in a container, invoke the shutdown procedure for the application/container).
  2. Perform the required maintenance.
  3. Start the application.

Monitoring the running cluster can be performed using JMX tools (see Monitoring and Deployment), for more detailed information on each instance see the messages.log file found in the db store directory.

[edit] Problems / troubleshooting

If an instance crashes or will not startup the first thing to do is to review the logs. Review messages/stacktraces printed to System.out and the db-store-dir/messages.log.

"org.neo4j.kernel.ha.zookeeper.ZooKeeperTimedOutException: Connection to ZooKeeper server timed out, keeper state=Disconnected" means that the Neo4j-HA instance was unable to connect to the ZooKeeper service. Make sure the ZooKeeper service is running and that the "ha.zoo_keeper_servers" configuration parameter has the right connection information. To test if the ZooKeeper service is running you can try to connect to it using the client tool supplied by ZooKeeper.

"Exception in thread "main" java.lang.RuntimeException: java.io.FileNotFoundException: <db-store-dir> (No such file or directory) at org.neo4j.kernel.ha.zookeeper.NeoStoreUtil.<init>(NeoStoreUtil.java:38)". The database store directory pointed out does not exist. Make sure a copy of the database exists locally on the instance in the specified folder.

In the messages.log file there will sometimes be a log message "newMaster(null) called <exception>: <message> " followed by a stack trace. This message usually means that the current master is unavailable. A new master will be automatically elected.

"Broken store, my last committed tx,machineId[txid,machineid] but master says machine id for that txId is other-machineid" means that the machine trying to connect to the running Neo4j-HA cluster actually has a different version of the database compared to the current master. This can happen when for example writing one transaction directly to the master and the master crashes right after it has committed (but before it has replicated the transaction to any slave). A new master is elected and the next transaction is executed. Once the original master tries to connect to the cluster it will be rejected since its view of the store is not consistent with the current's master. Always writing to a slave will minimize the risk of this happening since if the master crashes then one slave will have the latest committed transaction allowing for the old master to rejoin the cluster after a potential crash.

Neo4j のサイト
ツールボックス