$Id: zkdb.html,v 1.0 2000/07/27 08:46:36 sam Exp $
Revision |
Comments |
---|---|
1.1 |
Author(s): Francis L. (francisl@zks.net) Initial draft |
1.2 |
Author(s): Sam Sanjabi (sam@zks.net), Scarecrow revision. Partitioning functionality added. |
See also: Zero-Knowledge Database Server Protocol Specification
The Zero-Knowledge Database (zkDB) library provides a single interface for accessing both local and remote databases.
To open a database you use the zkDB_Open() function call, which takes a string indicating the filename of a database configuation file. You may think of this file as the database file itself, although you will have to ensure that the configuration is correct. Furthermore, these configuration files are now used by ndbutil to read it's variables, hence each section has some variables that are only used by ndbutil - these are labelled in comments in the following examples, but don't need to be. Incidentally, a comment is any line that begins with a semicolon (;).
A local database resides on the same machine as the process using it. The configuration file for opening a multi-partitioned local database will look something like this for a 2-partition case:
; ; Foo Database Local Access Configuration File ; [ main ] Database = foodb Partitions = foo-1,foo-2 DefaultHomeDir = /freedom/db/foodb DefaultTransactLogDir = /freedom/db/translog/foodb Isolated_Partitions = Yes ;---Variables used by ndbutil only Debug = Yes LogFile = ndbutil.log Prompt = No Operation = recover [ foo-1 ] IsRemote = No IsHashed = No DoHashing = No MaxLimit = l PartitionType = HASH ;---Variables used by ndbutil only recovery.Fatal = No [ foo-2 ] IsRemote = No IsHashed = No DoHashing = No MinLimit = m PartitionType = HASH ;---Variables used by ndbutil only recovery.Fatal = No
Point your zkDB_Open() call at this config file.
[ main ]
Main section title.
Database = foodb
The name of the database or partition (depending on which section it appears in). If not specified in the main section, then the database name will default to NULL. If not specified in the section for a particular partition, then the partition name will default to the section name.
Partitions = foo-1,foo-2
The syntax supports multiple partitions, a comma seperated list of the partitions described in the other sections of the fail. Make sure there are no spaces between the commas, otherwise zkDB could read the names incorrectly.
DefaultHomeDir = /freedom/db/foodb
Default home directory for the database files. Each partition file will appear in its own subdirectory. Both the subdirectory and file will have the same name as the partition (as defined by the Database variable within each partition). Note that this can be overidden by the HomeDir variable within each partition. In this case, if the HomeDir variable in foo-1 were not set, the database file for partition foo-1 would be:
/freedom/db/foodb/foo-1/foo-1
DefaultTransactLogDir = /freedom/db/translog/foodb
Transaction Logs for the database files. These hold the same information as the database, so keeping them on a separate disk from the database files provides "quick and dirty" disaster recovery. As with the DefaultHomeDir variable, each partition will be given a subdirectory, and the transaction log for that partition would appear in that subdirectory unless it is overridden by the TransactLogDir variable within the section for the partition. In this case, the default transaction log for partition foo-1 would be:
/freedom/db/foodb/foo-1/log.xxxxxxxx
Isolated_Partitions = Yes
Don't worry about this, but keep it the same. I don't know what it does, I don't know why it's there, and zkDB doesn't even read it, but someone wants it there (perhaps for some future version).
Debug = Yes
Used by ndbutil. Turns ndbutil debug mode on so that more information is logged when you use it.
LogFile = ndbutil.log
Used by ndbutil. Specifies its log file.
Prompt = Yes
Used by ndbutil when the Operation variable is set to verifydirs. Will prompt the user for permission before creating any directory.
Operation = recover
Used by ndbutil to determine which operation to perform. Can be one of recover and verifydirs. The first is like the old ndbutil as described in ndbutil(8). The second reads the configuration file and attempts to create the directory structure needed by the database.
[ foo-1 ]
Section name for the partition.
IsRemote = No
This is a local partition.
IsHashed = No
When set to No, the keys used to access the database are interpreted as strings. When set to Yes, the keys are hashed to some integer value. Presently, this hash is simply the ASCII equivalent of the first byte of the key.
DoHashing = No
Not currently used by zkDB, waiting for a later release for when we do the hashing of long keys.
MinLimit/MaxLimit = c
Used in a multi-partitioned case to specify which records go to which partition. The minimum and maximum limits within each partition specify the range of keys that belong to that partition. The first partition must have an unspecified MinLimit (since anything smaller than its MaxLimit will belong to it), and similarly the last partition must have an unspecified MaxLimit. Hence, for the example above the partition foo-1 will hold keys that start with the letter "l" or less, the partition foo-2 will hold keys that start with the letter "m" or more.
PartitionType = HASH
This describes the internal addressing format of the database field. It can be HASH or BTREE. BTREE is good for big databases, but needs small keys. HASH is slow for big databases. The conversion between the two is not trivial.
A remote database resides on different machine from the process using it. The database is accessed through a server. The use of the server is transparent to the user of the zkDB library, all you need to do is call zkDB_Open() with a good configuration file. A configuration file for opening a remote 2-partition database will look something like this:
;
; Foo Database Remote Access Configuration File
;
[ main ]
partitions = foo-1,foo-2
[ foo-1 ]
IsRemote = Yes
IsHashed = No
MaxLimit = l
AddressPath = dbhomeip.freedom.net:51131
AuthKey = jK3=;Sa0
ConnectionTimeout = 1500
[ foo-2 ]
IsRemote = Yes
IsHashed = No
MinLimit = m
AddressPath = dbhomeip.freedom.net:51132
AuthKey = jk3=;Sa0
ConnectionTimeout = 1500
The zkDB_Open() function should be pointed at this file. For each partition, there should also be a corresponding local configuration file on the machine where you want the partition to be stored. If the port number, the AuthKey, and the partition names match between the remote file and the local file used by the database server (see section 2.3), then everything should go smoothly.
[ main ]
partitions = foo-1,foo-2
The syntax supports multiple partitions, although at present the library only supports a single partition. Each partition listed must have a section for it elsewhere in the file.
[ foo-1 ]
Section name for the partition.
IsRemote = Yes
This is a remote partition.
IsHashed = No
This is the same as the corresponding variable used in the local partition configuration file
MinLimit/MaxLimit = c
These are the same as the corresponding variables used in the local partition configuration file.
AddressPath = dbhomeip.freedom.net:51131
Hostname of a machine where a database server is running and listening on the given port.
AuthKey = jK3=;Sa0
The database server and client share a secret to "authenticate" each other. This secret must match the one in the db server's config file.
ConnectionTimeout = 1500
Time to wait for responses from the database server.
The database server uses the same configuration file as a local database, but with additional parameters, it looks like this:
; ; Foo Database Server Access Configuration File ; [ CommandServer ] AuthKey = jK3=;Sa0 AccessFile = /freedom/etc/db/access.conf PidFile = /freedom/var/run/zkdbsrv-foodb.pid LogFile = /freedom/var/log/zkdbsrv-foodb.log MaxIdleTime = 0 MaxConnections = 100 AddressPath = :51131 [ main ] Database = foodb DefaultHomeDir = /freedom/db/foodb/ DefaultTransactLogDir = /freedom/db/translog/foodb/ partitions = foo-1 isolated_partitions = Yes [ foo-1 ] IsRemote = No IsHashed = No DoHashing = No PartitionType = HASH
This file is the file used by the database server that serves the first partition (foo-1) of the above foodb database. There should be one of these for each partition required. The main and foo-1 sections are identical to that for a local database since the server opens the database locally.
[ CommandServer ]
Database server section, named because it uses the ZKCP.
AuthKey = jK3=;Sa0
The "Authentication" Key. This must match the key used in the remote config file.
AccessFile = /freedom/etc/db/access.conf
The database server can restrict access based on IP Addresses, this file contains a list of IP Addresses and subnets from which it will accept connections, as well as comment lines starting with a semicolon. This file is mandatory in order for the database server to start.
PidFile =
/freedom/var/run/zkdbsrv-foodb.pid
LogFile =
/freedom/var/log/zkdbsrv-foodb.log
Take a wild stab in the dark. Just kidding. The .pid file will contain the process id of the originating zkdbsrv parent process. This process generates many child processes. The .log file is...alright that one's pretty obvious. Anyway, both of these need to be created in order for the server to start.
MaxIdleTime = 0
Terminate connections once they have been idle this long. 0 means don't terminate.
MaxConnections = 100
Maximum permitted connections. Be careful with this since the zkDB library may not handle refused connections well.
AddressPath = :51131
Port to listen on, this should match the remote config file
The zkDB library uses the zkConfig frame-work for configuration. So both command line and configuration file options are available. The following table lists the zkDB configuration options. Column D column indicates database options, column LP indicates local partition options and RP indicates remote partition options. The 1.2 column indicates if the option is available in Freedom 1.2 release. When an option is specified for both, the partition option has precedence.
Option name |
D |
LP |
RP |
1.2 |
Option Parameter |
Default |
Description |
DefaultHomeDir |
X |
|
|
X |
Directory |
--- |
Home directory of the database partitions. Each partition will be located in its directory (under HomeDir/partition_name). |
DefaultTransactLogDir |
X |
|
|
X |
Directory |
--- |
Transaction log directory. Same rule as for DefaultHomeDir. |
HomeDir |
X |
X |
Directory |
--- |
Used to explicitly specify a partition directory instead of using the default database directory. |
||
TransactLogDir |
X |
X |
Directory |
--- |
Used to explicitly specify a partition Transaction log directory instead of using the default directory. |
||
Partitions |
X |
X |
Partition list |
--- |
List of partitions for the database |
||
IsolatedPartitions |
X |
Boolean |
Yes |
Indicates if partitions are isolated or if they can overlap. Overlapping is not supported in Core 1.2 release. |
|||
IsRemote |
X |
X |
X |
Boolean |
No |
Indicates if the partition is local or remote. |
|
IsHashed |
X |
X |
X |
Boolean |
--- |
Used when partitioning is based on hashed keys to specify that keys passed to zkDB are already hashed. (Can not be set if DoHashing is set). The hash is currently just the ASCII value of the first character. |
|
DoHashing |
X |
X |
Boolean |
No |
Used when partitioning is based on hashed keys to specify that keys passed to zkDB should be hashed. (Can not be set if IsHashed is set). Not supported in Freedom 1.2 release. |
||
MinLimit |
X |
X |
X |
Integer |
--- |
Beginning of range served by a partition. If key hashing is performed than a value between 1 to 256 should be specified otherwise the real key values should be used. |
|
MaxLimit |
X |
X |
X |
Integer |
--- |
End of range served by a partition. If key hashing is performed than a value between 1 to 256 should be specified otherwise the real key values should be used. |
|
ReadOnly |
X |
X |
X |
Boolean |
No |
Indicates if partition is read only |
|
AddressPath |
X |
X |
String |
--- |
Server address path (e.g. "zkdbsrv.zks.net:51130") |
||
ConnectionTimeout |
X |
X |
Time (seconds) |
30 |
Maximum connection time for transaction |
||
Authkey |
X |
X |
Key |
--- |
Packet authentication key (must be same as DB server) |
||
PartitionType |
X |
X |
HASH or BTREE |
BTREE |
Database type of the partition |
The following table contains performance related options. The options are listed starting by the most impacting options to the least. Note that theses options only apply to local partition configuration.
Option name |
Option Parameter |
Default |
Description |
CacheSize |
Integer |
1048576 |
Partition cache size |
Checkpoint.MinTime |
Time (seconds) |
5 |
Minimum time between two checkpoints |
Checkpoint.MinSize |
Size (kBytes) |
32 |
Minimum size before doing checkpoints |
EarlyLockRecognition |
Boolean |
No |
Used for the quick lock recognition. Prevent lock stagnation problem in the situation of big number of processes doing update or fetch operations concurrently on the same record. |
PageSize |
Integer |
0 |
Partition page size (0 means use system settings). |
LogFileSize |
Integer |
10485760 |
Size of partition log file |
LogFlash |
Boolean |
No |
Force transactions to be logged to disk immediately (as supposed to letting the Operating System handle this) |
KeyLock |
Boolean |
Yes |
Indicates if key lock or page lock should be used. |
MaxLockAttempts |
Integer |
--- |
Maximum number of attempts for getting a page or key lock. |
The database options should be specified in a configuration file using the following section names:
[ Main ]
Partitions = part-1,part-2,part-3
[ part-1 ]
...
[ part-2 ]
...
[ part-3 ]
...
The name in italic are user defined and represent database partitions.
Even though the zkDB option syntax allows for multiple partitions, only one partition is supported in the Freedom 1.1 release. Also, it is possible to configure partitions so that their data range can overlap (specified with the Isolated_Partitions option), however, this is not supported in the 1.2 release.
It is possible to have zkDB perform hashing on keys and to partition the database using hashed keys. This provides an easy way of uniformly distributing data over the different partitions. When real keys are used (which can be strings for example), an analysis of the key distribution is required in order to ensure a good distribution of data over the multiple partitions. When key hashing is performed, the keys are mapped to unique values that are generated in pseudo-random fashion to provide a new pseudo-random ordering of the keys. The partition limits are then specified using values between 1 to 256 (and not using hash key values).
Copyright © 2000
Zero-Knowledge Systems Inc.
All rights reserved.