Zero-Knowledge Database Libraries

 

 
 











$Id: zkDB.html,v 1.4 2000/07/04 18:23:03 igor Exp $


Revision Comments
1.1 Author(s): Igor Stolbikov (igor@zks.net); 
Initial draft. 











zkDB Library Architecture

The zkDB  libraries implement several technologies which allow effectively implement database access:
1.  Configuration files.
   Database configuration files allow to customize database behavior or database settings.
   It include following group of the parameters:
        General set of configurations parameters. Include:
         -  Default Home and Transactional Log Directories
         -  List of the partitions (physical databases) , which are used in the one logical database.
         -  General configuration options of each partition, including partition logical name, what it is type ( remote or local), is it require hash operation,  partition limits.
         -  Remote configuration options. Remote partition has to have defined address path,  which include database server host name and port number,  and also  logical database name and authentication string.
         - Local configuration options:  physical indexes type  (type = BTREE or HASH) ,  Home and Log directory, Database Cash Size,  Page size, minimum checkpoint time, minimum checkpoint size, maximum  concurrent transactions, and so on. Using configuration file is opened possibilities for the flexible management and database turning.
2. Double Remote/Local access.
      ZkDB library support 2 different kind of data access:
         - Remote.  Provide access to the data through  zk command protocol to the zkDatabase server.
            Database server is responsible for all database management.  Several core services located on the different machines can share the same database server.
         - Local (embedded).   Core application will be responsible for the database management.
      Chosing the right type of access depend greatly from the several questions:
       - how many database clients will be accessing the data?
       - is data has to shared through different sevices?
       - what will be database size?
For a small local databases embeded access will be most efficient solution.
In case of the big databases shared by the number of clients,  remote client-server access will be more preferable.

 2.  Partitioning.
     Partitioning means presenting one big logical database as set of the physical databases (called partitions). Partitions    can be physically located on the different compute servers, different hard drives or simply in the different directories. All operations like search, update, scan are executed independently on each partitions. Partitions allow with maximum efficiency use multi-computer , multi-processor and multi hard drive hardware configurations.
 Partitions allow to take out database file size limitation of 2 GBytes on the Linux servers.  Current maximum number  partitions is 256. It is mean it can be up to 512 GBytes databases on the one Linux server. (Current limitation is 2 GBytes). Using distributed database network logical database size will be not limited!

                                                                 Pic. 1

  During zkDB_Open operations zkDB open database config file and read all necessary information concerning
number of partitions and partitions configurations. The list of the options for database config file can be found at
database configuration  documentation.

Partitions can be remote or local. In case of remote partitions
zkRemotePartition_* functions will be used. In case of the local zkLocalPartition_* functions.

  zkDB layer execute role dispatcher directing database calls to the necessary partitions. Depending from partition type
call will be directed to remote or local partition.
 
 


             Pic. 2
 

4  Partition selection.

In the most  zkDB operations, partitions selection is based on the input key value, string position from which comparison started -  LimitStartPos , min limit and max limit partition values. Each partition has to have they max Limit, Min Limit , LimitStartPos  parameters  set in database configuration file. MinLimit and MaxLimit options  define minimum and maximum values of the comparable sub key, which will qualify record as belonging to specific partition. The sub key will started from LimitStartPos position (count from 0)  of the record key and his length will be the same as the MaxLimit and MinLimit length. Usually it will be enough to use just one character to do partition selection.
For  the first partition MinLimit value is not required as well as MaxLimit value for the last partition.
It is two different way to specify Min and Max Limit values depending is IsHashed option set to Yes.  If IsHashed did not set or set to No then string comparison function is used to define if key belong to the partition. Here the example of the database with 3 partitions:

                           Partitions  = part_1;part_2;part_3;

                            [part_1]
                              MaxLimit = k
                             LimitStartPos = 3
                          [part_2]
                              MinLimit = l
                              MaxLimit = p
                              LimitStartPos = 3
                            [part_3]
                              MinLimit = q
                              LimitStartPos = 3

If input key value will start from any  character  from 'a' to 'k' or 'A' to 'K' it will belong to the first partition. Else if it  start  from 'l' to 'p'   or 'L' to 'P' it will go to the second. And in case 'q' or 'Q' and bigger it will go to the third partition.
The limit string can contain move than one character, but it is required that all limit strings have same length and form
uninterrupted range values covered all possible keys variation.
In case of hashed values or binary data another method of the setting limits can be used. Currently just 1 byte value is used to find partition. All range of values is splited on the range from 0 to 255 (maximum number of partitions).  And the limit values has to be specified in this range :
                          [part_1]
                             IsHashed = Yes
                            MinLimit  = 0
                            MaxLimit = 96
                            LimitStartPos = 3

                         [part_2]
                              IsHashed = Yes
                              MinLimit = 97
                              MaxLimit = 180
                              LimitStartPos = 3

                         [part_3]
                              IsHashed = Yes
                              MinLimit = 181
                              MaxLimit  =  255
                              LimitStartPos = 3
Default value of the  MinLimit = 0 and MaxLimit = 255. It is possible that in future versions  maximum number of partitions will be changed. It is important that LimitStartPos reference  the changeable part of the key and MinLimit and MaxLimit values take in consideration possible range values.
For example if all the keys has following pattern:
"Key 1" , "Key  2"  ...  "Key N".  If partition have LimitStartPos set to anything else than 4 , it will put all records to the same partition.  The same can happened if  MinLimit and MaxLimit values  did not respect range of values.
 Changing MinLimit and MaxLimit allow to

5. Library structure
  ZkDB library contains 3 layer.
  Dispatching  layer , which has all interface functions:
     Open/Close functions
             int zkDB_Open (ZkError err, const char * dbfile, ZkDB * db);
            int zkDB_Close (ZkError err, ZkDB * db);
     Operation functions
           zkDB_Fetch (ZkError err, ZkDB db, ZkDBT key, ZkDBT data, int * found);
                                                                                                                             and so on.
    This layer uses ZkDB object which is a complex object include  a array different kind of partitions (local or remote).
    Each function call is dispatched by the layer to corresponding partitions or set of partitions function calls,  remote or local.

To see the list of all ZkDB API functions see

   Remote connection layer. Operate on the remote partitions. Responsible for the connection with the database server,
sending commands and reciving response.
Contains similar set of functions
Open/Close functions
             int zkRemotePartition_Connect (ZkError err,  ZkRemotePartition partition);
             int zkRemotePartition_Disconnect(ZkError err,  ZkRemotePartition partition);
     Operation functions
           zkRemotePartition_Fetch (ZkError err, ZkRemotePartition partition, ZkDBT key, ZkDBT data, int * found);
                                                                                                                             and so on.

 Local database access layer. Operate on local physical databases (partitions).
    Open/Close functions
             int zkLocalPartition_Open (ZkError err,  ZkLocalPartition partition);
             int zkLocalPartition_Close(ZkError err,  ZkLocalPartition partition);
     Operation functions
           zkLocalPartition_Fetch (ZkError err, ZkLocalPartition partition, ZkDBT key, ZkDBT data, int * found);
                                                                                                                             and so on.
6. Error Handling
        zkDB process each error differently depending on the condition in which error happened. If error happened during open or close process, this error is consider to be critical and it's error type is urgent. Errors which are inform about bad database state and require database recovery, consider to be urgent as well. This approach allow Database server distinguish critical, fatal error from the regular error and behave differently depending from the error type.
The current standard behavior is to restart database child process each time it has not fatal error. Fatal error cause child process to shut down without restarting!