doc: complete dynamic test analysis for Issue #69

.gitignore

@@ -6,6 +6,8 @@ __*
 *.o
 *.bak
 core*
-3ts
 rundb
 runcl
+3TS
+3TS-DAI
+contrib/deneva/obj/deps

LICENSE.txt

@@ -1,7 +1,6 @@
 Tencent is pleased to support the open source community by making Tencent Transaction Processing Testbed System-(3TS--腾讯事务处理技术验证系统) available.
 
-Copyright (C) 2020 THL A29 Limited, a Tencent company.  All rights reserved. The below software in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All Tencent Modifications are Copyright (C) THL A29 Limited.
-
+Copyright © 2020 Tencent Holdings Limited, a Tencent company. All rights reserved. The software in this distribution may have been modified by Tencent Holdings Limited (“Tencent Modifications”). All Tencent Modifications are Copyright © Tencent Holdings.
 Tencent Transaction Processing Testbed System-(3TS--腾讯事务处理技术验证系统) is licensed under the following license, except for the third-party components listed below. 
 
 License for Tencent Transaction Processing Testbed System-(3TS--腾讯事务处理技术验证系统) :

README.md

@@ -4,7 +4,7 @@
 
 ![](https://img.shields.io/badge/license-GPLv3-brightgreen)
 
-**Tencent Transaction Processing Testbed System (3TS)** that is jointly developed by Tencent's CynosDB (TDSQL) team and the Key Laboratory of Data Engineering and Knowledge Engineering of the Ministry of Education of Renmin University of China. The system aims to design and construct a unified framework for transaction processing (including distributed transactions). It enables users to quickly build new concurrency control approaches via the access interface provided by the framework. Based on an comprehensive experiment study over the benchmarks, and the applications abstracted, users can select an optimal concurrency control approach. At present, 3TS has been integrated 13 mainstream concurrency control approaches, and provides common benchmarks such as TPC-C、PPS and YCSB. 3TS further provides a consistency level test benchmark, to address the issue of system selection difficulty caused by the blowout development of distributed database systems, and provides consistency level discrimination and performance test comparison.
+**Tencent Transaction Processing Testbed System (3TS)** that is jointly developed by Tencent's TDSQL team and the Key Laboratory of Data Engineering and Knowledge Engineering of the Ministry of Education of Renmin University of China. The system aims to design and construct a unified framework for transaction processing (including distributed transactions). It enables users to quickly build new concurrency control approaches via the access interface provided by the framework. Based on an comprehensive experiment study over the benchmarks, and the applications abstracted, users can select an optimal concurrency control approach. At present, 3TS has been integrated 13 mainstream concurrency control approaches, and provides common benchmarks such as TPC-C，PPS and YCSB. 3TS further provides a consistency level test benchmark, to address the issue of system selection difficulty caused by the blowout development of distributed database systems, and provides consistency level discrimination and performance test comparison.
 
 If you want to better understand the aims of our project, please view [3TS opensource announcement](doc/en/announcement.md).
 
@@ -16,25 +16,32 @@ If you want to better understand the aims of our project, please view [3TS opens
 
 ## Dependence
 
+### 3TS-Coo
+A consistency verification system. Please check out the newest update on branch ['coo-consistency-check'](https://github.com/Tencent/3TS/tree/coo-consistency-check).
+
+We update our result on [report webpage](https://axingguchen.github.io/3TS/).
+
 ### 3TS-DA
+A static random history generator.  Please check out the newest update on branch ['dev'](https://github.com/Tencent/3TS/tree/dev).
 
 - a compilter supporting C++17 or upper versions (recommend g++8)
 - libconfig 1.7.2
 - gflags 2.1.1
 - gtest 1.6.0
 
 ### Deneva
+A performance verification system. A consistency verification system. Please check out the newest update on branch ['dev'](https://github.com/Tencent/3TS/tree/dev).
 
 - protobuf 3.9.1
 - curl 7.29.0
 - nanomsg 5.1.0
 
 ## Usage
 
-- Run `make.sh` to compile the code. The `3ts` binary will be generated if compiling successfully.
+- Run `make.sh` to compile the code. The `3TS` binary will be generated if compiling successfully.
 - Run `cp config/config.cfg.template config.cfg` to copy the configuration file.
 - Run `vi config.cfg` to modify the configuration file to determine the behavior of the testbed.
-- Run `./t3s --conf_path=config.cfg` to execute test. The test result file will be generated when test is over.
+- Run `./3TS --conf_path=config.cfg` to execute test. The test result file will be generated when test is over.
 
 ## Principle
 
@@ -62,6 +69,7 @@ If you want to better understand the aims of our project, please view [3TS opens
       - `"SerializableAlgorithm_FINAL_SAME_RR"` // repeatable read strategy
       - `"SerializableAlgorithm_FINAL_SAME_SI"` // snapshot read strategy
   - Conflict Serializable Algorithm：`"ConflictSerializableAlgorithm"`
+  - Dynamic Line Intersect (Identify Anomaly): `"DLI_IDENTIFY"`
   - Serializable Snapshot Isolation：`"SSI"`
   - Write-Snapshot Isolation：`"WSI"`
   - Backward Optimistic Concurrency Control：`"BOCC"`

config.cfg.template

@@ -21,7 +21,7 @@ BenchmarkRun = {
   algorithms = ("SSI"); // concurrent algorithms
   history_num = 1024L; // number of histories to generate
 	with_abort = true; // generate history with Abort operation
-	tail_tcl = true; // generate TCL operation only on the tail of history
+	tcl_position = "TAIL"; // generate TCL operation in history position ("TAIL", "ANYWHERE", "NOWHERE")
   os = "cout"; // filename the benchmark result output to, "cout" means standard output
 };
 
@@ -35,7 +35,9 @@ TraversalGenerator = {
 	subtask_num = 10L; // number of subtasks
 	subtask_id = 0L; // the id of subtask to run
 	with_abort = true; // generate history with Abort operation
-	tail_tcl = false; // generate TCL operation only on the tail of history
+  with_scan = "NONE_HAVE"; // generate history with ScanOdd operation ("NONE_HAVE", "ALL_HAVE", "NO_LIMIT")
+  with_write = "NO_LIMIT"; // generate history with Write operation ("NONE_HAVE", "ALL_HAVE", "NO_LIMIT")
+	tcl_position = "ANYWHERE"; // generate TCL operation in history position ("TAIL", "ANYWHERE", "NOWHERE")
   allow_empty_trans = false; // transactions generated can be without DML operations
   dynamic_history_len = false; // number of DML operation can be less than <max_dml>
 };
@@ -52,7 +54,8 @@ RandomGenerator = {
 	max_dml = 2L; // max number of DML operations
   history_num = 100L; // number of histories to generate
 	with_abort = true; // generate history with Abort operation
-	tail_tcl = true; // generate TCL operation only on the tail of history
+  with_scan = "NONE_HAVE"; // generate history with ScanOdd operation ("NONE_HAVE", "ALL_HAVE", "NO_LIMIT")
+	tcl_position = "TAIL"; // generate TCL operation in history position ("TAIL", "ANYWHERE", "NOWHERE")
   allow_empty_trans = false; // transactions generated can be without DML operations
   dynamic_history_len = false; // number of DML operation can be less than <max_dml>
 }

contrib/deneva/Makefile

@@ -1,12 +1,12 @@
-CC=/usr/bin/g++
-CFLAGS=-Wall -Werror -std=c++11 -g3 -ggdb -O0 -fno-strict-aliasing -fno-omit-frame-pointer -D_GLIBCXX_USE_CXX11_ABI=0
+CC=g++
+CFLAGS=-Wall -Werror -std=c++17 -static-libstdc++ -g3 -ggdb -O0 -fno-strict-aliasing -fno-omit-frame-pointer -D_GLIBCXX_USE_CXX11_ABI=0 -static-libasan -fsanitize=address
 #CFLAGS += -fsanitize=address -fno-stack-protector -fno-omit-frame-pointer
 NNMSG=./nanomsg-0.5-beta
 
 .SUFFIXES: .o .cpp .h .cc
 
-SRC_DIRS = ./ ./benchmarks/ ./client/ ./concurrency_control/ ./storage/ ./transport/ ./system/ ./statistics/#./unit_tests/
-DEPS = -I. -I./benchmarks -I./client/ -I./concurrency_control -I./storage -I./transport -I./system -I./statistics #-I./unit_tests
+SRC_DIRS = ./ ./benchmarks/ ./client/ ./concurrency_control/ ./unified_concurrency_control/ ./storage/ ./transport/ ./system/ ./statistics/#./unit_tests/
+DEPS = -I. -I./benchmarks -I./client/ -I./concurrency_control -I./unified_concurrency_control/ -I./storage -I./transport -I./system -I./statistics #-I./unit_tests
 
 CFLAGS += $(DEPS) -D NOGRAPHITE=1 -Wno-sizeof-pointer-memaccess
 LDFLAGS = -Wall -L. -L$(NNMSG) -Wl,-rpath -pthread -lrt -lnanomsg -lanl -lcurl -lprotobuf -lpthread
@@ -55,6 +55,8 @@ unit_test : $(OBJS_UNIT)
 	$(CC) -c -DSTATS_ENABLE=false $(CFLAGS) $(INCLUDE) -o $@ $<
 ./obj/%.o: concurrency_control/%.cpp
 	$(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $<
+./obj/%.o: unified_concurrency_control/%.cpp
+	$(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $<
 ./obj/%.o: client/%.cpp
 	$(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $<
 ./obj/%.o: %.cpp
@@ -79,6 +81,8 @@ rundb : $(OBJS_DB)
 	$(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $<
 ./obj/%.o: concurrency_control/%.cpp
 	$(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $<
+./obj/%.o: unified_concurrency_control/%.cpp
+	$(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $<
 ./obj/%.o: client/%.cpp
 	$(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $<
 ./obj/%.o: %.cpp
@@ -103,6 +107,8 @@ runcl : $(OBJS_CL)
 	$(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $<
 ./obj/%.o: concurrency_control/%.cpp
 	$(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $<
+./obj/%.o: unified_concurrency_control/%.cpp
+	$(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $<
 ./obj/%.o: client/%.cpp
 	$(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $<
 ./obj/%.o: %.cpp

contrib/deneva/README.md

@@ -43,3 +43,45 @@ The DBMS can be run with
     ./runcl -nid[M]
 
 where N and M are the ID of a server and client, respectively
+
+
+DA 
+---
+The DBMS can use DA workload. This workload will executes a given sequence of transaction operations and prints out the actual execution results.
+
+To use this workload, you can only use a single node, a single worker thread, and a single messaging thread.
+Here are some of the configurations that need to be modified in the `config.h` file
+
+    #define NODE_CNT 1
+    #define THREAD_CNT 1
+    #define REM_THREAD_CNT 1
+    #define SEND_THREAD_CNT 1
+
+    #define CLIENT_NODE_CNT 1
+    #define CLIENT_THREAD_CNT 1
+    #define CLIENT_REM_THREAD_CNT 1
+    #define CLIENT_SEND_THREAD_CNT 1
+
+    #define WORKLOAD DA
+
+In addition, the client and server need to be placed only on one machine!
+Only two lines of the same IP address can be written in the `ifconfig.txt` file, and this IP address is the machine you want to test.
+Here is an example of this file:
+
+    10.77.110.148
+    10.77.110.148
+
+After modifying all the above parameters, the next step is to determine the sequence of transaction operations to be performed. This sequence needs to be written in the `input.txt` file. Examples are as follows:
+
+    W0a R1b W1a R1c C1 W0b C0
+    R2a R3b W2b W3a C2 C3
+
+A row represents a sequence.
+
+Now to test, you need to perform the following command on the machine which you want to test in:
+
+    ./rundb -nid0
+    ./runcl -nid1
+
+Finally, check the results, which are output in the `commit_histroy.txt` file.
+Compare whether the actual execution results in the file meet the logic of your concurrency control algorithm. If so, it is proved that the algorithm is implemented correctly.

contrib/deneva/benchmarks/da.h

@@ -6,6 +6,8 @@
 #include "txn.h"
 #include "wl.h"
 #include "creator.h"
+#include "da_query.h"
+#include <optional>
 
 class DAQuery;
 class DAQueryMessage;
@@ -15,51 +17,62 @@ class table_t;
 class INDEX;
 class DAQuery;
 
+extern std::string DA_history_mem;
+extern std::vector<Message*> DA_delayed_operations;
+extern bool abort_history;
+extern ofstream commit_file;
+extern ofstream abort_file;
+extern std::optional<ttts::Path> g_da_cycle;
+
 class DAWorkload : public Workload {
- public:
-  RC init();
-  RC init_table();
-  RC init_schema(const char* schema_file);
-  RC get_txn_man(TxnManager*& txn_manager);
-  void reset_tab_idx();
-  table_t* t_datab;
+  public:
+    RC init();
+    RC init_table();
+    RC init_schema(const char* schema_file);
+    RC get_txn_man(TxnManager*& txn_manager);
+    void reset_tab_idx();
+    table_t* t_datab;
     uint64_t nextstate;
-  INDEX* i_datab;
-  bool** delivering;
+    INDEX* i_datab;
+    bool** delivering;
 
- private:
-  //void init_tab_DAtab(int id, uint64_t w_id);
-  void init_tab_DAtab();
-  static void* threadInitDAtab(void* This);
+  private:
+    //void init_tab_DAtab(int id, uint64_t w_id);
+    void init_tab_DAtab();
+    static void* threadInitDAtab(void* This);
 };
 
 struct DA_thr_args {
-  DAWorkload* wl;
-  UInt32 id;
-  UInt32 tot;
+    DAWorkload* wl;
+    UInt32 id;
+    UInt32 tot;
 };
 
 class DATxnManager : public TxnManager {
- public:
-  void init(uint64_t thd_id, Workload* h_wl);
-  void reset();
-  RC acquire_locks();
-  RC run_txn();
-  RC run_txn_post_wait();
-  RC run_calvin_txn();
+  public:
+    void init(uint64_t thd_id, Workload* h_wl);
+    void reset();
+    RC acquire_locks();
+    RC run_txn();
+    RC run_txn_post_wait();
+    RC run_calvin_txn();
 
-  void copy_remote_items(DAQueryMessage* msg);
+    void copy_remote_items(DAQueryMessage* msg);
+    void set_not_waiting() { is_waiting_ = false; }
 
- private:
-  DAWorkload* _wl;
-  volatile RC _rc;
-  row_t* row;
+  private:
+    DAWorkload* _wl;
+    volatile RC _rc;
+    row_t* row;
+    bool is_waiting_;
+    std::vector<DAQuery> skip_queries_;
 
-  uint64_t next_item_id;
+    uint64_t next_item_id;
 
-  bool is_done();
-  bool is_local_item(uint64_t idx);
-  RC send_remote_request();
-  RC run_delivery(DAQuery* query);
+    bool is_done();
+    bool is_local_item(uint64_t idx);
+    RC send_remote_request();
+    RC run_delivery(DAQuery* query);
+    RC process_query(const DAQuery* const da_query);
 };
 #endif

contrib/deneva/benchmarks/da_block_queue.cpp

@@ -53,7 +53,6 @@ void DABlockQueue::push_data(BaseQuery* data)
     while(IsFull())
     {
         NotifyConsume();
-        std::cout<<"queue full,notify consume data,product stop!!"<<std::endl;
         ProductWait();
     }
 
@@ -68,7 +67,6 @@ BaseQuery* DABlockQueue::pop_data()
     while(IsEmpty())
     {
         NotifyProduct();
-        std::cout<<"queue empty,notify product data,consume stop!!"<<std::endl;
         list<BaseQuery*>().swap(q);
         ConsumeWait();
     }

contrib/deneva/benchmarks/da_query.cpp

@@ -117,23 +117,17 @@ BaseQuery * DAQueryGenerator::create_query(Workload * h_wl, uint64_t home_partit
           t_version[DAQ_t->item_id]++;
         DAQ_t->write_version=t_version[DAQ_t->item_id];
         da_gen_qry_queue.push_data(DAQ_t);
-        /*
-        while(!(pu=da_query_queue.push(DAQ_t)));
-        if(pu)
-          printf("true ");
-        else
-          printf("false ");
-        fflush(stdout);
-        */
+
     }
     free(t_version);
     seq_num++;
-    printf("product: %lu\n",seq_num);
     fflush(stdout);
   };
 
+  printf("start generating histories\n");
   creator.DeliverActionSequences(handle);
-  printf("history thread exit\n");
+  da_gen_qry_queue.push_data(nullptr); // tell consumer no histories anymore
+  printf("finish generating histories, total product: %lu\n", seq_num);
   fflush(stdout);
   return ret;
 }

contrib/deneva/benchmarks/da_schema.txt

@@ -0,0 +1,7 @@
+//size,type,name
+TABLE=DAtab
+    8,int64_t,id
+    8,int64_t,value
+
+INDEX=DAtab_IDX
+DAtab,0