diff --git a/.gitignore b/.gitignore index 66e84ace..8a6c8120 100644 --- a/.gitignore +++ b/.gitignore @@ -6,6 +6,8 @@ __* *.o *.bak core* -3ts rundb runcl +3TS +3TS-DAI +contrib/deneva/obj/deps diff --git a/LICENSE.txt b/LICENSE.txt index 0057ddc3..53877e30 100644 --- a/LICENSE.txt +++ b/LICENSE.txt @@ -1,7 +1,6 @@ Tencent is pleased to support the open source community by making Tencent Transaction Processing Testbed System-(3TS--腾讯事务处理技术验证系统) available. -Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All Tencent Modifications are Copyright (C) THL A29 Limited. - +Copyright © 2020 Tencent Holdings Limited, a Tencent company. All rights reserved. The software in this distribution may have been modified by Tencent Holdings Limited (“Tencent Modifications”). All Tencent Modifications are Copyright © Tencent Holdings. Tencent Transaction Processing Testbed System-(3TS--腾讯事务处理技术验证系统) is licensed under the following license, except for the third-party components listed below. License for Tencent Transaction Processing Testbed System-(3TS--腾讯事务处理技术验证系统) : diff --git a/README.md b/README.md index 7057de41..99b0abd7 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ ![](https://img.shields.io/badge/license-GPLv3-brightgreen) -**Tencent Transaction Processing Testbed System (3TS)** that is jointly developed by Tencent's CynosDB (TDSQL) team and the Key Laboratory of Data Engineering and Knowledge Engineering of the Ministry of Education of Renmin University of China. The system aims to design and construct a unified framework for transaction processing (including distributed transactions). It enables users to quickly build new concurrency control approaches via the access interface provided by the framework. Based on an comprehensive experiment study over the benchmarks, and the applications abstracted, users can select an optimal concurrency control approach. At present, 3TS has been integrated 13 mainstream concurrency control approaches, and provides common benchmarks such as TPC-C、PPS and YCSB. 3TS further provides a consistency level test benchmark, to address the issue of system selection difficulty caused by the blowout development of distributed database systems, and provides consistency level discrimination and performance test comparison. +**Tencent Transaction Processing Testbed System (3TS)** that is jointly developed by Tencent's TDSQL team and the Key Laboratory of Data Engineering and Knowledge Engineering of the Ministry of Education of Renmin University of China. The system aims to design and construct a unified framework for transaction processing (including distributed transactions). It enables users to quickly build new concurrency control approaches via the access interface provided by the framework. Based on an comprehensive experiment study over the benchmarks, and the applications abstracted, users can select an optimal concurrency control approach. At present, 3TS has been integrated 13 mainstream concurrency control approaches, and provides common benchmarks such as TPC-C，PPS and YCSB. 3TS further provides a consistency level test benchmark, to address the issue of system selection difficulty caused by the blowout development of distributed database systems, and provides consistency level discrimination and performance test comparison. If you want to better understand the aims of our project, please view [3TS opensource announcement](doc/en/announcement.md). @@ -16,7 +16,13 @@ If you want to better understand the aims of our project, please view [3TS opens ## Dependence +### 3TS-Coo +A consistency verification system. Please check out the newest update on branch ['coo-consistency-check'](https://github.com/Tencent/3TS/tree/coo-consistency-check). + +We update our result on [report webpage](https://axingguchen.github.io/3TS/). + ### 3TS-DA +A static random history generator. Please check out the newest update on branch ['dev'](https://github.com/Tencent/3TS/tree/dev). - a compilter supporting C++17 or upper versions (recommend g++8) - libconfig 1.7.2 @@ -24,6 +30,7 @@ If you want to better understand the aims of our project, please view [3TS opens - gtest 1.6.0 ### Deneva +A performance verification system. A consistency verification system. Please check out the newest update on branch ['dev'](https://github.com/Tencent/3TS/tree/dev). - protobuf 3.9.1 - curl 7.29.0 @@ -31,10 +38,10 @@ If you want to better understand the aims of our project, please view [3TS opens ## Usage -- Run `make.sh` to compile the code. The `3ts` binary will be generated if compiling successfully. +- Run `make.sh` to compile the code. The `3TS` binary will be generated if compiling successfully. - Run `cp config/config.cfg.template config.cfg` to copy the configuration file. - Run `vi config.cfg` to modify the configuration file to determine the behavior of the testbed. -- Run `./t3s --conf_path=config.cfg` to execute test. The test result file will be generated when test is over. +- Run `./3TS --conf_path=config.cfg` to execute test. The test result file will be generated when test is over. ## Principle @@ -62,6 +69,7 @@ If you want to better understand the aims of our project, please view [3TS opens - `"SerializableAlgorithm_FINAL_SAME_RR"` // repeatable read strategy - `"SerializableAlgorithm_FINAL_SAME_SI"` // snapshot read strategy - Conflict Serializable Algorithm：`"ConflictSerializableAlgorithm"` + - Dynamic Line Intersect (Identify Anomaly): `"DLI_IDENTIFY"` - Serializable Snapshot Isolation：`"SSI"` - Write-Snapshot Isolation：`"WSI"` - Backward Optimistic Concurrency Control：`"BOCC"` diff --git a/config.cfg.template b/config.cfg.template index 88d02b27..fa3f60ef 100644 --- a/config.cfg.template +++ b/config.cfg.template @@ -21,7 +21,7 @@ BenchmarkRun = { algorithms = ("SSI"); // concurrent algorithms history_num = 1024L; // number of histories to generate with_abort = true; // generate history with Abort operation - tail_tcl = true; // generate TCL operation only on the tail of history + tcl_position = "TAIL"; // generate TCL operation in history position ("TAIL", "ANYWHERE", "NOWHERE") os = "cout"; // filename the benchmark result output to, "cout" means standard output }; @@ -35,7 +35,9 @@ TraversalGenerator = { subtask_num = 10L; // number of subtasks subtask_id = 0L; // the id of subtask to run with_abort = true; // generate history with Abort operation - tail_tcl = false; // generate TCL operation only on the tail of history + with_scan = "NONE_HAVE"; // generate history with ScanOdd operation ("NONE_HAVE", "ALL_HAVE", "NO_LIMIT") + with_write = "NO_LIMIT"; // generate history with Write operation ("NONE_HAVE", "ALL_HAVE", "NO_LIMIT") + tcl_position = "ANYWHERE"; // generate TCL operation in history position ("TAIL", "ANYWHERE", "NOWHERE") allow_empty_trans = false; // transactions generated can be without DML operations dynamic_history_len = false; // number of DML operation can be less than }; @@ -52,7 +54,8 @@ RandomGenerator = { max_dml = 2L; // max number of DML operations history_num = 100L; // number of histories to generate with_abort = true; // generate history with Abort operation - tail_tcl = true; // generate TCL operation only on the tail of history + with_scan = "NONE_HAVE"; // generate history with ScanOdd operation ("NONE_HAVE", "ALL_HAVE", "NO_LIMIT") + tcl_position = "TAIL"; // generate TCL operation in history position ("TAIL", "ANYWHERE", "NOWHERE") allow_empty_trans = false; // transactions generated can be without DML operations dynamic_history_len = false; // number of DML operation can be less than } diff --git a/contrib/deneva/Makefile b/contrib/deneva/Makefile index 5390c9e5..90b32c27 100644 --- a/contrib/deneva/Makefile +++ b/contrib/deneva/Makefile @@ -1,12 +1,12 @@ -CC=/usr/bin/g++ -CFLAGS=-Wall -Werror -std=c++11 -g3 -ggdb -O0 -fno-strict-aliasing -fno-omit-frame-pointer -D_GLIBCXX_USE_CXX11_ABI=0 +CC=g++ +CFLAGS=-Wall -Werror -std=c++17 -static-libstdc++ -g3 -ggdb -O0 -fno-strict-aliasing -fno-omit-frame-pointer -D_GLIBCXX_USE_CXX11_ABI=0 -static-libasan -fsanitize=address #CFLAGS += -fsanitize=address -fno-stack-protector -fno-omit-frame-pointer NNMSG=./nanomsg-0.5-beta .SUFFIXES: .o .cpp .h .cc -SRC_DIRS = ./ ./benchmarks/ ./client/ ./concurrency_control/ ./storage/ ./transport/ ./system/ ./statistics/#./unit_tests/ -DEPS = -I. -I./benchmarks -I./client/ -I./concurrency_control -I./storage -I./transport -I./system -I./statistics #-I./unit_tests +SRC_DIRS = ./ ./benchmarks/ ./client/ ./concurrency_control/ ./unified_concurrency_control/ ./storage/ ./transport/ ./system/ ./statistics/#./unit_tests/ +DEPS = -I. -I./benchmarks -I./client/ -I./concurrency_control -I./unified_concurrency_control/ -I./storage -I./transport -I./system -I./statistics #-I./unit_tests CFLAGS += $(DEPS) -D NOGRAPHITE=1 -Wno-sizeof-pointer-memaccess LDFLAGS = -Wall -L. -L$(NNMSG) -Wl,-rpath -pthread -lrt -lnanomsg -lanl -lcurl -lprotobuf -lpthread @@ -55,6 +55,8 @@ unit_test : $(OBJS_UNIT) $(CC) -c -DSTATS_ENABLE=false $(CFLAGS) $(INCLUDE) -o $@ $< ./obj/%.o: concurrency_control/%.cpp $(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $< +./obj/%.o: unified_concurrency_control/%.cpp + $(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $< ./obj/%.o: client/%.cpp $(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $< ./obj/%.o: %.cpp @@ -79,6 +81,8 @@ rundb : $(OBJS_DB) $(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $< ./obj/%.o: concurrency_control/%.cpp $(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $< +./obj/%.o: unified_concurrency_control/%.cpp + $(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $< ./obj/%.o: client/%.cpp $(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $< ./obj/%.o: %.cpp @@ -103,6 +107,8 @@ runcl : $(OBJS_CL) $(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $< ./obj/%.o: concurrency_control/%.cpp $(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $< +./obj/%.o: unified_concurrency_control/%.cpp + $(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $< ./obj/%.o: client/%.cpp $(CC) -c $(CFLAGS) $(INCLUDE) -o $@ $< ./obj/%.o: %.cpp diff --git a/contrib/deneva/README.md b/contrib/deneva/README.md index d5ae2508..53af0229 100644 --- a/contrib/deneva/README.md +++ b/contrib/deneva/README.md @@ -43,3 +43,45 @@ The DBMS can be run with ./runcl -nid[M] where N and M are the ID of a server and client, respectively + + +DA +--- +The DBMS can use DA workload. This workload will executes a given sequence of transaction operations and prints out the actual execution results. + +To use this workload, you can only use a single node, a single worker thread, and a single messaging thread. +Here are some of the configurations that need to be modified in the `config.h` file + + #define NODE_CNT 1 + #define THREAD_CNT 1 + #define REM_THREAD_CNT 1 + #define SEND_THREAD_CNT 1 + + #define CLIENT_NODE_CNT 1 + #define CLIENT_THREAD_CNT 1 + #define CLIENT_REM_THREAD_CNT 1 + #define CLIENT_SEND_THREAD_CNT 1 + + #define WORKLOAD DA + +In addition, the client and server need to be placed only on one machine! +Only two lines of the same IP address can be written in the `ifconfig.txt` file, and this IP address is the machine you want to test. +Here is an example of this file: + + 10.77.110.148 + 10.77.110.148 + +After modifying all the above parameters, the next step is to determine the sequence of transaction operations to be performed. This sequence needs to be written in the `input.txt` file. Examples are as follows: + + W0a R1b W1a R1c C1 W0b C0 + R2a R3b W2b W3a C2 C3 + +A row represents a sequence. + +Now to test, you need to perform the following command on the machine which you want to test in: + + ./rundb -nid0 + ./runcl -nid1 + +Finally, check the results, which are output in the `commit_histroy.txt` file. +Compare whether the actual execution results in the file meet the logic of your concurrency control algorithm. If so, it is proved that the algorithm is implemented correctly. \ No newline at end of file diff --git a/contrib/deneva/benchmarks/da.h b/contrib/deneva/benchmarks/da.h index 967d2189..e2c118bf 100644 --- a/contrib/deneva/benchmarks/da.h +++ b/contrib/deneva/benchmarks/da.h @@ -6,6 +6,8 @@ #include "txn.h" #include "wl.h" #include "creator.h" +#include "da_query.h" +#include class DAQuery; class DAQueryMessage; @@ -15,51 +17,62 @@ class table_t; class INDEX; class DAQuery; +extern std::string DA_history_mem; +extern std::vector DA_delayed_operations; +extern bool abort_history; +extern ofstream commit_file; +extern ofstream abort_file; +extern std::optional g_da_cycle; + class DAWorkload : public Workload { - public: - RC init(); - RC init_table(); - RC init_schema(const char* schema_file); - RC get_txn_man(TxnManager*& txn_manager); - void reset_tab_idx(); - table_t* t_datab; + public: + RC init(); + RC init_table(); + RC init_schema(const char* schema_file); + RC get_txn_man(TxnManager*& txn_manager); + void reset_tab_idx(); + table_t* t_datab; uint64_t nextstate; - INDEX* i_datab; - bool** delivering; + INDEX* i_datab; + bool** delivering; - private: - //void init_tab_DAtab(int id, uint64_t w_id); - void init_tab_DAtab(); - static void* threadInitDAtab(void* This); + private: + //void init_tab_DAtab(int id, uint64_t w_id); + void init_tab_DAtab(); + static void* threadInitDAtab(void* This); }; struct DA_thr_args { - DAWorkload* wl; - UInt32 id; - UInt32 tot; + DAWorkload* wl; + UInt32 id; + UInt32 tot; }; class DATxnManager : public TxnManager { - public: - void init(uint64_t thd_id, Workload* h_wl); - void reset(); - RC acquire_locks(); - RC run_txn(); - RC run_txn_post_wait(); - RC run_calvin_txn(); + public: + void init(uint64_t thd_id, Workload* h_wl); + void reset(); + RC acquire_locks(); + RC run_txn(); + RC run_txn_post_wait(); + RC run_calvin_txn(); - void copy_remote_items(DAQueryMessage* msg); + void copy_remote_items(DAQueryMessage* msg); + void set_not_waiting() { is_waiting_ = false; } - private: - DAWorkload* _wl; - volatile RC _rc; - row_t* row; + private: + DAWorkload* _wl; + volatile RC _rc; + row_t* row; + bool is_waiting_; + std::vector skip_queries_; - uint64_t next_item_id; + uint64_t next_item_id; - bool is_done(); - bool is_local_item(uint64_t idx); - RC send_remote_request(); - RC run_delivery(DAQuery* query); + bool is_done(); + bool is_local_item(uint64_t idx); + RC send_remote_request(); + RC run_delivery(DAQuery* query); + RC process_query(const DAQuery* const da_query); }; #endif diff --git a/contrib/deneva/benchmarks/da_block_queue.cpp b/contrib/deneva/benchmarks/da_block_queue.cpp index 8bf0dd61..56677c76 100644 --- a/contrib/deneva/benchmarks/da_block_queue.cpp +++ b/contrib/deneva/benchmarks/da_block_queue.cpp @@ -53,7 +53,6 @@ void DABlockQueue::push_data(BaseQuery* data) while(IsFull()) { NotifyConsume(); - std::cout<<"queue full,notify consume data,product stop!!"<().swap(q); ConsumeWait(); } diff --git a/contrib/deneva/benchmarks/da_query.cpp b/contrib/deneva/benchmarks/da_query.cpp index 1c349831..8bac4249 100644 --- a/contrib/deneva/benchmarks/da_query.cpp +++ b/contrib/deneva/benchmarks/da_query.cpp @@ -117,23 +117,17 @@ BaseQuery * DAQueryGenerator::create_query(Workload * h_wl, uint64_t home_partit t_version[DAQ_t->item_id]++; DAQ_t->write_version=t_version[DAQ_t->item_id]; da_gen_qry_queue.push_data(DAQ_t); - /* - while(!(pu=da_query_queue.push(DAQ_t))); - if(pu) - printf("true "); - else - printf("false "); - fflush(stdout); - */ + } free(t_version); seq_num++; - printf("product: %lu\n",seq_num); fflush(stdout); }; + printf("start generating histories\n"); creator.DeliverActionSequences(handle); - printf("history thread exit\n"); + da_gen_qry_queue.push_data(nullptr); // tell consumer no histories anymore + printf("finish generating histories, total product: %lu\n", seq_num); fflush(stdout); return ret; } diff --git a/contrib/deneva/benchmarks/da_schema.txt b/contrib/deneva/benchmarks/da_schema.txt new file mode 100644 index 00000000..eba94812 --- /dev/null +++ b/contrib/deneva/benchmarks/da_schema.txt @@ -0,0 +1,7 @@ +//size,type,name +TABLE=DAtab + 8,int64_t,id + 8,int64_t,value + +INDEX=DAtab_IDX +DAtab,0 diff --git a/contrib/deneva/benchmarks/da_txn.cpp b/contrib/deneva/benchmarks/da_txn.cpp index 2f87cfa8..edf4ab82 100644 --- a/contrib/deneva/benchmarks/da_txn.cpp +++ b/contrib/deneva/benchmarks/da_txn.cpp @@ -15,159 +15,192 @@ #include "transport.h" #include "wl.h" +std::string DA_history_mem; +std::vector DA_delayed_operations; +bool abort_history; +ofstream commit_file; +ofstream abort_file; +std::optional g_da_cycle; + void DATxnManager::init(uint64_t thd_id, Workload *h_wl) { - TxnManager::init(thd_id, h_wl); - _wl = (DAWorkload *)h_wl; - reset(); + TxnManager::init(thd_id, h_wl); + _wl = (DAWorkload *)h_wl; + reset(); } + RC DATxnManager::run_txn_post_wait() { - get_row_post_wait(row); - return RCOK; + get_row_post_wait(row); + return RCOK; } -RC DATxnManager::acquire_locks(){return RCOK;} -RC DATxnManager::run_calvin_txn(){return RCOK;} -void DATxnManager::reset(){TxnManager::reset();} -RC DATxnManager::run_txn() { -#if MODE == SETUP_MODE - return RCOK; -#endif - RC rc = RCOK; - //uint64_t starttime = get_sys_clock(); - if (IS_LOCAL(txn->txn_id)) { - DEBUG("Running txn %ld\n", txn->txn_id); -#if DISTR_DEBUG - query->print(); -#endif - query->partitions_touched.add_unique(GET_PART_ID(0, g_node_id)); - } - DAQuery *da_query = (DAQuery *)query; - uint64_t trans_id = da_query->trans_id; - uint64_t item_id = da_query->item_id; // item_id from 0-2 represent X,Y,Z - //uint64_t seq_id = da_query->seq_id; - uint64_t state = da_query->state; - uint64_t version = da_query->write_version; - //uint64_t next_state = da_query->next_state; - //uint64_t last_state = da_query->last_state; - DATxnType txn_type = da_query->txn_type; - bool jump=false; - switch (txn_type) - { - case DA_WRITE: - DA_history_mem.push_back('W'); - break; - case DA_READ: - DA_history_mem.push_back('R'); - break; - case DA_COMMIT: - DA_history_mem.push_back('C'); - break; - case DA_ABORT: - DA_history_mem.push_back('A'); - break; - case DA_SCAN: - DA_history_mem.push_back('S'); - break; - } - DA_history_mem.push_back(static_cast('0'+trans_id));//trans_id - if(txn_type==DA_WRITE || txn_type==DA_READ) - DA_history_mem.push_back(static_cast('a'+item_id));//item_id - DA_history_mem.push_back(' '); - #if WORKLOAD ==DA - printf("thd_id:%lu check: state:%lu nextstate:%lu \n",h_thd->_thd_id, state, _wl->nextstate); - fflush(stdout); - #endif - if(_wl->nextstate!=0) - { - while (state != _wl->nextstate&&!simulation->is_done()); - } +RC DATxnManager::acquire_locks() { return RCOK; } +RC DATxnManager::run_calvin_txn() { return RCOK; } +void DATxnManager::reset() { + is_waiting_ = false; + skip_queries_.clear(); + TxnManager::reset(); +} - if(already_abort_tab.count(trans_id)>0) - { - if(txn_type==DA_WRITE || txn_type==DA_READ||txn_type==DA_COMMIT||txn_type==DA_ABORT) - { - jump=true; - if(txn_type==DA_ABORT) - INC_STATS(get_thd_id(), positive_txn_abort_cnt, 1); - } - //else if(txn_type==DA_COMMIT) - //txn_type=DA_ABORT; - } +RC DATxnManager::process_query(const DAQuery* const da_query) { + RC rc = RCOK; + uint64_t trans_id = da_query->trans_id; + uint64_t item_id = da_query->item_id; // item_id from 0-2 represent X,Y,Z + assert(trans_id < TRANS_CNT); + assert(item_id < ITEM_CNT); + //uint64_t seq_id = da_query->seq_id; + uint64_t version = da_query->write_version; + //uint64_t next_state = da_query->next_state; + //uint64_t last_state = da_query->last_state; + DATxnType txn_type = da_query->txn_type; - if(!jump) - { //enum RC { RCOK = 0, Commit, Abort, WAIT, WAIT_REM, ERROR, FINISH, NONE }; - itemid_t *item; - INDEX *index = _wl->i_datab; - uint64_t value[3]; + INDEX* index = _wl->i_datab; + uint64_t value[ITEM_CNT]; - item = index_read(index, item_id, 0); - assert(item != NULL); - row_t *TempRow = ((row_t *)item->location); - - switch (txn_type) { - case DA_WRITE: { - rc = get_row(TempRow, WR, row); - if(rc == RCOK) - row->set_value(VALUE, version); - else - { - rc = start_abort(); - already_abort_tab.insert(trans_id); - } - break; - } - case DA_READ: { - rc = get_row(TempRow, RD, row); - if(rc == RCOK) - row->get_value(VALUE, value[0]); - else - { - rc = start_abort(); - already_abort_tab.insert(trans_id); - } - break; + if (is_waiting_) { + rc = WAIT; + } else { + switch (txn_type) { + case DA_WRITE: { + itemid_t* item = index_read(index, item_id, 0); + assert(item != NULL); + row_t* TempRow = ((row_t *)item->location); + rc = get_row(TempRow, WR, row); + if (rc == RCOK) { + row->set_value(VALUE, version); + } else if (rc == Abort){ + rc = start_abort(); + already_abort_tab.insert(trans_id); + } + break; + } + case DA_READ: { + itemid_t* item = index_read(index, item_id, 0); + assert(item != NULL); + row_t* TempRow = ((row_t *)item->location); + rc = get_row(TempRow, RD, row); + if (rc == RCOK) { + row->get_value(VALUE, value[0]); + } else if (rc == Abort) { + rc = start_abort(); + already_abort_tab.insert(trans_id); + } + break; + } + case DA_COMMIT: { + rc = start_commit(); + break; + } + case DA_ABORT: { + INC_STATS(get_thd_id(), positive_txn_abort_cnt, 1); + rc = start_abort(); + break; + } + case DA_SCAN: { + row_t *TempRow; + for (int i = 0; i < ITEM_CNT && rc == RCOK; i++) { + itemid_t* item = index_read(index, i, 0); + assert(item != NULL); + TempRow = ((row_t *)item->location); + rc = get_row(TempRow, SCAN, row); + if (rc == RCOK) { + row->get_value(VALUE, value[i]); + } else if (rc == Abort) { + rc = start_abort(); + already_abort_tab.insert(trans_id); + } + } + break; + } } - case DA_COMMIT: { - rc=start_commit(); - break; + } + + // print operation + switch (txn_type) { + case DA_WRITE: + DA_history_mem.push_back('W'); + break; + case DA_READ: + DA_history_mem.push_back('R'); + break; + case DA_COMMIT: + DA_history_mem.push_back('C'); + break; + case DA_ABORT: + DA_history_mem.push_back('A'); + break; + case DA_SCAN: + DA_history_mem.push_back('S'); + break; + } + DA_history_mem.push_back(static_cast('0' + trans_id));//trans_id + if (txn_type == DA_WRITE || txn_type == DA_READ) { + DA_history_mem.push_back(static_cast('a' + item_id));//item_id + } + DA_history_mem.push_back('='); + + // print result + if (rc == Abort) { + rc = start_abort(); + DA_history_mem += "Abort "; + if (txn_type != DA_ABORT) { + abort_history = true; } - case DA_ABORT: { - INC_STATS(get_thd_id(), positive_txn_abort_cnt, 1); - rc = start_abort(); - break; + } else if (rc == WAIT) { + DA_history_mem += "Wait "; + is_waiting_ = true; + skip_queries_.emplace_back(*da_query); + } else if (rc == RCOK) { + if (txn_type == DA_WRITE) { + DA_history_mem.push_back(static_cast('0' + version));//item_id + } else if (txn_type == DA_READ) { + DA_history_mem.push_back(static_cast('0' + value[0]));//item_id + } else if (txn_type == DA_SCAN) { + DA_history_mem.push_back('{'); + for (const auto v : value) { + DA_history_mem.push_back(static_cast('0' + v));//item_id + DA_history_mem.push_back(','); + } + DA_history_mem.back() = '}'; } - case DA_SCAN: { - row_t *TempRow; - for (int i = 0; i < ITEM_CNT; i++) { - item = index_read(index, item_id, 0); - assert(item != NULL); - TempRow = ((row_t *)item->location); - rc = get_row(TempRow, WR, row); - row->get_value(VALUE, value[i]); - } - break; + DA_history_mem.push_back(' '); + } else if (rc == Commit) { + DA_history_mem += "Commit "; + } else { + DA_history_mem += "rc"; + DA_history_mem += std::to_string(rc); + DA_history_mem.push_back(' '); + } + + return rc; +} + +RC DATxnManager::run_txn() { +#if MODE == SETUP_MODE + return RCOK; +#endif + //uint64_t starttime = get_sys_clock(); + if (IS_LOCAL(txn->txn_id)) { + DEBUG("Running txn %ld\n", txn->txn_id); +#if DISTR_DEBUG + query->print(); +#endif + query->partitions_touched.add_unique(GET_PART_ID(0, g_node_id)); + } + if (!is_waiting_ && !skip_queries_.empty()) { + // only happen in RTXN_CONT, and this->query has been stored in skip_queries_ + RC rc = RCOK; + std::vector skip_queries; + std::swap(skip_queries_, skip_queries); + for (const DAQuery& query : skip_queries) { + DA_history_mem += "*"; + rc = process_query(&query); + if (rc == Abort || rc == Commit) { + return rc; } } - //if(rc==Abort||rc==WAIT) - //{ - // rc = start_abort(); - //} - } - _wl->nextstate = da_query->next_state; - if(_wl->nextstate==0) - { - if(abort_history) - abort_file<reset_tab_idx(); - already_abort_tab.clear(); - da_start_trans_tab.clear(); - } - return rc; + return RCOK; + } else { + return process_query((DAQuery *)query); + } } diff --git a/contrib/deneva/benchmarks/da_wl.cpp b/contrib/deneva/benchmarks/da_wl.cpp index 9d979365..c88f2e97 100644 --- a/contrib/deneva/benchmarks/da_wl.cpp +++ b/contrib/deneva/benchmarks/da_wl.cpp @@ -64,11 +64,11 @@ RC DAWorkload::init_table() { void DAWorkload::init_tab_DAtab() { row_t *row; - uint64_t row_id; - for (int i = 0; i < ITEM_CNT; i++) { + for (int64_t i = 0; i < ITEM_CNT; i++) { + uint64_t row_id = i; t_datab->get_new_row(row, 0, row_id); row->set_value(ID, i); - row->set_value(VALUE, 0); + row->set_value(VALUE, (int64_t)0); //insert_row(row, _wl->t_datab); index_insert(i_datab, i, row, 0); } diff --git a/contrib/deneva/benchmarks/tpcc_query.cpp b/contrib/deneva/benchmarks/tpcc_query.cpp index 81119b65..46305a95 100644 --- a/contrib/deneva/benchmarks/tpcc_query.cpp +++ b/contrib/deneva/benchmarks/tpcc_query.cpp @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn diff --git a/contrib/deneva/benchmarks/tpcc_txn.cpp b/contrib/deneva/benchmarks/tpcc_txn.cpp index effb03d4..71b5b709 100644 --- a/contrib/deneva/benchmarks/tpcc_txn.cpp +++ b/contrib/deneva/benchmarks/tpcc_txn.cpp @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn diff --git a/contrib/deneva/benchmarks/ycsb_query.cpp b/contrib/deneva/benchmarks/ycsb_query.cpp index f5d0ea9c..c841dd7e 100644 --- a/contrib/deneva/benchmarks/ycsb_query.cpp +++ b/contrib/deneva/benchmarks/ycsb_query.cpp @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn diff --git a/contrib/deneva/benchmarks/ycsb_txn.cpp b/contrib/deneva/benchmarks/ycsb_txn.cpp index 294f4a56..ad703202 100644 --- a/contrib/deneva/benchmarks/ycsb_txn.cpp +++ b/contrib/deneva/benchmarks/ycsb_txn.cpp @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn diff --git a/contrib/deneva/client/client_main.cpp b/contrib/deneva/client/client_main.cpp index 75a2da50..41a1f7df 100644 --- a/contrib/deneva/client/client_main.cpp +++ b/contrib/deneva/client/client_main.cpp @@ -158,14 +158,16 @@ int main(int argc, char *argv[]) { warmup_done = true; pthread_barrier_init( &warmup_bar, NULL, all_thd_cnt); - uint64_t cpu_cnt = 0; #if SET_AFFINITY + uint64_t cpu_cnt = 0; cpu_set_t cpus; #endif // spawn and run txns again. starttime = get_server_clock(); simulation->run_starttime = starttime; +#if WORKLOAD == DA simulation->last_da_query_time = starttime; +#endif uint64_t id = 0; for (uint64_t i = 0; i < thd_cnt; i++) { #if SET_AFFINITY diff --git a/contrib/deneva/client/client_query.cpp b/contrib/deneva/client/client_query.cpp index 5ec16a77..f102e6a1 100644 --- a/contrib/deneva/client/client_query.cpp +++ b/contrib/deneva/client/client_query.cpp @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn @@ -59,14 +57,14 @@ Client_query_queue::init(Workload * h_wl) { query_cnt[id] = (uint64_t*)mem_allocator.align_alloc(sizeof(uint64_t)); } next_tid = 0; -/* #if WORKLOAD == DA - pthread_t * p_thds = new pthread_t[1]; - pthread_create(&p_thds[0], NULL, initQueriesHelper, this); - pthread_join(p_thds[0], NULL); + FUNC_ARGS *arg=(FUNC_ARGS*)mem_allocator.align_alloc(sizeof(FUNC_ARGS)); + arg->context=this; + arg->thd_id=g_init_parallelism - 1; + pthread_t p_thds_main; + pthread_create(&p_thds_main, NULL, initQueriesHelper, (void*)arg ); + pthread_detach(p_thds_main); #else -*/ - pthread_t * p_thds = new pthread_t[g_init_parallelism - 1]; for (uint64_t i = 0; i < g_init_parallelism - 1; i++) { FUNC_ARGS *arg=(FUNC_ARGS*)mem_allocator.align_alloc(sizeof(FUNC_ARGS)); @@ -77,17 +75,13 @@ Client_query_queue::init(Workload * h_wl) { FUNC_ARGS *arg=(FUNC_ARGS*)mem_allocator.align_alloc(sizeof(FUNC_ARGS)); arg->context=this; arg->thd_id=g_init_parallelism - 1; - #if WORKLOAD == DA - pthread_t p_thds_main; - pthread_create(&p_thds_main, NULL, initQueriesHelper, (void*)arg ); - pthread_detach(p_thds_main); - #else - initQueriesHelper(arg); - #endif + + initQueriesHelper(arg); for (uint32_t i = 0; i < g_init_parallelism - 1; i++) { pthread_join(p_thds[i], NULL); } +#endif } void * diff --git a/contrib/deneva/concurrency_control/bocc.cpp b/contrib/deneva/concurrency_control/bocc.cpp index 552c4749..6620f03c 100644 --- a/contrib/deneva/concurrency_control/bocc.cpp +++ b/contrib/deneva/concurrency_control/bocc.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/bocc.h b/contrib/deneva/concurrency_control/bocc.h index 3fb7df70..8c9a282b 100644 --- a/contrib/deneva/concurrency_control/bocc.h +++ b/contrib/deneva/concurrency_control/bocc.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/dta.cpp b/contrib/deneva/concurrency_control/dta.cpp new file mode 100644 index 00000000..e5c78d67 --- /dev/null +++ b/contrib/deneva/concurrency_control/dta.cpp @@ -0,0 +1,356 @@ +/* + Copyright 2016 Massachusetts Institute of Technology + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +#include "dta.h" + +#include "../system/global.h" +#include "../system/helper.h" +#include "../system/manager.h" +#include "../system/mem_alloc.h" +#include "../system/txn.h" +#include "row_dta.h" + +void get_rw_set(TxnManager* txn, std::list& rset, std::list& wset) { + uint64_t len = txn->get_access_cnt(); + for (uint64_t i = 0; i < len; i++) { + if (txn->get_access_type(i) == WR) + wset.emplace_back(txn->get_access_original_row(i), txn->get_access_version(i)); + else + rset.emplace_back(txn->get_access_original_row(i), txn->get_access_version(i)); + } +} + +dta_set_ent::dta_set_ent() { + set_size = 0; + txn = NULL; + rows = NULL; + next = NULL; +} + +void Dta::init() { + sem_init(&_semaphore, 0, 1); + sem_init(&sem_rwset_, 0, 1); +} + +void Dta::finish(RC rc, TxnManager* txn) { + std::list rset, wset; + ::get_rw_set(txn, rset, wset); + wset.sort(); + sem_wait(&sem_rwset_); + auto it = rwset_it_[txn]; + sem_post(&sem_rwset_); + + it->latch(); + if (rc == RCOK) { + assert(it->wset.size() == wset.size()); + it->wset = std::move(wset); + it->lower = txn->get_commit_timestamp(); + it->upper = it->lower + 1; + it->state = DTA_COMMITTED; + } else { + it->state = DTA_ABORTED; + } + it->release(); +} +RC Dta::validate(TxnManager* txn) { + RC rc = RCOK; +#if CC_ALG == DTA + uint64_t start_time = get_sys_clock(); + uint64_t timespan; + sem_wait(&_semaphore); + + timespan = get_sys_clock() - start_time; + txn->txn_stats.cc_block_time += timespan; + txn->txn_stats.cc_block_time_short += timespan; + INC_STATS(txn->get_thd_id(), dta_cs_wait_time, timespan); + start_time = get_sys_clock(); + uint64_t lower = dta_time_table.get_lower(txn->get_thd_id(), txn->get_txn_id()); + uint64_t upper = dta_time_table.get_upper(txn->get_thd_id(), txn->get_txn_id()); + DEBUG("DTA Validate Start %ld: [%lu,%lu]\n", txn->get_txn_id(), lower, upper); + + dta_set_ent* wset; + dta_set_ent* rset; + get_rw_set(txn, rset, wset); + + DEBUG("DTA write set size %ld: %u \n", txn->get_txn_id(), wset->set_size); + if (!wset->set_size) { + goto VALID_END; + } + + // lower bound of txn greater than write timestamp + if (lower <= txn->greatest_write_timestamp) { + lower = txn->greatest_write_timestamp + 1; + INC_STATS(txn->get_thd_id(), dta_case1_cnt, 1); + } + + // Examine each element in the write set + for (UInt32 i = 0; i < wset->set_size; i++) { + // 1. get the max read timestamp, and just the lower + row_t* cur_wrow = wset->rows[i]; + if (lower <= cur_wrow->manager->timestamp_last_read) { + lower = cur_wrow->manager->timestamp_last_read + 1; + } + if (lower >= upper) goto VALID_END; + + // 2. write in the key's write xid + if (cur_wrow->manager->write_trans) { + if (cur_wrow->manager->write_trans != txn->get_txn_id()) { + dta_time_table.set_state(txn->get_thd_id(), txn->get_txn_id(), DTA_ABORTED); + rc = Abort; + goto FINISH; + } + } else { + cur_wrow->manager->write_trans = txn->get_txn_id(); + } + + // 3. find the key's read xids, adjust their lower and upper + std::set* readxid_list = cur_wrow->manager->uncommitted_reads; + + for (auto it = readxid_list->begin(); it != readxid_list->end(); it++) { + if (lower >= upper) goto VALID_END; + + uint64_t txn_id = *it; + DEBUG(" UR %ld -- %ld: %ld\n", txn->get_txn_id(), cur_wrow->get_primary_key(), txn_id); + if (txn_id == txn->get_txn_id()) continue; + uint64_t it_upper = dta_time_table.get_upper(txn->get_thd_id(), *it); + uint64_t it_lower = dta_time_table.get_lower(txn->get_thd_id(), *it); + if (it_lower >= it_upper) continue; + DTAState state = dta_time_table.get_state(txn->get_thd_id(), *it); + if (state == DTA_VALIDATED || state == DTA_COMMITTED) { + INC_STATS(txn->get_thd_id(), dta_case4_cnt, 1); + if (lower < it_upper) { + lower = it_upper; + } + } else if (state == DTA_RUNNING) { + if (lower <= it_lower) { + // TRANS_LOG_WARN("DTAvalidation set lower = dta_txn->lower + 1, transid:%lu + // running_txn_id:%lu lower:%lu upper:%lu running_txn_id.lower:%lu + // running_txn_id.upper:%lu", ctx, part_ctx->GetTransID(), lower, upper, dta_txn->lower, + // dta_txn->upper); + lower = it_lower + 1; + it_upper = lower < it_upper ? lower : it_upper; + dta_time_table.set_upper(txn->get_thd_id(), *it, it_upper); + } else if (lower < it_upper) { + // TRANS_LOG_WARN("DTAvalidation set running_txn.upper < ctx.lower, transid:%lu + // running_txn_id:%lu lower:%lu upper:%lu running_txn_id.lower:%lu + // running_txn_id.upper:%lu", ctx, part_ctx->GetTransID(), lower, upper, dta_txn->lower, + // dta_txn->upper); + it_upper = lower; + dta_time_table.set_upper(txn->get_thd_id(), *it, it_upper); + } + } + } + } + +VALID_END: + if (lower >= upper) { + dta_time_table.set_state(txn->get_thd_id(), txn->get_txn_id(), DTA_ABORTED); + rc = Abort; + } else { + dta_time_table.set_state(txn->get_thd_id(), txn->get_txn_id(), DTA_VALIDATED); + rc = RCOK; + assert(lower < upper); + INC_STATS(txn->get_thd_id(), dta_range, upper - lower); + INC_STATS(txn->get_thd_id(), dta_commit_cnt, 1); + } + + dta_time_table.set_lower(txn->get_thd_id(), txn->get_txn_id(), lower); + dta_time_table.set_upper(txn->get_thd_id(), txn->get_txn_id(), upper); + +FINISH: + INC_STATS(txn->get_thd_id(), dta_validate_cnt, 1); + timespan = get_sys_clock() - start_time; + INC_STATS(txn->get_thd_id(), dta_validate_time, timespan); + txn->txn_stats.cc_time += timespan; + txn->txn_stats.cc_time_short += timespan; + + DEBUG("DTA Validate End %ld: %d [%lu,%lu]\n", txn->get_txn_id(), rc == RCOK, lower, upper); + sem_post(&_semaphore); +#endif + return rc; +} + +RC Dta::get_rw_set(TxnManager* txn, dta_set_ent*& rset, dta_set_ent*& wset) { + wset = (dta_set_ent*)mem_allocator.alloc(sizeof(dta_set_ent)); + rset = (dta_set_ent*)mem_allocator.alloc(sizeof(dta_set_ent)); + wset->set_size = txn->get_write_set_size(); + rset->set_size = txn->get_read_set_size(); + wset->rows = (row_t**)mem_allocator.alloc(sizeof(row_t*) * wset->set_size); + rset->rows = (row_t**)mem_allocator.alloc(sizeof(row_t*) * rset->set_size); + wset->txn = txn; + rset->txn = txn; + + UInt32 n = 0, m = 0; + for (uint64_t i = 0; i < wset->set_size + rset->set_size; i++) { + if (txn->get_access_type(i) == WR) + wset->rows[n++] = txn->get_access_original_row(i); + else + rset->rows[m++] = txn->get_access_original_row(i); + } + + assert(n == wset->set_size); + assert(m == rset->set_size); + + return RCOK; +} + +RC Dta::find_bound(TxnManager* txn) { + RC rc = RCOK; +#if CC_ALG == DTA + uint64_t lower = dta_time_table.get_lower(txn->get_thd_id(), txn->get_txn_id()); + uint64_t upper = dta_time_table.get_upper(txn->get_thd_id(), txn->get_txn_id()); + if (lower >= upper) { + dta_time_table.set_state(txn->get_thd_id(), txn->get_txn_id(), DTA_VALIDATED); + rc = Abort; + } else { + dta_time_table.set_state(txn->get_thd_id(), txn->get_txn_id(), DTA_COMMITTED); + // TODO: can commit_time be selected in a smarter way? + txn->commit_timestamp = lower; + // dta_time_table.set_upper(txn->get_thd_id(),txn->get_txn_id(),txn->commit_timestamp+1); + } + DEBUG("DTA Bound %ld: %d [%lu,%lu] %lu\n", txn->get_txn_id(), rc, lower, upper, + txn->commit_timestamp); +#endif + return rc; +} + +void DtaTimeTable::init() { + // table_size = g_inflight_max * g_node_cnt * 2 + 1; + table_size = g_inflight_max + 1; + DEBUG_M("DtaTimeTable::init table alloc\n"); + table = (DtaTimeTableNode*)mem_allocator.alloc(sizeof(DtaTimeTableNode) * table_size); + for (uint64_t i = 0; i < table_size; i++) { + table[i].init(); + } +} + +uint64_t DtaTimeTable::hash(uint64_t key) { return key % table_size; } + +DtaTimeTableEntry* DtaTimeTable::find(uint64_t key) { + DtaTimeTableEntry* entry = table[hash(key)].head; + while (entry) { + if (entry->key == key) break; + entry = entry->next; + } + return entry; +} + +void DtaTimeTable::init(uint64_t thd_id, uint64_t key, uint64_t ts) { + uint64_t idx = hash(key); + uint64_t mtx_wait_starttime = get_sys_clock(); + pthread_mutex_lock(&table[idx].mtx); + INC_STATS(thd_id, mtx[34], get_sys_clock() - mtx_wait_starttime); + DtaTimeTableEntry* entry = find(key); + if (!entry) { + DEBUG_M("DtaTimeTable::init entry alloc\n"); + entry = (DtaTimeTableEntry*)mem_allocator.alloc(sizeof(DtaTimeTableEntry)); + entry->init(key, ts); + LIST_PUT_TAIL(table[idx].head, table[idx].tail, entry); + } + pthread_mutex_unlock(&table[idx].mtx); +} + +void DtaTimeTable::release(uint64_t thd_id, uint64_t key) { + uint64_t idx = hash(key); + uint64_t mtx_wait_starttime = get_sys_clock(); + pthread_mutex_lock(&table[idx].mtx); + INC_STATS(thd_id, mtx[35], get_sys_clock() - mtx_wait_starttime); + DtaTimeTableEntry* entry = find(key); + if (entry) { + LIST_REMOVE_HT(entry, table[idx].head, table[idx].tail); + DEBUG_M("DtaTimeTable::release entry free\n"); + mem_allocator.free(entry, sizeof(DtaTimeTableEntry)); + } + pthread_mutex_unlock(&table[idx].mtx); +} + +uint64_t DtaTimeTable::get_lower(uint64_t thd_id, uint64_t key) { + uint64_t idx = hash(key); + uint64_t value = 0; + uint64_t mtx_wait_starttime = get_sys_clock(); + pthread_mutex_lock(&table[idx].mtx); + INC_STATS(thd_id, mtx[36], get_sys_clock() - mtx_wait_starttime); + DtaTimeTableEntry* entry = find(key); + if (entry) { + value = entry->lower; + } + pthread_mutex_unlock(&table[idx].mtx); + return value; +} + +uint64_t DtaTimeTable::get_upper(uint64_t thd_id, uint64_t key) { + uint64_t idx = hash(key); + uint64_t value = UINT64_MAX; + uint64_t mtx_wait_starttime = get_sys_clock(); + pthread_mutex_lock(&table[idx].mtx); + INC_STATS(thd_id, mtx[37], get_sys_clock() - mtx_wait_starttime); + DtaTimeTableEntry* entry = find(key); + if (entry) { + value = entry->upper; + } + pthread_mutex_unlock(&table[idx].mtx); + return value; +} + +void DtaTimeTable::set_lower(uint64_t thd_id, uint64_t key, uint64_t value) { + uint64_t idx = hash(key); + uint64_t mtx_wait_starttime = get_sys_clock(); + pthread_mutex_lock(&table[idx].mtx); + INC_STATS(thd_id, mtx[38], get_sys_clock() - mtx_wait_starttime); + DtaTimeTableEntry* entry = find(key); + if (entry) { + entry->lower = value; + } + pthread_mutex_unlock(&table[idx].mtx); +} + +void DtaTimeTable::set_upper(uint64_t thd_id, uint64_t key, uint64_t value) { + uint64_t idx = hash(key); + uint64_t mtx_wait_starttime = get_sys_clock(); + pthread_mutex_lock(&table[idx].mtx); + INC_STATS(thd_id, mtx[39], get_sys_clock() - mtx_wait_starttime); + DtaTimeTableEntry* entry = find(key); + if (entry) { + entry->upper = value; + } + pthread_mutex_unlock(&table[idx].mtx); +} + +DTAState DtaTimeTable::get_state(uint64_t thd_id, uint64_t key) { + uint64_t idx = hash(key); + DTAState state = DTA_ABORTED; + uint64_t mtx_wait_starttime = get_sys_clock(); + pthread_mutex_lock(&table[idx].mtx); + INC_STATS(thd_id, mtx[40], get_sys_clock() - mtx_wait_starttime); + DtaTimeTableEntry* entry = find(key); + if (entry) { + state = entry->state; + } + pthread_mutex_unlock(&table[idx].mtx); + return state; +} + +void DtaTimeTable::set_state(uint64_t thd_id, uint64_t key, DTAState value) { + uint64_t idx = hash(key); + uint64_t mtx_wait_starttime = get_sys_clock(); + pthread_mutex_lock(&table[idx].mtx); + INC_STATS(thd_id, mtx[41], get_sys_clock() - mtx_wait_starttime); + DtaTimeTableEntry* entry = find(key); + if (entry) { + entry->state = value; + } + pthread_mutex_unlock(&table[idx].mtx); +} diff --git a/contrib/deneva/concurrency_control/dta.h b/contrib/deneva/concurrency_control/dta.h new file mode 100644 index 00000000..1278b2bf --- /dev/null +++ b/contrib/deneva/concurrency_control/dta.h @@ -0,0 +1,139 @@ +/* + Copyright 2016 Massachusetts Institute of Technology + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +#ifndef _DTA_H_ +#define _DTA_H_ + +#include "../storage/row.h" +#include "semaphore.h" + +class TxnManager; + +enum DTAState { + DTA_RUNNING = 0, + DTA_VALIDATED, + DTA_COMMITTED, + DTA_ABORTED +}; +struct dta_item { + row_t* data; + uint64_t version; + dta_item(row_t* data, uint64_t version) : data(data), version(version) {} + bool operator<(const dta_item& rhs) { return data < rhs.data; } +}; +struct dta_rwset { + std::list rset; + std::list wset; + DTAState state; + pthread_mutex_t mtx; + ts_t commit_ts; + uint64_t lower, upper; + uint64_t thd_id; + dta_rwset(const std::list& rset, const std::list& wset, DTAState state, + const ts_t commit_ts, const uint64_t thd_id) + : rset(rset), + wset(wset), + state(state), + commit_ts(commit_ts), + lower(0), + upper(0), + thd_id(thd_id) { + pthread_mutex_init(&mtx, NULL); + } + + void latch() { pthread_mutex_lock(&mtx); } + void release() { pthread_mutex_unlock(&mtx); } +}; + +class dta_set_ent { + public: + dta_set_ent(); + UInt64 tn; + TxnManager* txn; + UInt32 set_size; + row_t** rows; //[MAX_WRITE_SET]; + dta_set_ent* next; +}; + +class Dta { + public: + void init(); + RC validate(TxnManager* txn); + RC find_bound(TxnManager* txn); + void finish(RC rc, TxnManager* txn); + + private: + RC get_rw_set(TxnManager* txni, dta_set_ent*& rset, dta_set_ent*& wset); + sem_t _semaphore; + + std::list rwset_; + std::map::iterator> rwset_it_; + sem_t sem_rwset_; +}; + +struct DtaTimeTableEntry { + uint64_t lower; + uint64_t upper; + uint64_t key; + DTAState state; + DtaTimeTableEntry* next; + DtaTimeTableEntry* prev; + void init(uint64_t key, uint64_t ts) { + lower = ts; + upper = UINT64_MAX; + this->key = key; + state = DTA_RUNNING; + next = NULL; + prev = NULL; + } +}; + +struct DtaTimeTableNode { + DtaTimeTableEntry* head; + DtaTimeTableEntry* tail; + pthread_mutex_t mtx; + void init() { + head = NULL; + tail = NULL; + pthread_mutex_init(&mtx, NULL); + } +}; + +class DtaTimeTable { + public: + void init(); + void init(uint64_t thd_id, uint64_t key, uint64_t ts); + void release(uint64_t thd_id, uint64_t key); + uint64_t get_lower(uint64_t thd_id, uint64_t key); + uint64_t get_upper(uint64_t thd_id, uint64_t key); + void set_lower(uint64_t thd_id, uint64_t key, uint64_t value); + void set_upper(uint64_t thd_id, uint64_t key, uint64_t value); + DTAState get_state(uint64_t thd_id, uint64_t key); + void set_state(uint64_t thd_id, uint64_t key, DTAState value); + + private: + // hash table + uint64_t hash(uint64_t key); + uint64_t table_size; + DtaTimeTableNode* table; + DtaTimeTableEntry* find(uint64_t key); + + DtaTimeTableEntry* find_entry(uint64_t id); + + sem_t _semaphore; +}; + +#endif diff --git a/contrib/deneva/concurrency_control/focc.cpp b/contrib/deneva/concurrency_control/focc.cpp index 5574b81b..d787730b 100644 --- a/contrib/deneva/concurrency_control/focc.cpp +++ b/contrib/deneva/concurrency_control/focc.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/focc.h b/contrib/deneva/concurrency_control/focc.h index 0ac130e1..f0aeb2cd 100644 --- a/contrib/deneva/concurrency_control/focc.h +++ b/contrib/deneva/concurrency_control/focc.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/occ_critical_section.cpp b/contrib/deneva/concurrency_control/occ_critical_section.cpp index 3403cc73..1b20ab79 100644 --- a/contrib/deneva/concurrency_control/occ_critical_section.cpp +++ b/contrib/deneva/concurrency_control/occ_critical_section.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/occ_critical_section.h b/contrib/deneva/concurrency_control/occ_critical_section.h index 70e4621f..ac4b77d2 100644 --- a/contrib/deneva/concurrency_control/occ_critical_section.h +++ b/contrib/deneva/concurrency_control/occ_critical_section.h @@ -1,17 +1,13 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * */ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/row_dta.cpp b/contrib/deneva/concurrency_control/row_dta.cpp new file mode 100644 index 00000000..928d908c --- /dev/null +++ b/contrib/deneva/concurrency_control/row_dta.cpp @@ -0,0 +1,508 @@ +/* + Copyright 2016 Massachusetts Institute of Technology + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +#include "row_dta.h" + +#include "dta.h" +#include "helper.h" +#include "manager.h" +#include "mem_alloc.h" +#include "txn.h" + +void Row_dta::init(row_t* row) { + _row = row; + max_version_ = 1; + timestamp_last_read = 0; + timestamp_last_write = 0; + dta_avail = true; + // uncommitted_writes = new std::set(); + uncommitted_reads = new std::set(); + write_trans = 0; + // assert(uncommitted_writes->begin() == uncommitted_writes->end()); + // assert(uncommitted_writes->size() == 0); + + // multi-version part + readreq_mvcc = NULL; + prereq_mvcc = NULL; + // readhis = NULL; + writehis = NULL; + // readhistail = NULL; + writehistail = NULL; + blatch = false; + latch = (pthread_mutex_t*)mem_allocator.alloc(sizeof(pthread_mutex_t)); + pthread_mutex_init(latch, NULL); + whis_len = 0; + // rhis_len = 0; + rreq_len = 0; + preq_len = 0; +} + +RC Row_dta::access(TsType type, TxnManager* txn, row_t* row, uint64_t& version) { + uint64_t starttime = get_sys_clock(); + RC rc = RCOK; + + rc = read_and_write(type, txn, row, version); + + uint64_t timespan = get_sys_clock() - starttime; + txn->txn_stats.cc_time += timespan; + txn->txn_stats.cc_time_short += timespan; + return rc; +} + +RC Row_dta::read_and_write(TsType type, TxnManager* txn, row_t* row, uint64_t& version) { + assert(CC_ALG == DTA); + RC rc = RCOK; + + uint64_t mtx_wait_starttime = get_sys_clock(); + while (!ATOM_CAS(dta_avail, true, false)) { + } + INC_STATS(txn->get_thd_id(), mtx[30], get_sys_clock() - mtx_wait_starttime); + + if (type == 0) { + DEBUG("READ %ld -- %ld: lw %ld\n", txn->get_txn_id(), _row->get_primary_key(), + timestamp_last_read); + } else if (type == 1) { + DEBUG("COMMIT-WRITE %ld -- %ld: lw %ld\n", txn->get_txn_id(), _row->get_primary_key(), + timestamp_last_read); + } else if (type == 2) { + DEBUG("WRITE (P_REQ) %ld -- %ld: lw %ld\n", txn->get_txn_id(), _row->get_primary_key(), + timestamp_last_read); + } else if (type == 3) { + DEBUG("XP-REQ %ld -- %ld: lw %ld\n", txn->get_txn_id(), _row->get_primary_key(), + timestamp_last_read); + } + + // Add to uncommitted reads (soft lock) + uncommitted_reads->insert(txn->get_txn_id()); + + // =======TODO: Do we need using write_txn to adjust it's upper? + + // Fetch the previous version + ts_t ts = txn->get_timestamp(); + + if (g_central_man) + glob_manager.lock_row(_row); + else + pthread_mutex_lock(latch); + + if (type == R_REQ) { + // figure out if ts is in interval(prewrite(x)) + // bool conf = conflict(type, ts); + bool conf = false; + if (conf && rreq_len < g_max_read_req) { + rc = WAIT; + // txn->wait_starttime = get_sys_clock(); + DEBUG("buf R_REQ %ld %ld\n", txn->get_txn_id(), _row->get_primary_key()); + buffer_req(R_REQ, txn); + txn->ts_ready = false; + } else { + // return results immediately. + rc = RCOK; + DTAMVHisEntry* whis = writehis; + while (whis_len && whis != NULL && whis->ts > ts) whis = whis->next; + row_t* ret = (whis == NULL) ? _row : whis->row; + version = (whis == NULL) ? 0 : whis->version; + txn->cur_row = ret; + // insert_history(ts, NULL); + assert(strstr(_row->get_table_name(), ret->get_table_name())); + } + + // Adjust txn.lower + // uint64_t lower = dta_time_table.get_lower(txn->get_thd_id(),txn->get_txn_id()); + // if (lower < timestamp_last_read) + // dta_time_table.set_lower(txn->get_thd_id(),txn->get_txn_id(), timestamp_last_read + 1); + + } else if (type == P_REQ) { + /*if ( conflict(type, ts) ) { + rc = Abort; + } else*/ + if (preq_len < g_max_pre_req) { + DEBUG("buf P_REQ %ld %ld\n", txn->get_txn_id(), _row->get_primary_key()); + buffer_req(P_REQ, txn); + rc = RCOK; + } else { + rc = Abort; + } + } else + assert(false); + + if (rc == RCOK) { + if (whis_len > g_his_recycle_len) { + // if (whis_len > g_his_recycle_len || rhis_len > g_his_recycle_len) { + ts_t t_th = glob_manager.get_min_ts(txn->get_thd_id()); + // if (readhistail && readhistail->ts < t_th) + // clear_history(R_REQ, t_th); + // Here is a tricky bug. The oldest transaction might be + // reading an even older version whose timestamp < t_th. + // But we cannot recycle that version because it is still being used. + // So the HACK here is to make sure that the first version older than + // t_th not be recycled. + if (whis_len > 1 && writehistail->prev->ts < t_th) { + row_t* latest_row = clear_history(W_REQ, t_th); + if (latest_row != NULL) { + assert(_row != latest_row); + _row->copy(latest_row); + } + } + } + } + + if (g_central_man) + glob_manager.release_row(_row); + else + pthread_mutex_unlock(latch); + + ATOM_CAS(dta_avail, false, true); + + return rc; +} + +RC Row_dta::prewrite(TxnManager* txn) { + assert(CC_ALG == DTA); + RC rc = RCOK; + + uint64_t mtx_wait_starttime = get_sys_clock(); + while (!ATOM_CAS(dta_avail, true, false)) { + } + INC_STATS(txn->get_thd_id(), mtx[31], get_sys_clock() - mtx_wait_starttime); + DEBUG("PREWRITE %ld -- %ld: lw %ld, lr %ld\n", txn->get_txn_id(), _row->get_primary_key(), + timestamp_last_write, timestamp_last_read); + + // // Copy uncommitted reads + // for(auto it = uncommitted_reads->begin(); it != uncommitted_reads->end(); it++) { + // uint64_t txn_id = *it; + // txn->uncommitted_reads->insert(txn_id); + // DEBUG(" UR %ld -- %ld: %ld\n",txn->get_txn_id(),_row->get_primary_key(),txn_id); + // } + + // // Copy uncommitted writes + // for(auto it = uncommitted_writes->begin(); it != uncommitted_writes->end(); it++) { + // uint64_t txn_id = *it; + // txn->uncommitted_writes_y->insert(txn_id); + // DEBUG(" UW %ld -- %ld: %ld\n",txn->get_txn_id(),_row->get_primary_key(),txn_id); + // } + + // // Copy read timestamp + // if(txn->greatest_read_timestamp < timestamp_last_read) + // txn->greatest_read_timestamp = timestamp_last_read; + + // // Copy write timestamp + // if(txn->greatest_write_timestamp < timestamp_last_write) + // txn->greatest_write_timestamp = timestamp_last_write; + + // Add to uncommitted writes (soft lock) + // uncommitted_writes->insert(txn->get_txn_id()); + + ATOM_CAS(dta_avail, false, true); + + return rc; +} + +RC Row_dta::abort(access_t type, TxnManager* txn) { + uint64_t mtx_wait_starttime = get_sys_clock(); + while (!ATOM_CAS(dta_avail, true, false)) { + } + INC_STATS(txn->get_thd_id(), mtx[32], get_sys_clock() - mtx_wait_starttime); + DEBUG("dta Abort %ld: %d -- %ld\n", txn->get_txn_id(), type, _row->get_primary_key()); + + uncommitted_reads->erase(txn->get_txn_id()); + + if (type == WR && write_trans == txn->get_txn_id()) { + write_trans = 0; + } + + if (type == XP) { + write_trans = 0; + DEBUG("debuf %ld %ld\n", txn->get_txn_id(), _row->get_primary_key()); + DTAMVReqEntry* req = debuffer_req(P_REQ, txn); + assert(req != NULL); + return_req_entry(req); + // update_buffer(txn); + } + + ATOM_CAS(dta_avail, false, true); + return Abort; +} + +RC Row_dta::commit(access_t type, TxnManager* txn, row_t* data, uint64_t& version) { + uint64_t mtx_wait_starttime = get_sys_clock(); + while (!ATOM_CAS(dta_avail, true, false)) { + } + INC_STATS(txn->get_thd_id(), mtx[33], get_sys_clock() - mtx_wait_starttime); + DEBUG("dta Commit %ld: %d,%lu -- %ld\n", txn->get_txn_id(), type, txn->get_commit_timestamp(), + _row->get_primary_key()); + + uint64_t txn_commit_ts = txn->get_commit_timestamp(); + + if (txn_commit_ts > timestamp_last_read) timestamp_last_read = txn_commit_ts; + uncommitted_reads->erase(txn->get_txn_id()); + + if (type == WR) { + // ts_t ts = txn->get_timestamp(); + + // the corresponding prewrite request is debuffered. + version = insert_history(txn_commit_ts, data); + DEBUG("dta insert histroy %ld: %lu -- %ld\n", txn->get_txn_id(), txn->get_commit_timestamp(), + data->get_primary_key()); + DEBUG("debuf %ld %ld\n", txn->get_txn_id(), _row->get_primary_key()); + DTAMVReqEntry* req = debuffer_req(P_REQ, txn); + assert(req != NULL); + return_req_entry(req); + // update_buffer(txn); + + write_trans = 0; + } + + ATOM_CAS(dta_avail, false, true); + return RCOK; +} + +row_t* Row_dta::clear_history(TsType type, ts_t ts) { + DTAMVHisEntry** queue; + DTAMVHisEntry** tail; + switch (type) { + // case R_REQ : queue = &readhis; tail = &readhistail; break; + case W_REQ: + queue = &writehis; + tail = &writehistail; + break; + default: + assert(false); + } + DTAMVHisEntry* his = *tail; + DTAMVHisEntry* prev = NULL; + row_t* row = NULL; + while (his && his->prev && his->prev->ts < ts) { + prev = his->prev; + assert(prev->ts >= his->ts); + if (row != NULL) { + row->free_row(); + mem_allocator.free(row, sizeof(row_t)); + } + row = his->row; + his->row = NULL; + return_his_entry(his); + his = prev; + // if (type == R_REQ) rhis_len --; + // else + whis_len--; + } + *tail = his; + if (*tail) (*tail)->next = NULL; + if (his == NULL) *queue = NULL; + return row; +} + +DTAMVReqEntry* Row_dta::get_req_entry() { + return (DTAMVReqEntry*)mem_allocator.alloc(sizeof(DTAMVReqEntry)); +} + +void Row_dta::return_req_entry(DTAMVReqEntry* entry) { + mem_allocator.free(entry, sizeof(DTAMVReqEntry)); +} + +DTAMVHisEntry* Row_dta::get_his_entry() { + return (DTAMVHisEntry*)mem_allocator.alloc(sizeof(DTAMVHisEntry)); +} + +void Row_dta::return_his_entry(DTAMVHisEntry* entry) { + if (entry->row != NULL) { + entry->row->free_row(); + mem_allocator.free(entry->row, sizeof(row_t)); + } + mem_allocator.free(entry, sizeof(DTAMVHisEntry)); +} + +void Row_dta::buffer_req(TsType type, TxnManager* txn) { + DTAMVReqEntry* req_entry = get_req_entry(); + assert(req_entry != NULL); + req_entry->txn = txn; + req_entry->ts = txn->get_timestamp(); + req_entry->starttime = get_sys_clock(); + if (type == R_REQ) { + rreq_len++; + STACK_PUSH(readreq_mvcc, req_entry); + } else if (type == P_REQ) { + preq_len++; + STACK_PUSH(prereq_mvcc, req_entry); + } +} + +// for type == R_REQ +// debuffer all non-conflicting requests +// for type == P_REQ +// debuffer the request with matching txn. +DTAMVReqEntry* Row_dta::debuffer_req(TsType type, TxnManager* txn) { + DTAMVReqEntry** queue; + DTAMVReqEntry* return_queue = NULL; + switch (type) { + case R_REQ: + queue = &readreq_mvcc; + break; + case P_REQ: + queue = &prereq_mvcc; + break; + default: + assert(false); + } + + DTAMVReqEntry* req = *queue; + DTAMVReqEntry* prev_req = NULL; + if (txn != NULL) { + assert(type == P_REQ); + while (req != NULL && req->txn != txn) { + prev_req = req; + req = req->next; + } + assert(req != NULL); + if (prev_req != NULL) + prev_req->next = req->next; + else { + assert(req == *queue); + *queue = req->next; + } + preq_len--; + req->next = return_queue; + return_queue = req; + } else { + assert(type == R_REQ); + // should return all non-conflicting read requests + // The following code makes the assumption that each write op + // must read the row first. i.e., there is no write-only operation. + uint64_t min_pts = UINT64_MAX; + // uint64_t min_pts = (1UL << 32); + for (DTAMVReqEntry* preq = prereq_mvcc; preq != NULL; preq = preq->next) + if (preq->ts < min_pts) min_pts = preq->ts; + while (req != NULL) { + if (req->ts <= min_pts) { + if (prev_req == NULL) { + assert(req == *queue); + *queue = (*queue)->next; + } else + prev_req->next = req->next; + rreq_len--; + req->next = return_queue; + return_queue = req; + req = (prev_req == NULL) ? *queue : prev_req->next; + } else { + prev_req = req; + req = req->next; + } + } + } + + return return_queue; +} + +uint64_t Row_dta::insert_history(ts_t ts, row_t* row) { + DTAMVHisEntry* new_entry = get_his_entry(); + new_entry->version = max_version_; + ++max_version_; + new_entry->ts = ts; + new_entry->row = row; + if (row != NULL) whis_len++; + // else rhis_len ++; + // DTAMVHisEntry ** queue = (row == NULL)? + // &(readhis) : &(writehis); + // DTAMVHisEntry ** tail = (row == NULL)? + // &(readhistail) : &(writehistail); + + DTAMVHisEntry** queue = &(writehis); + DTAMVHisEntry** tail = &(writehistail); + DTAMVHisEntry* his = *queue; + while (his != NULL && ts < his->ts) { + his = his->next; + } + + if (his) { + LIST_INSERT_BEFORE(his, new_entry, (*queue)); + // if (his == *queue) + // *queue = new_entry; + } else + LIST_PUT_TAIL((*queue), (*tail), new_entry); + return new_entry->version; +} + +bool Row_dta::conflict(TsType type, ts_t ts) { + // find the unique prewrite-read couple (prewrite before read) + // if no such couple found, no conflict. + // else + // if exists writehis between them, NO conflict!!!! + // else, CONFLICT!!! + ts_t rts; + ts_t pts; + if (type == R_REQ) { + rts = ts; + pts = 0; + DTAMVReqEntry* req = prereq_mvcc; + while (req != NULL) { + if (req->ts < ts && req->ts > pts) { + pts = req->ts; + } + req = req->next; + } + if (pts == 0) // no such couple exists + return false; + } else if (type == P_REQ) { + rts = 0; + pts = ts; + // DTAMVHisEntry * his = readhis; + // while (his != NULL) { + // if (his->ts > ts) { + // rts = his->ts; + // } else + // break; + // his = his->next; + // } + if (rts == 0) // no couple exists + return false; + assert(rts > pts); + } + DTAMVHisEntry* whis = writehis; + while (whis != NULL && whis->ts > pts) { + if (whis->ts < rts) return false; + whis = whis->next; + } + return true; +} + +void Row_dta::update_buffer(TxnManager* txn) { + DTAMVReqEntry* ready_read = debuffer_req(R_REQ, NULL); + DTAMVReqEntry* req = ready_read; + DTAMVReqEntry* tofree = NULL; + + while (req != NULL) { + // find the version for the request + DTAMVHisEntry* whis = writehis; + while (whis != NULL && whis->ts > req->ts) whis = whis->next; + row_t* row = (whis == NULL) ? _row : whis->row; + req->txn->cur_row = row; + // insert_history(req->ts, NULL); + assert(row->get_data() != NULL); + assert(row->get_table() != NULL); + assert(row->get_schema() == _row->get_schema()); + + req->txn->ts_ready = true; + uint64_t timespan = get_sys_clock() - req->starttime; + req->txn->txn_stats.cc_block_time += timespan; + req->txn->txn_stats.cc_block_time_short += timespan; + txn_table.restart_txn(txn->get_thd_id(), req->txn->get_txn_id(), 0); + tofree = req; + req = req->next; + // free ready_read + return_req_entry(tofree); + } +} diff --git a/contrib/deneva/concurrency_control/row_dta.h b/contrib/deneva/concurrency_control/row_dta.h new file mode 100644 index 00000000..798e605b --- /dev/null +++ b/contrib/deneva/concurrency_control/row_dta.h @@ -0,0 +1,94 @@ +/* + Copyright 2016 Massachusetts Institute of Technology + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +#ifndef ROW_DTA_H +#define ROW_DTA_H +#include "../storage/row.h" +class table_t; +class Catalog; +class TxnManager; + +struct DTAMVReqEntry { + TxnManager* txn; + ts_t ts; + ts_t starttime; + DTAMVReqEntry* next; +}; + +struct DTAMVHisEntry { + ts_t ts; + // only for write history. The value needs to be stored. + // char * data; + row_t* row; + uint64_t version; + DTAMVHisEntry* next; + DTAMVHisEntry* prev; +}; + +class Row_dta { + public: + void init(row_t* row); + RC access(TsType type, TxnManager* txn, row_t* row, uint64_t& version); + RC read_and_write(TsType type, TxnManager* txn, row_t* row, uint64_t& version); + RC prewrite(TxnManager* txn); + RC abort(access_t type, TxnManager* txn); + RC commit(access_t type, TxnManager* txn, row_t* data, uint64_t& version); + void write(row_t* data); + + private: + volatile bool dta_avail; + uint64_t max_version_; + row_t* _row; + + public: + std::set* uncommitted_reads; + // std::set * uncommitted_writes; + + uint64_t write_trans; + uint64_t timestamp_last_read; + uint64_t timestamp_last_write; + + // multi-verison part + private: + pthread_mutex_t* latch; + bool blatch; + + DTAMVReqEntry* get_req_entry(); + void return_req_entry(DTAMVReqEntry* entry); + DTAMVHisEntry* get_his_entry(); + void return_his_entry(DTAMVHisEntry* entry); + + bool conflict(TsType type, ts_t ts); + void buffer_req(TsType type, TxnManager* txn); + DTAMVReqEntry* debuffer_req(TsType type, TxnManager* txn = NULL); + void update_buffer(TxnManager* txn); + uint64_t insert_history(ts_t ts, row_t* row); + + row_t* clear_history(TsType type, ts_t ts); + + DTAMVReqEntry* readreq_mvcc; + DTAMVReqEntry* prereq_mvcc; + // DTAMVHisEntry * readhis; + DTAMVHisEntry* writehis; + // DTAMVHisEntry * readhistail; + DTAMVHisEntry* writehistail; + uint64_t whis_len; + // uint64_t rhis_len; + uint64_t rreq_len; + uint64_t preq_len; +}; + +#endif diff --git a/contrib/deneva/concurrency_control/row_maat.cpp b/contrib/deneva/concurrency_control/row_maat.cpp index 178553ef..6cf2782a 100644 --- a/contrib/deneva/concurrency_control/row_maat.cpp +++ b/contrib/deneva/concurrency_control/row_maat.cpp @@ -40,7 +40,7 @@ RC Row_maat::access(access_t type, TxnManager * txn) { #if WORKLOAD == TPCC read_and_prewrite(txn); #else - if (type == RD) read(txn); + if (type == RD || type == SCAN) read(txn); if (type == WR) prewrite(txn); #endif uint64_t timespan = get_sys_clock() - starttime; @@ -179,7 +179,7 @@ RC Row_maat::abort(access_t type, TxnManager * txn) { uncommitted_reads->erase(txn->get_txn_id()); uncommitted_writes->erase(txn->get_txn_id()); #else - if(type == RD) { + if(type == RD || type == SCAN) { uncommitted_reads->erase(txn->get_txn_id()); } @@ -250,14 +250,14 @@ RC Row_maat::commit(access_t type, TxnManager * txn, row_t * data) { #else uint64_t txn_commit_ts = txn->get_commit_timestamp(); - if(type == RD) { + if (type == RD || type == SCAN) { if (txn_commit_ts > timestamp_last_read) timestamp_last_read = txn_commit_ts; uncommitted_reads->erase(txn->get_txn_id()); // Forward validation // Check uncommitted writes against this txn's - for(auto it = uncommitted_writes->begin(); it != uncommitted_writes->end();it++) { - if(txn->uncommitted_writes->count(*it) == 0) { + for (auto it = uncommitted_writes->begin(); it != uncommitted_writes->end();it++) { + if (txn->uncommitted_writes->count(*it) == 0) { // apply timestamps // these write txns need to come AFTER this txn uint64_t it_lower = time_table.get_lower(txn->get_thd_id(),*it); diff --git a/contrib/deneva/concurrency_control/row_mvcc.cpp b/contrib/deneva/concurrency_control/row_mvcc.cpp index c35cfeef..591cbe3f 100644 --- a/contrib/deneva/concurrency_control/row_mvcc.cpp +++ b/contrib/deneva/concurrency_control/row_mvcc.cpp @@ -366,12 +366,10 @@ void Row_mvcc::update_buffer(TxnManager * txn) { assert(row->get_schema() == _row->get_schema()); req->txn->ts_ready = true; - #if WORKLOAD!=DA //DA do not need restart uint64_t timespan = get_sys_clock() - req->starttime; req->txn->txn_stats.cc_block_time += timespan; req->txn->txn_stats.cc_block_time_short += timespan; txn_table.restart_txn(txn->get_thd_id(),req->txn->get_txn_id(),0); - #endif tofree = req; req = req->next; // free ready_read diff --git a/contrib/deneva/concurrency_control/row_null.cpp b/contrib/deneva/concurrency_control/row_null.cpp index 0e4e075d..3b058ecf 100644 --- a/contrib/deneva/concurrency_control/row_null.cpp +++ b/contrib/deneva/concurrency_control/row_null.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/row_null.h b/contrib/deneva/concurrency_control/row_null.h index 115657aa..cd601605 100644 --- a/contrib/deneva/concurrency_control/row_null.h +++ b/contrib/deneva/concurrency_control/row_null.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/row_si.cpp b/contrib/deneva/concurrency_control/row_si.cpp new file mode 100644 index 00000000..9f77cd8d --- /dev/null +++ b/contrib/deneva/concurrency_control/row_si.cpp @@ -0,0 +1,53 @@ +#include "row_si.h" + +#include "storage/row.h" +#include "system/manager.h" +#include "system/mem_alloc.h" +#include "system/txn.h" + +inline bool TupleSatisfiesMVCC(const SIEntry& tuple, const ts_t start_ts) { + return tuple.commit_ts < start_ts; +} +void Row_si::init(row_t* row) { + row_ = row; + versions_ = new TSNode(row_, 0, 0); + latch_ = (pthread_mutex_t*)mem_allocator.alloc(sizeof(pthread_mutex_t)); + pthread_mutex_init(latch_, NULL); + w_trans = 0; + timestamp_last_write = 0; +} +RC Row_si::access(TxnManager* txn, TsType type, uint64_t& version) { + uint64_t start_time = get_sys_clock(); + ts_t ts = txn->get_start_timestamp(); + if (type == R_REQ) { + txn->cur_row = row_; + version = 0; + for (TSNode* v = versions_.load(); v != nullptr; v = v->next_) { + if (v->commit_ts < ts) { + txn->cur_row = v->row; + version = v->commit_ts; + //std::lock_guard l(v->mutex); + //v->r_trans_ts.push_back(ts); + break; + } + } + } else { + assert(false); + } + uint64_t timespan = get_sys_clock() - start_time; + txn->txn_stats.cc_time += timespan; + txn->txn_stats.cc_time_short += timespan; + return RCOK; +} +void Row_si::latch() { pthread_mutex_lock(latch_); } +void Row_si::release() { pthread_mutex_unlock(latch_); } + +uint64_t Row_si::write(row_t* data, TxnManager* txn, access_t type) { + uint64_t res = 0; + if (type == WR) { + res = TSNode::push_front(versions_, data, txn->get_start_timestamp(), txn->get_commit_timestamp())->commit_ts; + timestamp_last_write = txn->get_commit_timestamp(); + } + if (w_trans == txn->get_start_timestamp()) w_trans = 0; + return res; +} diff --git a/contrib/deneva/concurrency_control/row_si.h b/contrib/deneva/concurrency_control/row_si.h new file mode 100644 index 00000000..54ddd0a0 --- /dev/null +++ b/contrib/deneva/concurrency_control/row_si.h @@ -0,0 +1,56 @@ +#ifndef ROW_SI_H +#define ROW_SI_H +#include +#include +#include +#include + +#include "storage/row.h" +#include "system/global.h" + +class TxnManager; +class row_t; + +struct SIEntry { + SIEntry(row_t* const row, const ts_t w_ts, const ts_t commit_ts) + : row(row), w_ts(w_ts), commit_ts(commit_ts) {} + SIEntry(const SIEntry&) = delete; + SIEntry(SIEntry&&) = default; + row_t* row; + ts_t w_ts; + ts_t commit_ts; + std::mutex mutex; +}; + +template +class TSNode : public T { + public: + using T::T; + + template + static TSNode* push_front(std::atomic*>& head, Args&&... args) { + TSNode* new_head = new TSNode(std::forward(args)...); + new_head->next_ = head.load(std::memory_order_relaxed); + while (!head.compare_exchange_weak(new_head->next_, new_head, std::memory_order_release, std::memory_order_relaxed)); + return new_head; + } + + TSNode* next_ = nullptr; +}; + +class Row_si { + public: + void init(row_t* row); + RC access(TxnManager* txn, TsType type, uint64_t& version); + uint64_t write(row_t* data, TxnManager* txn, access_t type); + void latch(); + void release(); + std::atomic w_trans; + std::atomic timestamp_last_write; + + private: + pthread_mutex_t* latch_; + row_t* row_; + std::atomic*> versions_; +}; +#endif diff --git a/contrib/deneva/concurrency_control/row_silo.cpp b/contrib/deneva/concurrency_control/row_silo.cpp index f1ac6e26..b10f0b54 100644 --- a/contrib/deneva/concurrency_control/row_silo.cpp +++ b/contrib/deneva/concurrency_control/row_silo.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: zhanhaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/row_silo.h b/contrib/deneva/concurrency_control/row_silo.h index 4dfeb5be..7e6e13d3 100644 --- a/contrib/deneva/concurrency_control/row_silo.h +++ b/contrib/deneva/concurrency_control/row_silo.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: zhanhaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/row_ssi.cpp b/contrib/deneva/concurrency_control/row_ssi.cpp index de71c42d..1344fdff 100644 --- a/contrib/deneva/concurrency_control/row_ssi.cpp +++ b/contrib/deneva/concurrency_control/row_ssi.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@tencent.com * @@ -30,7 +28,7 @@ void Row_ssi::init(row_t * row) { pthread_mutex_init(latch, NULL); whis_len = 0; rhis_len = 0; - + commit_lock = 0; preq_len = 0; } diff --git a/contrib/deneva/concurrency_control/row_ssi.h b/contrib/deneva/concurrency_control/row_ssi.h index b8a64c99..39e50908 100644 --- a/contrib/deneva/concurrency_control/row_ssi.h +++ b/contrib/deneva/concurrency_control/row_ssi.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@tencent.com * diff --git a/contrib/deneva/concurrency_control/row_sundial.cpp b/contrib/deneva/concurrency_control/row_sundial.cpp index ba9f1935..fdee0f61 100644 --- a/contrib/deneva/concurrency_control/row_sundial.cpp +++ b/contrib/deneva/concurrency_control/row_sundial.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * @@ -86,7 +84,7 @@ Row_sundial::access(access_t type, TxnManager * txn, row_t *& row, uint64_t &wts, uint64_t &rts) { uint64_t starttime = get_sys_clock(); RC rc = RCOK; - if(type == RD || !OCC_WAW_LOCK) + if(type == RD || type == SCAN || !OCC_WAW_LOCK) rc = read(txn, wts, rts, true); else if(type == WR) rc = write(txn, wts, rts); diff --git a/contrib/deneva/concurrency_control/row_sundial.h b/contrib/deneva/concurrency_control/row_sundial.h index eb02f3b3..2c88caa2 100644 --- a/contrib/deneva/concurrency_control/row_sundial.h +++ b/contrib/deneva/concurrency_control/row_sundial.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/row_ts.cpp b/contrib/deneva/concurrency_control/row_ts.cpp index dc350df0..08ef9505 100644 --- a/contrib/deneva/concurrency_control/row_ts.cpp +++ b/contrib/deneva/concurrency_control/row_ts.cpp @@ -280,17 +280,17 @@ void Row_ts::update_buffer(uint64_t thd_id) { // for each debuffered readreq, perform read. TsReqEntry * req = ready_read; while (req != NULL) { - req->txn->cur_row->copy(_row); - if (rts < req->ts) { - rts = req->ts; - } - req->txn->ts_ready = true; + if (req->txn->txn && req->ts == req->txn->get_timestamp()) { + req->txn->cur_row->copy(_row); + if (rts < req->ts) { + rts = req->ts; + } + req->txn->ts_ready = true; uint64_t timespan = get_sys_clock() - req->starttime; req->txn->txn_stats.cc_block_time += timespan; req->txn->txn_stats.cc_block_time_short += timespan; - #if WORKLOAD!=DA - txn_table.restart_txn(thd_id,req->txn->get_txn_id(),0); - #endif + txn_table.restart_txn(thd_id,req->txn->get_txn_id(),0); + } req = req->next; } // return all the req_entry back to freelist diff --git a/contrib/deneva/concurrency_control/row_unified.h b/contrib/deneva/concurrency_control/row_unified.h new file mode 100644 index 00000000..da52e419 --- /dev/null +++ b/contrib/deneva/concurrency_control/row_unified.h @@ -0,0 +1,80 @@ +/* Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ + +#ifndef _ROW_UNIFIED_H_ +#define _ROW_UNIFIED_H_ + +#include "../system/global.h" +#include "../config.h" +#include "unified_util.h" + +#if IS_GENERIC_ALG + +class row_t; + +template +class Row_unified +{ + public: + void init(row_t* orig_row); + RC read(TxnManager& txn); + RC prewrite(TxnManager& txn); + void write(row_t* ver_row, TxnManager& txn); + void revoke(row_t* ver_row, TxnManager& txn); + + private: + row_t* orig_row_; + std::unique_ptr> row_man_; +}; + +// prevent cycle include +#include "../system/txn.h" +#include "../storage/row.h" + +template +void Row_unified::init(row_t* orig_row) { + orig_row_ = orig_row; + row_man_ = std::make_unique>(orig_row->get_row_id(), orig_row); +} + +template +RC Row_unified::read(TxnManager& txn) { + assert(row_man_ != nullptr); + txn.cur_row = mem_allocator.construct(); + txn.cur_row->init(orig_row_->get_table(), orig_row_->get_part_id()); + const std::optional read_ver_row = row_man_->Read(*txn.uni_txn_man_); + if (read_ver_row.has_value()) { + txn.cur_row->copy(*read_ver_row); + return RCOK; + } else { + return Abort; + } +} + +// TODO: Here is a memory leak. We alloc the row_t outside algorithm but not free it because only algorithm +// knows when should row_t free. +template +RC Row_unified::prewrite(TxnManager& txn) { + assert(row_man_ != nullptr); + txn.cur_row = mem_allocator.construct(); + txn.cur_row->init(orig_row_->get_table(), orig_row_->get_part_id()); + return row_man_->Prewrite(txn.cur_row, *txn.uni_txn_man_) ? RCOK : Abort; +} + +template +void Row_unified::write(row_t* ver_row, TxnManager& txn) { + row_man_->Write(ver_row, *txn.uni_txn_man_); +} + +template +void Row_unified::revoke(row_t* ver_row, TxnManager& txn) { + row_man_->Revoke(ver_row, *txn.uni_txn_man_); +} + +#endif +#endif diff --git a/contrib/deneva/concurrency_control/row_wsi.cpp b/contrib/deneva/concurrency_control/row_wsi.cpp index 4aa00b96..42457f89 100644 --- a/contrib/deneva/concurrency_control/row_wsi.cpp +++ b/contrib/deneva/concurrency_control/row_wsi.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/row_wsi.h b/contrib/deneva/concurrency_control/row_wsi.h index 464077fd..360021a8 100644 --- a/contrib/deneva/concurrency_control/row_wsi.h +++ b/contrib/deneva/concurrency_control/row_wsi.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/silo.cpp b/contrib/deneva/concurrency_control/silo.cpp index ba2dbb94..6c5ad026 100644 --- a/contrib/deneva/concurrency_control/silo.cpp +++ b/contrib/deneva/concurrency_control/silo.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: zhanhaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/ssi.cpp b/contrib/deneva/concurrency_control/ssi.cpp index fba16f7e..f8a62daf 100644 --- a/contrib/deneva/concurrency_control/ssi.cpp +++ b/contrib/deneva/concurrency_control/ssi.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@tencent.com * diff --git a/contrib/deneva/concurrency_control/ssi.h b/contrib/deneva/concurrency_control/ssi.h index 7c2f56f0..9e8da5bc 100644 --- a/contrib/deneva/concurrency_control/ssi.h +++ b/contrib/deneva/concurrency_control/ssi.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@tencent.com * diff --git a/contrib/deneva/concurrency_control/sundial.cpp b/contrib/deneva/concurrency_control/sundial.cpp index 3189ef96..baf19802 100644 --- a/contrib/deneva/concurrency_control/sundial.cpp +++ b/contrib/deneva/concurrency_control/sundial.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/sundial.h b/contrib/deneva/concurrency_control/sundial.h index dcc2377d..ea1698c5 100644 --- a/contrib/deneva/concurrency_control/sundial.h +++ b/contrib/deneva/concurrency_control/sundial.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/unified_util.h b/contrib/deneva/concurrency_control/unified_util.h new file mode 100644 index 00000000..a9f97fcb --- /dev/null +++ b/contrib/deneva/concurrency_control/unified_util.h @@ -0,0 +1,28 @@ +/* Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ + +#ifndef _ROW_UNIFIED_UTIL_H_ +#define _ROW_UNIFIED_UTIL_H_ + +#include "../unified_concurrency_control/row_prece.h" +#include "../unified_concurrency_control/alg_dli_identify_cycle.h" +#include "../unified_concurrency_control/alg_dli_identify_chain.h" +#include "../unified_concurrency_control/txn_dli_identify.h" + +class row_t; + +template constexpr const ttts::UniAlgs uni_alg; + +template <> constexpr const ttts::UniAlgs uni_alg = ttts::UniAlgs::UNI_DLI_IDENTIFY_CYCLE; +template <> constexpr const ttts::UniAlgs uni_alg = ttts::UniAlgs::UNI_DLI_IDENTIFY_CHAIN; + +template using UniRowManager = ttts::RowManager, row_t*>; +template using UniTxnManager = ttts::TxnManager, row_t*>; +template using UniAlgManager = ttts::AlgManager, row_t*>; + +#endif diff --git a/contrib/deneva/concurrency_control/wsi.cpp b/contrib/deneva/concurrency_control/wsi.cpp index 6d5f92b0..6629b7da 100644 --- a/contrib/deneva/concurrency_control/wsi.cpp +++ b/contrib/deneva/concurrency_control/wsi.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/concurrency_control/wsi.h b/contrib/deneva/concurrency_control/wsi.h index a236545c..6947eae4 100644 --- a/contrib/deneva/concurrency_control/wsi.h +++ b/contrib/deneva/concurrency_control/wsi.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/config.h b/contrib/deneva/config.h index 33b19e36..e94fd130 100644 --- a/contrib/deneva/config.h +++ b/contrib/deneva/config.h @@ -28,7 +28,7 @@ #define CREATOR_USE_T false //TraversalActionSequenceCreator -#define TRANS_CNT 2 +#define TRANS_CNT 4 #define ITEM_CNT 4 #define SUBTASK_NUM 1 #define SUBTASK_ID 0 @@ -50,14 +50,18 @@ // #define NO_REMOTE // remove all remote txn #define TXN_QUEUE_PERCENT 0.9 // The proportion of the transaction to take from txn_queue firstly. // ! end of these parameters +// + +// print detail log for each operation execution +#define DA_PRINT_LOG false /***********************************************/ // Simulation + Hardware /***********************************************/ -#define NODE_CNT 2 -#define THREAD_CNT 4 //trans_num -#define REM_THREAD_CNT 2 -#define SEND_THREAD_CNT 2 +#define NODE_CNT 1 +#define THREAD_CNT 1 //trans_num +#define REM_THREAD_CNT 1 +#define SEND_THREAD_CNT 1 #define CORE_CNT 2 // PART_CNT should be at least NODE_CNT #define PART_CNT NODE_CNT @@ -88,7 +92,7 @@ // # of transactions to run for warmup #define WARMUP 0 // YCSB or TPCC or PPS or DA -#define WORKLOAD YCSB +#define WORKLOAD TPCC // print the transaction latency distribution #define PRT_LAT_DISTR false #define STATS_ENABLE true @@ -148,7 +152,7 @@ // WAIT_DIE, NO_WAIT, TIMESTAMP, MVCC, CALVIN, MAAT, SUNDIAL, SILO, BOCC, FOCC, SSI, WSI #define ISOLATION_LEVEL SERIALIZABLE -#define CC_ALG BOCC +#define CC_ALG DLI_IDENTIFY_CHAIN #define YCSB_ABORT_MODE false #define QUEUE_CAPACITY_NEW 1000000 // all transactions acquire tuples according to the primary key order. @@ -383,8 +387,16 @@ enum PPSTxnType { #define BOCC 16 #define SSI 17 #define WSI 18 + +#define DTA 26 + #define SILO 27 #define CNULL 28 +#define DLI_IDENTIFY_CYCLE 29 +#define DLI_IDENTIFY_CHAIN 30 + +#define IS_GENERIC_ALG (CC_ALG == DLI_IDENTIFY_CYCLE || CC_ALG == DLI_IDENTIFY_CHAIN) + // TIMESTAMP allocation method. #define TS_MUTEX 1 #define TS_CAS 2 @@ -393,7 +405,7 @@ enum PPSTxnType { #define LTS_CURL_CLOCK 5 #define LTS_TCP_CLOCK 6 -#define LTS_TCP_IP "10.77.110.147" +#define LTS_TCP_IP "xx.xx.xx.xx" // wirte the IP #define LTS_TCP_PORT 62389 // MODES // NORMAL < NOCC < QRY_ONLY < SETUP < SIMPLE diff --git a/contrib/deneva/statistics/stats.cpp b/contrib/deneva/statistics/stats.cpp index 3ffa56fd..d56c301e 100644 --- a/contrib/deneva/statistics/stats.cpp +++ b/contrib/deneva/statistics/stats.cpp @@ -234,6 +234,12 @@ void Stats_thd::clear() { sched_epoch_cnt=0; sched_epoch_diff=0; + // DLI_MVCC_OCC + dli_mvcc_occ_validate_time = 0; + dli_mvcc_occ_check_cnt = 0; + dli_mvcc_occ_abort_check_cnt = 0; + dli_mvcc_occ_ts_abort_cnt = 0; + //OCC occ_validate_time=0; occ_cs_wait_time=0; @@ -1451,7 +1457,7 @@ uint64_t Stats::get_txn_cnts() { for (uint64_t tid = 0; tid < limit; tid ++) { total_txn_cnt += _stats[tid]->txn_cnt; } - + return total_txn_cnt; } diff --git a/contrib/deneva/statistics/stats.h b/contrib/deneva/statistics/stats.h index 1e4d4c3e..52adcc59 100644 --- a/contrib/deneva/statistics/stats.h +++ b/contrib/deneva/statistics/stats.h @@ -210,6 +210,12 @@ class Stats_thd { uint64_t sched_epoch_cnt; double sched_epoch_diff; + // DLI_MVCC_OCC + double dli_mvcc_occ_validate_time; + uint64_t dli_mvcc_occ_check_cnt; + uint64_t dli_mvcc_occ_abort_check_cnt; + uint64_t dli_mvcc_occ_ts_abort_cnt; + // OCC double occ_validate_time; double occ_cs_wait_time; diff --git a/contrib/deneva/storage/row.cpp b/contrib/deneva/storage/row.cpp index b1b60b7a..521d594f 100644 --- a/contrib/deneva/storage/row.cpp +++ b/contrib/deneva/storage/row.cpp @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn @@ -41,6 +39,9 @@ #include "row_wsi.h" #include "row_null.h" #include "row_silo.h" +#include "row_si.h" +#include "row_dta.h" +#include "row_unified.h" #include "mem_alloc.h" #include "manager.h" @@ -91,6 +92,9 @@ void row_t::init_manager(row_t * row) { manager = (Row_null *) mem_allocator.align_alloc(sizeof(Row_null)); #elif CC_ALG == SILO manager = (Row_silo *) mem_allocator.align_alloc(sizeof(Row_silo)); +#elif IS_GENERIC_ALG + void* const p = mem_allocator.align_alloc(sizeof(Row_unified)); + manager = new(p) Row_unified(); #endif #if CC_ALG != HSTORE && CC_ALG != HSTORE_SPEC @@ -208,14 +212,14 @@ RC row_t::get_lock(access_t type, TxnManager * txn) { } RC row_t::get_row(access_t type, TxnManager *txn, Access *access) { - RC rc = RCOK; -#if MODE==NOCC_MODE || MODE==QRY_ONLY_MODE + RC rc = RCOK; +#if MODE == NOCC_MODE || MODE == QRY_ONLY_MODE access->data = this; - return rc; + return rc; #endif #if ISOLATION_LEVEL == NOLOCK access->data = this; - return rc; + return rc; #endif #if CC_ALG == CNULL @@ -230,11 +234,11 @@ RC row_t::get_row(access_t type, TxnManager *txn, Access *access) { #endif #if CC_ALG == MAAT - DEBUG_M("row_t::get_row MAAT alloc \n"); + DEBUG_M("row_t::get_row MAAT alloc \n"); txn->cur_row = (row_t *) mem_allocator.alloc(sizeof(row_t)); txn->cur_row->init(get_table(), get_part_id()); - rc = this->manager->access(type,txn); - txn->cur_row->copy(this); + rc = this->manager->access(type,txn); + txn->cur_row->copy(this); access->data = txn->cur_row; assert(rc == RCOK); @@ -290,7 +294,7 @@ RC row_t::get_row(access_t type, TxnManager *txn, Access *access) { assert(access->data->get_table_name() != NULL); } } - if (rc != Abort && (CC_ALG == MVCC || CC_ALG == SSI || CC_ALG == WSI) && type == WR) { + if (rc != Abort && (CC_ALG == MVCC || CC_ALG == SSI || CC_ALG == WSI) && type == WR) { DEBUG_M("row_t::get_row MVCC alloc \n"); row_t * newr = (row_t *) mem_allocator.alloc(sizeof(row_t)); newr->init(this->get_table(), get_part_id()); @@ -298,6 +302,22 @@ RC row_t::get_row(access_t type, TxnManager *txn, Access *access) { access->data = newr; } goto end; +#elif IS_GENERIC_ALG + if (type == WR) { + rc = this->manager->prewrite(*txn); + } else if (type == RD || type == SCAN) { + rc = this->manager->read(*txn); + } else { + assert(false); + } + if (rc == RCOK) { + access->data = txn->cur_row; + assert(access->data->get_data() != NULL); + assert(access->data->get_table() != NULL); + assert(access->data->get_schema() == this->get_schema()); + assert(access->data->get_table_name() != NULL); + } + goto end; #elif CC_ALG == OCC || CC_ALG == FOCC || CC_ALG == BOCC // OCC always make a local copy regardless of read or write DEBUG_M("row_t::get_row OCC alloc \n"); @@ -311,13 +331,13 @@ RC row_t::get_row(access_t type, TxnManager *txn, Access *access) { DEBUG_M("row_t::get_row SILO alloc \n"); txn->cur_row = (row_t *) mem_allocator.alloc(sizeof(row_t)); txn->cur_row->init(get_table(), get_part_id()); - TsType ts_type = (type == RD)? R_REQ : P_REQ; + TsType ts_type = (type == RD || type == SCAN)? R_REQ : P_REQ; rc = this->manager->access(txn, ts_type, txn->cur_row); access->data = txn->cur_row; goto end; #elif CC_ALG == HSTORE || CC_ALG == HSTORE_SPEC || CC_ALG == CALVIN #if CC_ALG == HSTORE_SPEC - if(txn_table.spec_mode) { + if (txn_table.spec_mode) { DEBUG_M("row_t::get_row HSTORE_SPEC alloc \n"); txn->cur_row = (row_t *) mem_allocator.alloc(sizeof(row_t)); txn->cur_row->init(get_table(), get_part_id()); @@ -371,7 +391,7 @@ RC row_t::get_row_post_wait(access_t type, TxnManager * txn, row_t *& row) { //ts_t endtime = get_sys_clock(); row = this; -#elif CC_ALG == MVCC || CC_ALG == TIMESTAMP || CC_ALG == SUNDIAL || CC_ALG == SSI || CC_ALG == WSI +#elif CC_ALG == MVCC || CC_ALG == TIMESTAMP || CC_ALG == DTA assert(txn->ts_ready); //INC_STATS(thd_id, time_wait, t2 - t1); row = txn->cur_row; @@ -453,7 +473,7 @@ uint64_t row_t::return_row(RC rc, access_t type, TxnManager *txn, row_t *row) { } row->free_row(); - DEBUG_M("row_t::return_row Maat free \n"); + DEBUG_M("roversionreturn_row Maat free \n"); mem_allocator.free(row, sizeof(row_t)); return 0; #elif CC_ALG == MAAT @@ -495,7 +515,25 @@ uint64_t row_t::return_row(RC rc, access_t type, TxnManager *txn, row_t *row) { DEBUG_M("row_t::return_row XP free \n"); mem_allocator.free(row, sizeof(row_t)); return 0; +#elif IS_GENERIC_ALG + assert (row != NULL); + if (type == RD || type == SCAN) { + // We have copy the ver_row when get_row with R_REQ to prevent ver_row released. + // To prevent memory leak, we need release the duplication here. + row->free_row(); + DEBUG_M("row_t::return_row GENERIC_ALG %d free \n", CC_ALG); + mem_allocator.free(row, sizeof(row_t)); + } else if (type == XP) { + // ver_row should be released in row manager + manager->revoke(row, *txn); + } else if (type == WR) { + assert(row != NULL); + assert(row->get_schema() == this->get_schema()); + manager->write(row, *txn); + } + return 0; #else assert(false); #endif } + diff --git a/contrib/deneva/storage/row.h b/contrib/deneva/storage/row.h index b2e3f936..ceb00d37 100644 --- a/contrib/deneva/storage/row.h +++ b/contrib/deneva/storage/row.h @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn @@ -26,6 +24,8 @@ #define _ROW_H_ #include + +#include "../concurrency_control/row_si.h" #include "global.h" @@ -58,6 +58,7 @@ class Row_sundial; class Row_si; class Row_null; class Row_silo; +template class Row_unified; class row_t { public: @@ -136,6 +137,8 @@ class row_t { Row_null * manager; #elif CC_ALG == SILO Row_silo * manager; + #elif IS_GENERIC_ALG + Row_unified *manager; #endif char * data; int tuple_size; diff --git a/contrib/deneva/system/client_thread.cpp b/contrib/deneva/system/client_thread.cpp index 93c15ea4..10271b78 100644 --- a/contrib/deneva/system/client_thread.cpp +++ b/contrib/deneva/system/client_thread.cpp @@ -75,11 +75,16 @@ RC ClientThread::run() { continue; #endif m_query = client_query_queue.get_next_query(next_node,_thd_id); + if (!m_query) { + break; // there are no histories anymore + } if(last_send_time > 0) { INC_STATS(get_thd_id(),cl_send_intv,get_sys_clock() - last_send_time); } last_send_time = get_sys_clock(); + #if WORKLOAD == DA simulation->last_da_query_time = get_sys_clock(); + #endif #elif LOAD_METHOD == LOAD_RATE if ((inf_cnt = client_man.inc_inflight(next_node)) < 0) continue; diff --git a/contrib/deneva/system/global.cpp b/contrib/deneva/system/global.cpp index 55d8daf9..0861114a 100644 --- a/contrib/deneva/system/global.cpp +++ b/contrib/deneva/system/global.cpp @@ -26,6 +26,7 @@ #include "focc.h" #include "ssi.h" #include "wsi.h" +#include "dta.h" #include "transport.h" #include "work_queue.h" #include "abort_queue.h" @@ -69,6 +70,10 @@ Maat maat_man; ssi ssi_man; wsi wsi_man; Sundial sundial_man; +Dta dta_man; +#if IS_GENERIC_ALG +UniAlgManager uni_alg_man; +#endif Transport tport_man; TxnManPool txn_man_pool; TxnPool txn_pool; @@ -85,6 +90,7 @@ Client_txn client_man; Sequencer seq_man; Logger logger; TimeTable time_table; +DtaTimeTable dta_time_table; InOutTable inout_table; // QTcpQueue tcp_queue; TcpTimestamp tcp_ts; @@ -95,10 +101,6 @@ map da_start_stamp_tab; set da_start_trans_tab; map da_stamp_tab; set already_abort_tab; -string DA_history_mem; -bool abort_history; -ofstream commit_file; -ofstream abort_file; bool volatile warmup_done = false; bool volatile enable_thread_mem_pool = false; @@ -241,3 +243,4 @@ UInt32 g_repl_type = REPL_TYPE; UInt32 g_repl_cnt = REPLICA_CNT; map g_params; + diff --git a/contrib/deneva/system/global.h b/contrib/deneva/system/global.h index 6bffd819..2eadf079 100644 --- a/contrib/deneva/system/global.h +++ b/contrib/deneva/system/global.h @@ -51,6 +51,7 @@ #include #include "da_block_queue.h" //#include "maat.h" +#include "../concurrency_control/unified_util.h" using namespace std; @@ -66,6 +67,7 @@ class ssi; class wsi; class Maat; class Sundial; +class Dta; class Transport; class Remote_query; class TxnManPool; @@ -84,6 +86,7 @@ class Client_txn; class Sequencer; class Logger; class TimeTable; +class DtaTimeTable; class InOutTable; class DAQuery; @@ -92,6 +95,9 @@ class KeyXidCache; class RtsCache; // class QTcpQueue; class TcpTimestamp; +class Message; + +class Path; typedef uint32_t UInt32; typedef int32_t SInt32; @@ -116,6 +122,10 @@ extern ssi ssi_man; extern wsi wsi_man; extern Maat maat_man; extern Sundial sundial_man; +extern Dta dta_man; +#if IS_GENERIC_ALG +extern UniAlgManager uni_alg_man; +#endif extern Transport tport_man; extern TxnManPool txn_man_pool; extern TxnPool txn_pool; @@ -132,6 +142,7 @@ extern Client_txn client_man; extern Sequencer seq_man; extern Logger logger; extern TimeTable time_table; +extern DtaTimeTable dta_time_table; extern InOutTable inout_table; // extern QTcpQueue tcp_queue; @@ -261,10 +272,6 @@ extern map da_start_stamp_tab; extern set da_start_trans_tab; extern map da_stamp_tab; extern set already_abort_tab; -extern string DA_history_mem; -extern bool abort_history; -extern ofstream commit_file; -extern ofstream abort_file; // CALVIN extern UInt32 g_seq_thread_cnt; @@ -339,7 +346,7 @@ enum access_t {RD, WR, XP, SCAN}; /* LOCK */ enum lock_t {LOCK_EX = 0, LOCK_SH, LOCK_NONE, LOCK_COM}; /* TIMESTAMP */ -enum TsType {R_REQ = 0, W_REQ, P_REQ, XP_REQ}; +enum TsType {R_REQ = 0, W_REQ, P_REQ, XP_REQ, S_REQ}; /*DA query build queue*/ //queue query_build_queue; diff --git a/contrib/deneva/system/hash.cpp b/contrib/deneva/system/hash.cpp index c5ec045c..05138b47 100644 --- a/contrib/deneva/system/hash.cpp +++ b/contrib/deneva/system/hash.cpp @@ -5,8 +5,8 @@ #include "hash.h" -int hash_any(register const unsigned char *k, register int keylen) { - register uint32_t a, b, c, len; +int hash_any(const unsigned char *k, int keylen) { + uint32_t a, b, c, len; /* Set up the internal state */ len = keylen; @@ -15,7 +15,7 @@ int hash_any(register const unsigned char *k, register int keylen) { /* If the source pointer is word-aligned, we use word-wide fetches */ if (((uintptr_t)k & UINT32_ALIGN_MASK) == 0) { /* Code path for aligned source data */ - register const uint32_t *ka = (const uint32_t *)k; + const uint32_t *ka = (const uint32_t *)k; /* handle most of the key */ while (len >= 12) { diff --git a/contrib/deneva/system/hash.h b/contrib/deneva/system/hash.h index 2c330ff7..7ee44205 100644 --- a/contrib/deneva/system/hash.h +++ b/contrib/deneva/system/hash.h @@ -51,6 +51,6 @@ c -= rot(b, 24); \ } -int hash_any(register const unsigned char *k, register int keylen); +int hash_any(const unsigned char *k, int keylen); -#endif \ No newline at end of file +#endif diff --git a/contrib/deneva/system/http.cpp b/contrib/deneva/system/http.cpp index ca8229bc..85927ba8 100644 --- a/contrib/deneva/system/http.cpp +++ b/contrib/deneva/system/http.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/system/http.h b/contrib/deneva/system/http.h index d770a43f..5e5d68ac 100644 --- a/contrib/deneva/system/http.h +++ b/contrib/deneva/system/http.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/system/io_thread.cpp b/contrib/deneva/system/io_thread.cpp index 8a839a4e..30edcc63 100644 --- a/contrib/deneva/system/io_thread.cpp +++ b/contrib/deneva/system/io_thread.cpp @@ -33,6 +33,8 @@ #include "txn.h" #include "ycsb.h" +static std::atomic g_one_io_thread_exit(false); + void InputThread::setup() { std::vector * msgs; @@ -86,7 +88,6 @@ RC InputThread::run() { } else { server_recv_loop(); } - return FINISH; } @@ -135,7 +136,7 @@ RC InputThread::client_recv_loop() { } - printf("FINISH %ld:%ld\n",_node_id,_thd_id); + printf("InputThread Client FINISH %ld:%ld\n",_node_id,_thd_id); fflush(stdout); return FINISH; } @@ -186,7 +187,11 @@ RC InputThread::server_recv_loop() { uint64_t starttime; std::vector * msgs; - while (!simulation->is_done()) { + while (!simulation->is_done() +#if WORKLOAD == DA + && !simulation->da_server_iothread_timeout() +#endif + ) { heartbeat(); starttime = get_sys_clock(); @@ -219,13 +224,17 @@ RC InputThread::server_recv_loop() { if (fakeprocess(msg)) #endif work_queue.enqueue(get_thd_id(),msg,false); +#if WORKLOAD == DA + simulation->last_da_recv_query_time = get_sys_clock(); + simulation->da_has_recved_query = true; +#endif msgs->erase(msgs->begin()); } delete msgs; INC_STATS(_thd_id,mtx[29], get_sys_clock() - starttime); } - printf("FINISH %ld:%ld\n",_node_id,_thd_id); + printf("InputThread Server FINISH %ld:%ld\n",_node_id,_thd_id); fflush(stdout); return FINISH; } @@ -244,12 +253,15 @@ RC OutputThread::run() { tsetup(); printf("Running OutputThread %ld\n",_thd_id); - while (!simulation->is_done()) { + while (!simulation->is_done() +#if WORKLOAD == DA + && !simulation->da_server_iothread_timeout() +#endif + ) { heartbeat(); messager->run(); } - - printf("FINISH %ld:%ld\n",_node_id,_thd_id); + printf("OutputThread FINISH %ld:%ld\n", _node_id, _thd_id); fflush(stdout); return FINISH; } diff --git a/contrib/deneva/system/libtcpforcpp.cpp b/contrib/deneva/system/libtcpforcpp.cpp index f87fb71e..c93d567e 100644 --- a/contrib/deneva/system/libtcpforcpp.cpp +++ b/contrib/deneva/system/libtcpforcpp.cpp @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/system/libtcpforcpp.h b/contrib/deneva/system/libtcpforcpp.h index 1fce5df3..6312a145 100644 --- a/contrib/deneva/system/libtcpforcpp.h +++ b/contrib/deneva/system/libtcpforcpp.h @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/system/ltsrpc.proto b/contrib/deneva/system/ltsrpc.proto index 2af7a904..82e0baaf 100644 --- a/contrib/deneva/system/ltsrpc.proto +++ b/contrib/deneva/system/ltsrpc.proto @@ -1,8 +1,6 @@ /* Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: hongyaozhao@ruc.edu.cn * diff --git a/contrib/deneva/system/main.cpp b/contrib/deneva/system/main.cpp index 4e987004..71f7d9e8 100644 --- a/contrib/deneva/system/main.cpp +++ b/contrib/deneva/system/main.cpp @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn @@ -52,6 +50,7 @@ #include "wsi.h" #include "focc.h" #include "bocc.h" +#include "dta.h" #include "client_query.h" #include "sundial.h" #include "http.h" @@ -305,6 +304,17 @@ int main(int argc, char *argv[]) { printf("Done\n"); #endif +#if CC_ALG == DTA + printf("Initializing DTA Time Table... "); + fflush(stdout); + dta_time_table.init(); + printf("Done\n"); + printf("Initializing DTA manager... "); + fflush(stdout); + dta_man.init(); + printf("Done\n"); +#endif + endtime = get_server_clock(); printf("Initialization Time = %ld\n", endtime - starttime); fflush(stdout); @@ -318,7 +328,10 @@ int main(int argc, char *argv[]) { // spawn and run txns again. starttime = get_server_clock(); simulation->run_starttime = starttime; +#if WORKLOAD == DA simulation->last_da_query_time = starttime; + simulation->last_da_recv_query_time = starttime; +#endif uint64_t id = 0; for (uint64_t i = 0; i < wthd_cnt; i++) { @@ -348,7 +361,7 @@ int main(int argc, char *argv[]) { pthread_create(&p_thds[id++], NULL, run_thread, (void *)&log_thds[0]); #endif -#if CC_ALG != CALVIN +#if CC_ALG != CALVIN && WORKLOAD != DA abort_thds[0].init(id,g_node_id,m_wl); pthread_create(&p_thds[id++], NULL, run_thread, (void *)&abort_thds[0]); #endif diff --git a/contrib/deneva/system/manager.cpp b/contrib/deneva/system/manager.cpp index ea2b09a1..41e48a7a 100644 --- a/contrib/deneva/system/manager.cpp +++ b/contrib/deneva/system/manager.cpp @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn diff --git a/contrib/deneva/system/mem_alloc.cpp b/contrib/deneva/system/mem_alloc.cpp index 9e3de003..4acfa102 100644 --- a/contrib/deneva/system/mem_alloc.cpp +++ b/contrib/deneva/system/mem_alloc.cpp @@ -33,6 +33,17 @@ void mem_alloc::free(void * ptr, uint64_t size) { #endif } +void mem_alloc::free(void * ptr) { + if (NO_FREE) { + } + DEBUG_M("free 0x%lx\n", (uint64_t)ptr); +#ifdef N_MALLOC + std::free(ptr); +#else + je_free(ptr); +#endif +} + void * mem_alloc::alloc(uint64_t size) { void * ptr; diff --git a/contrib/deneva/system/mem_alloc.h b/contrib/deneva/system/mem_alloc.h index 8f1e6334..b7d8b92b 100644 --- a/contrib/deneva/system/mem_alloc.h +++ b/contrib/deneva/system/mem_alloc.h @@ -18,6 +18,7 @@ #define _MEM_ALLOC_H_ #include "global.h" +#include class mem_alloc { public: @@ -25,6 +26,15 @@ class mem_alloc { void * align_alloc(uint64_t size); void * realloc(void * ptr, uint64_t size); void free(void * block, uint64_t size); + void free(void * block); + template T* construct(Args&&... args) { + void* ptr = alloc(sizeof(T)); + return new(ptr) T(std::forward(args)...); + } + template std::shared_ptr make_shared(Args&&... args) { + return std::shared_ptr(construct(std::forward(args)...), + [this](T* const ptr) { free(ptr, sizeof(T)); }); + } }; #endif diff --git a/contrib/deneva/system/parser.cpp b/contrib/deneva/system/parser.cpp index e7833e0a..b1142ba4 100644 --- a/contrib/deneva/system/parser.cpp +++ b/contrib/deneva/system/parser.cpp @@ -179,6 +179,13 @@ void parser(int argc, char * argv[]) { assert(false); } } + +#if WORKLOAD == DA + g_thread_cnt = 1; // worker thread + g_rem_thread_cnt = 1; // input thread + g_send_thread_cnt = 1; // output thread + g_abort_thread_cnt = 0; // abort thread +#endif g_total_thread_cnt = g_thread_cnt + g_rem_thread_cnt + g_send_thread_cnt + g_abort_thread_cnt + 1; #if LOGGING g_total_thread_cnt += g_logger_thread_cnt; // logger thread diff --git a/contrib/deneva/system/sim_manager.cpp b/contrib/deneva/system/sim_manager.cpp index 7464329e..3eb17fcd 100644 --- a/contrib/deneva/system/sim_manager.cpp +++ b/contrib/deneva/system/sim_manager.cpp @@ -33,7 +33,11 @@ void SimManager::init() { #if TIME_ENABLE run_starttime = get_sys_clock(); +#if WORKLOAD == DA last_da_query_time = get_sys_clock(); + last_da_recv_query_time = get_sys_clock(); + da_has_recved_query = false; +#endif #else run_starttime = get_wall_clock(); #endif @@ -44,27 +48,36 @@ void SimManager::init() { void SimManager::set_starttime(uint64_t starttime) { if(ATOM_CAS(start_set, false, true)) { run_starttime = starttime; +#if WORKLOAD == DA last_da_query_time = starttime; + last_da_recv_query_time = starttime; +#endif last_worker_epoch_time = starttime; sim_done = false; printf("Starttime set to %ld\n",run_starttime); } } + +#if WORKLOAD == DA +bool SimManager::da_timeout_(const uint64_t last_time) { + uint64_t now = get_sys_clock(); + if (now < last_time) + { + now = last_time; + } + bool res = ((get_sys_clock() - run_starttime) >= (g_done_timer + g_warmup_timer) / 12) && + ((now - last_time) >= (g_done_timer + g_warmup_timer) / 6); + if (res) { + printf("SimManager::da_timeout_ success\n"); + } + return res; +} +#endif + bool SimManager::timeout() { #if TIME_ENABLE #if WORKLOAD == DA - uint64_t t=last_da_query_time; - uint64_t now=get_sys_clock(); - if(now= (g_done_timer + g_warmup_timer)/12) - &&((now - t) >= (g_done_timer + g_warmup_timer)/6); - if (res) { - printf("123\n"); - } - return res; + return da_timeout_(last_da_query_time); #else return (get_sys_clock() - run_starttime) >= g_done_timer + g_warmup_timer; #endif @@ -73,6 +86,12 @@ bool SimManager::timeout() { #endif } +#if WORKLOAD == DA +bool SimManager::da_server_iothread_timeout() { + return da_has_recved_query && da_timeout_(last_da_recv_query_time); +} +#endif + bool SimManager::is_done() { bool done = sim_done || timeout(); if(done && !sim_done) { diff --git a/contrib/deneva/system/sim_manager.h b/contrib/deneva/system/sim_manager.h index 2789dbf9..55fbae92 100644 --- a/contrib/deneva/system/sim_manager.h +++ b/contrib/deneva/system/sim_manager.h @@ -35,7 +35,6 @@ class SimManager { int64_t epoch_txn_cnt; uint64_t txn_cnt; uint64_t inflight_cnt; - uint64_t last_da_query_time; void init(); bool is_setup_done(); @@ -56,6 +55,17 @@ class SimManager { void inc_epoch_txn_cnt(); void decr_epoch_txn_cnt(); double seconds_from_start(uint64_t time); + +#if WORKLOAD == DA + std::atomic last_da_query_time; // last time client generated a history or server ran a history + std::atomic last_da_recv_query_time; // last time server received a history + bool da_has_recved_query; + bool da_server_iothread_timeout(); + +private: + bool da_timeout_(const uint64_t last_time); +#endif + }; #endif diff --git a/contrib/deneva/system/txn.cpp b/contrib/deneva/system/txn.cpp index bc976fc7..6db1befc 100644 --- a/contrib/deneva/system/txn.cpp +++ b/contrib/deneva/system/txn.cpp @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn @@ -34,6 +32,7 @@ #include "focc.h" #include "bocc.h" #include "row_occ.h" +#include "dta.h" #include "table.h" #include "catalog.h" #include "index_btree.h" @@ -55,6 +54,11 @@ #include "ssi.h" #include "wsi.h" #include "manager.h" +#include "../unified_concurrency_control/txn_dli_identify.h" + +#if WORKLOAD == DA +extern std::optional g_da_cycle; +#endif void TxnStats::init() { starttime=0; @@ -293,8 +297,7 @@ void Transaction::release_accesses(uint64_t thd_id) { void Transaction::release_inserts(uint64_t thd_id) { for(uint64_t i = 0; i < insert_rows.size(); i++) { row_t * row = insert_rows[i]; -#if CC_ALG != MAAT && CC_ALG != OCC && \ - CC_ALG != SUNDIAL && CC_ALG != BOCC && CC_ALG != FOCC +#if CC_ALG != MAAT && CC_ALG != OCC && CC_ALG != SUNDIAL && CC_ALG != BOCC && CC_ALG != FOCC DEBUG_M("TxnManager::cleanup row->manager free\n"); mem_allocator.free(row->manager, 0); #endif @@ -430,11 +433,12 @@ void TxnManager::release() { #if CC_ALG == CALVIN calvin_locked_rows.release(); -#endif -#if CC_ALG == SILO +#elif CC_ALG == SILO num_locks = 0; memset(write_set, 0, 100); // mem_allocator.free(write_set, sizeof(int) * 100); +#elif IS_GENERIC_ALG + uni_txn_man_ = nullptr; #endif txn_ready = true; } @@ -499,7 +503,9 @@ RC TxnManager::abort() { //assert(time_table.get_state(get_txn_id()) == MAAT_ABORTED); time_table.release(get_thd_id(),get_txn_id()); #endif - +#if CC_ALG == DTA + dta_time_table.release(get_thd_id(), get_txn_id()); +#endif uint64_t timespan = get_sys_clock() - txn_stats.restart_starttime; if (IS_LOCAL(get_txn_id()) && warmup_done) { INC_STATS_ARR(get_thd_id(),start_abort_commit_latency, timespan); @@ -727,9 +733,14 @@ void TxnManager::register_thread(Thread * h_thd) { #endif } -void TxnManager::set_txn_id(txnid_t txn_id) { txn->txn_id = txn_id; } +void TxnManager::set_txn_id(txnid_t txn_id) { + txn->txn_id = txn_id; +#if IS_GENERIC_ALG + uni_txn_man_ = std::make_unique>(txn_id); +#endif +} -txnid_t TxnManager::get_txn_id() { return txn->txn_id; } +txnid_t TxnManager::get_txn_id() const { return txn->txn_id; } Workload *TxnManager::get_wl() { return h_wl; } @@ -834,6 +845,7 @@ void TxnManager::cleanup_row(RC rc, uint64_t rid) { } } #endif + #endif if (type == WR) txn->accesses[rid]->version = version; #if CC_ALG == SUNDIAL @@ -862,6 +874,22 @@ void TxnManager::cleanup(RC rc) { #if (CC_ALG == WSI) && MODE == NORMAL_MODE wsi_man.finish(rc,this); #endif +#if IS_GENERIC_ALG && MODE == NORMAL_MODE + if (rc == RCOK) { + uni_alg_man.Commit(*uni_txn_man_); + } else { + assert(rc == Abort); + uni_alg_man.Abort(*uni_txn_man_); + } +#endif + + +#if WORKLOAD == DA && (CC_ALG == DLI_IDENTIFY || CC_ALG == DLI_IDENTIFY_2) + if (uni_txn_man_->cycle_ != nullptr && !g_da_cycle.has_value()) { + g_da_cycle.emplace(*uni_txn_man_->cycle_); + } +#endif + ts_t starttime = get_sys_clock(); uint64_t row_cnt = txn->accesses.get_count(); assert(txn->accesses.get_count() == txn->row_cnt); @@ -881,6 +909,10 @@ void TxnManager::cleanup(RC rc) { } #endif +#if CC_ALG == DTA + dta_man.finish(rc, this); +#endif + if (rc == Abort) { txn->release_inserts(get_thd_id()); txn->insert_rows.clear(); @@ -926,14 +958,14 @@ RC TxnManager::get_row(row_t * row, access_t type, row_t *& row_rtn) { if (!access) { access_pool.get(get_thd_id(),access); rc = row->get_row(type, this, access->data, access->orig_wts, access->orig_rts); - if (!OCC_WAW_LOCK || type == RD) { + if (!OCC_WAW_LOCK || type == RD || type == SCAN) { _min_commit_ts = _min_commit_ts > access->orig_wts ? _min_commit_ts : access->orig_wts; } else { if (rc == WAIT) ATOM_ADD_FETCH(_num_lock_waits, 1); if (rc == Abort || rc == WAIT) return rc; } } - if (!OCC_WAW_LOCK || type == RD) { + if (!OCC_WAW_LOCK || type == RD || type == SCAN) { access->locked = false; } else { _min_commit_ts = _min_commit_ts > access->orig_rts + 1 ? _min_commit_ts : access->orig_rts + 1; @@ -1083,32 +1115,42 @@ RC TxnManager::validate() { #if MODE != NORMAL_MODE return RCOK; #endif - if (CC_ALG != OCC && CC_ALG != MAAT && - CC_ALG != SUNDIAL && CC_ALG != BOCC && CC_ALG != FOCC && CC_ALG != WSI && - CC_ALG != SSI && CC_ALG != SILO) { - return RCOK; - } RC rc = RCOK; uint64_t starttime = get_sys_clock(); - if (CC_ALG == OCC && rc == RCOK) rc = occ_man.validate(this); - if(CC_ALG == BOCC && rc == RCOK) rc = bocc_man.validate(this); - if(CC_ALG == FOCC && rc == RCOK) rc = focc_man.validate(this); - if(CC_ALG == SSI && rc == RCOK) rc = ssi_man.validate(this); - if(CC_ALG == WSI && rc == RCOK) rc = wsi_man.validate(this); - if(CC_ALG == MAAT && rc == RCOK) { + if (CC_ALG == OCC) { + rc = occ_man.validate(this); + } else if (CC_ALG == BOCC) { + rc = bocc_man.validate(this); + } else if (CC_ALG == FOCC) { + rc = focc_man.validate(this); + } else if (CC_ALG == SSI) { + rc = ssi_man.validate(this); + } else if (CC_ALG == WSI) { + rc = wsi_man.validate(this); + } else if (CC_ALG == MAAT) { rc = maat_man.validate(this); // Note: home node must be last to validate if(IS_LOCAL(get_txn_id()) && rc == RCOK) { rc = maat_man.find_bound(this); } - } - if(CC_ALG == SUNDIAL && rc == RCOK) { + } else if (CC_ALG == SUNDIAL) { rc = sundial_man.validate(this); + } else if (CC_ALG == DTA) { + rc = dta_man.validate(this); + // Note: home node must be last to validate + if (IS_LOCAL(get_txn_id()) && rc == RCOK) { + rc = dta_man.find_bound(this); + } } +#if IS_GENERIC_ALG + else if (IS_GENERIC_ALG) { + rc = uni_alg_man.Validate(*this->uni_txn_man_) ? RCOK : Abort; + } +#endif #if CC_ALG == SILO - if(CC_ALG == SILO && rc == RCOK) { + else if (CC_ALG == SILO) { rc = validate_silo(); - if(IS_LOCAL(get_txn_id()) && rc == RCOK) { + if (IS_LOCAL(get_txn_id()) && rc == RCOK) { _cur_tid ++; commit_timestamp = _cur_tid; DEBUG("Validate success: %ld, cts: %ld \n", get_txn_id(), commit_timestamp); diff --git a/contrib/deneva/system/txn.h b/contrib/deneva/system/txn.h index d8a8ddd4..6a2560b7 100644 --- a/contrib/deneva/system/txn.h +++ b/contrib/deneva/system/txn.h @@ -1,12 +1,10 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn - + Copyright 2016 Massachusetts Institute of Technology Licensed under the Apache License, Version 2.0 (the "License"); @@ -25,11 +23,13 @@ #ifndef _TXN_H_ #define _TXN_H_ +#include #include "global.h" #include "helper.h" #include "semaphore.h" #include "array.h" #include "transport/message.h" +#include "../concurrency_control/unified_util.h" //#include "wl.h" class Workload; @@ -41,6 +41,7 @@ class INDEX; class TxnQEntry; class YCSBQuery; class TPCCQuery; +class DliValidatedTxn; //class r_query; enum TxnState {START,INIT,EXEC,PREP,FIN,DONE}; @@ -136,6 +137,10 @@ class TxnStats { double lat_other_time_start; }; +struct TxnNode; +struct Path; +class VersionInfo; + /* Execution of transactions Manipulates/manages Transaction (contains txn-specific data) @@ -160,7 +165,7 @@ class TxnManager { uint64_t get_thd_id(); Workload * get_wl(); void set_txn_id(txnid_t txn_id); - txnid_t get_txn_id(); + txnid_t get_txn_id() const; void set_query(BaseQuery * qry); BaseQuery * get_query(); bool is_done(); @@ -293,6 +298,9 @@ class TxnManager { int last_txn_id; Message* last_msg; +#if IS_GENERIC_ALG + std::unique_ptr, row_t*>> uni_txn_man_; +#endif protected: diff --git a/contrib/deneva/system/txn_table.cpp b/contrib/deneva/system/txn_table.cpp index 584a01d4..b9c3f13b 100644 --- a/contrib/deneva/system/txn_table.cpp +++ b/contrib/deneva/system/txn_table.cpp @@ -144,6 +144,10 @@ TxnManager * TxnTable::get_transaction_manager(uint64_t thd_id, uint64_t txn_id, return txn_man; } +#if WORKLOAD == DA +extern std::vector DA_delayed_operations; +#endif + void TxnTable::restart_txn(uint64_t thd_id, uint64_t txn_id,uint64_t batch_id){ uint64_t pool_id = txn_id % pool_size; // set modify bit for this pool: txn_id % pool_size @@ -156,6 +160,9 @@ void TxnTable::restart_txn(uint64_t thd_id, uint64_t txn_id,uint64_t batch_id){ if(is_matching_txn_node(t_node,txn_id,batch_id)) { #if CC_ALG == CALVIN work_queue.enqueue(thd_id,Message::create_message(t_node->txn_man,RTXN),false); +#elif WORKLOAD == DA + //TODO: if DA run distributedly, we need judge IS_LOCAL + DA_delayed_operations.push_back(Message::create_message(t_node->txn_man,RTXN_CONT)); #else if(IS_LOCAL(txn_id)) work_queue.enqueue(thd_id,Message::create_message(t_node->txn_man,RTXN_CONT),false); diff --git a/contrib/deneva/system/wl.cpp b/contrib/deneva/system/wl.cpp index f326e46c..40328e27 100644 --- a/contrib/deneva/system/wl.cpp +++ b/contrib/deneva/system/wl.cpp @@ -130,7 +130,7 @@ RC Workload::init_schema(const char * schema_file) { table_size = MAX_PPS_SUPPLIER_KEY; } #elif WORKLOAD == DA - if (!tname.compare(1, 5, "DAtab")) { + if (!tname.compare(0, 5, "DAtab")) { table_size = MAX_DA_TABLE_SIZE; } #else diff --git a/contrib/deneva/system/work_queue.cpp b/contrib/deneva/system/work_queue.cpp index 70bb675d..9e54ba7b 100644 --- a/contrib/deneva/system/work_queue.cpp +++ b/contrib/deneva/system/work_queue.cpp @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn diff --git a/contrib/deneva/system/work_queue.h b/contrib/deneva/system/work_queue.h index 7807771c..5251cb26 100644 --- a/contrib/deneva/system/work_queue.h +++ b/contrib/deneva/system/work_queue.h @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn diff --git a/contrib/deneva/system/worker_thread.cpp b/contrib/deneva/system/worker_thread.cpp index 17cd9042..a9087ef9 100644 --- a/contrib/deneva/system/worker_thread.cpp +++ b/contrib/deneva/system/worker_thread.cpp @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn @@ -46,6 +44,15 @@ #include "ssi.h" #include "focc.h" #include "bocc.h" +#include "dta.h" +#include "da.h" + +#define SUPPORT_ANOMALY_IDENTIFY (CC_ALG == DLI_IDENTIFY || CC_ALG == DLI_IDENTIFY_2) + +#if SUPPORT_ANOMALY_IDENTIFY && WORKLOAD == DA +static std::array, ttts::Count()> g_anomaly_counts = {0}; +static uint64_t g_no_anomaly_count = 0; +#endif void WorkerThread::setup() { if( get_thd_id() == 0) { @@ -275,12 +282,16 @@ void WorkerThread::abort() { INC_STATS(get_thd_id(), trans_finish_time, finish_timespan); INC_STATS(get_thd_id(), trans_abort_time, finish_timespan); INC_STATS(get_thd_id(), trans_total_run_time, timespan_short); - #if WORKLOAD != DA //actually DA do not need real abort. Just count it and do not send real abort msg. +#if WORKLOAD == DA + // DA do not need retry abort transactions. Just count it and do not send real abort msg. + // If not release txn man, later transactions will use the same txn_man which remains the + // current data and will not be initialized. + release_txn_man(); +#else uint64_t penalty = abort_queue.enqueue(get_thd_id(), txn_man->get_txn_id(), txn_man->get_abort_cnt()); - txn_man->txn_stats.total_abort_time += penalty; - #endif +#endif } TxnManager * WorkerThread::get_transaction_manager(Message * msg) { @@ -312,6 +323,36 @@ char type2char(DATxnType txn_type) } } +#if SUPPORT_ANOMALY_IDENTIFY && WORKLOAD == DA +static void print_anomaly_type_rates() { + std::cout.setf(std::ios::right); + std::cout.precision(4); + + static auto print_percent = [](auto&& value, auto&& total) { + std::cout << std::setw(10) << static_cast(value) / total * 100 << "% = " << std::setw(5) << value << " / " << std::setw(5) << total; + }; + + const auto anomaly_count = std::accumulate(g_anomaly_counts.begin(), g_anomaly_counts.end(), 0); + std::cout << "=== DLI_IDENTIFY ===" << std::endl; + + std::cout << std::setw(40) << "True Rollback: "; + print_percent(anomaly_count, anomaly_count + g_no_anomaly_count); + std::cout << std::endl; + + std::vector> sorted_g_anomaly_counts; + for (const auto anomaly : ttts::Members()) { + sorted_g_anomaly_counts.emplace_back(anomaly, g_anomaly_counts.at(static_cast(anomaly))); + } + std::sort(sorted_g_anomaly_counts.begin(), sorted_g_anomaly_counts.end(), [](auto&& _1, auto&& _2) { return _1.second > _2.second; }); + for (const auto& [anomaly, count] : sorted_g_anomaly_counts) { + std::cout << std::setw(40) << (std::string("[") + ToString(anomaly) + "] "); + print_percent(count, anomaly_count); + std::cout << std::endl; + } + std::cout << "=== DLI_IDENTIFY END ===" << std::endl; +} +#endif + RC WorkerThread::run() { tsetup(); printf("Running WorkerThread %ld\n",_thd_id); @@ -350,8 +391,10 @@ RC WorkerThread::run() { if (idle_starttime == 0) idle_starttime = get_sys_clock(); continue; } - simulation->last_da_query_time = get_sys_clock(); #if WORKLOAD == DA + simulation->last_da_query_time = get_sys_clock(); +#endif +#if WORKLOAD == DA && DA_PRINT_LOG == true printf("thd_id:%lu stxn_id:%lu batch_id:%lu seq_id:%lu type:%c rtype:%d trans_id:%lu item:%c laststate:%lu state:%lu next_state:%lu\n", this->_thd_id, ((DAClientQueryMessage*)msg)->txn_id, @@ -430,6 +473,9 @@ RC WorkerThread::run() { #endif INC_STATS(get_thd_id(),worker_release_msg_time,get_sys_clock() - ready_starttime); } +#if SUPPORT_ANOMALY_IDENTIFY && WORKLOAD == DA + print_anomaly_type_rates(); +#endif printf("FINISH %ld:%ld\n",_node_id,_thd_id); fflush(stdout); return FINISH; @@ -488,6 +534,23 @@ RC WorkerThread::process_rack_prep(Message * msg) { time_table.set_state(get_thd_id(),msg->get_txn_id(),MAAT_ABORTED); } #endif +#if CC_ALG == DTA + // Integrate bounds + uint64_t lower = ((AckMessage*)msg)->lower; + uint64_t upper = ((AckMessage*)msg)->upper; + if (lower > dta_time_table.get_lower(get_thd_id(), msg->get_txn_id())) { + dta_time_table.set_lower(get_thd_id(), msg->get_txn_id(), lower); + } + if (upper < dta_time_table.get_upper(get_thd_id(), msg->get_txn_id())) { + dta_time_table.set_upper(get_thd_id(), msg->get_txn_id(), upper); + } + DEBUG("%ld bound set: [%ld,%ld] -> [%ld,%ld]\n", msg->get_txn_id(), lower, upper, + dta_time_table.get_lower(get_thd_id(), msg->get_txn_id()), + dta_time_table.get_upper(get_thd_id(), msg->get_txn_id())); + if (((AckMessage*)msg)->rc != RCOK) { + dta_time_table.set_state(get_thd_id(), msg->get_txn_id(), DTA_ABORTED); + } +#endif #if CC_ALG == SILO uint64_t max_tid = ((AckMessage*)msg)->max_tid; @@ -589,6 +652,10 @@ RC WorkerThread::process_rqry(Message * msg) { #endif #if CC_ALG == MAAT time_table.init(get_thd_id(),txn_man->get_txn_id()); +#endif +#if CC_ALG == DTA + txn_table.update_min_ts(get_thd_id(), txn_man->get_txn_id(), 0, txn_man->get_timestamp()); + dta_time_table.init(get_thd_id(), txn_man->get_txn_id(), txn_man->get_timestamp()); #endif rc = txn_man->run_txn(); @@ -662,114 +729,188 @@ RC WorkerThread::process_rtxn(Message * msg) { RC rc = RCOK; uint64_t txn_id = UINT64_MAX; - if(msg->get_rtype() == CL_QRY) { - // This is a new transaction - // Only set new txn_id when txn first starts - #if WORKLOAD == DA - msg->txn_id=((DAClientQueryMessage*)msg)->trans_id; - txn_id=((DAClientQueryMessage*)msg)->trans_id; - #else - txn_id = get_next_txn_id(); - msg->txn_id = txn_id; - #endif - // Put txn in txn_table - txn_man = txn_table.get_transaction_manager(get_thd_id(),txn_id,0); - txn_man->register_thread(this); - uint64_t ready_starttime = get_sys_clock(); - bool ready = txn_man->unset_ready(); - INC_STATS(get_thd_id(),worker_activate_txn_time,get_sys_clock() - ready_starttime); - assert(ready); - if (CC_ALG == WAIT_DIE) { - #if WORKLOAD == DA //mvcc use timestamp - if (da_stamp_tab.count(txn_man->get_txn_id())==0) +#if WORKLOAD == DA && DA_PRINT_LOG == true + printf("thd_id:%lu check: state:%lu nextstate:%lu \n",h_thd->_thd_id, da_query->state, _wl->nextstate); + fflush(stdout); +#endif + +#if WORKLOAD == DA + auto da_wl = static_cast(_wl); + if (da_wl->nextstate != 0) { + while (((DAClientQueryMessage*)msg)->state != da_wl->nextstate && !simulation->is_done()); + } + + if (already_abort_tab.find(((DAClientQueryMessage*)msg)->trans_id) == already_abort_tab.end()) { +#endif + + if (msg->get_rtype() == CL_QRY) { + // This is a new transaction + // Only set new txn_id when txn first starts +#if WORKLOAD == DA + msg->txn_id=((DAClientQueryMessage*)msg)->trans_id; + txn_id=((DAClientQueryMessage*)msg)->trans_id; +#else + txn_id = get_next_txn_id(); + msg->txn_id = txn_id; +#endif + // Put txn in txn_table + // + txn_man = txn_table.get_transaction_manager(get_thd_id(),txn_id,0); + txn_man->register_thread(this); + uint64_t ready_starttime = get_sys_clock(); + bool ready = txn_man->unset_ready(); + INC_STATS(get_thd_id(),worker_activate_txn_time,get_sys_clock() - ready_starttime); + assert(ready); + if (CC_ALG == WAIT_DIE) { +#if WORKLOAD == DA //mvcc use timestamp + if (da_stamp_tab.count(txn_man->get_txn_id())==0) + { + da_stamp_tab[txn_man->get_txn_id()]=get_next_ts(); + txn_man->set_timestamp(da_stamp_tab[txn_man->get_txn_id()]); + } + else + txn_man->set_timestamp(da_stamp_tab[txn_man->get_txn_id()]); +#else + txn_man->set_timestamp(get_next_ts()); +#endif + } + txn_man->txn_stats.starttime = get_sys_clock(); + txn_man->txn_stats.restart_starttime = txn_man->txn_stats.starttime; + msg->copy_to_txn(txn_man); + DEBUG("START %ld %f %lu\n", txn_man->get_txn_id(), + simulation->seconds_from_start(get_sys_clock()), txn_man->txn_stats.starttime); +#if WORKLOAD==DA + if(da_start_trans_tab.count(txn_man->get_txn_id())==0) + { + da_start_trans_tab.insert(txn_man->get_txn_id()); + INC_STATS(get_thd_id(),local_txn_start_cnt,1); + } +#else + INC_STATS(get_thd_id(), local_txn_start_cnt, 1); +#endif + + } else { + txn_man->txn_stats.restart_starttime = get_sys_clock(); + DEBUG("RESTART %ld %f %lu\n", txn_man->get_txn_id(), + simulation->seconds_from_start(get_sys_clock()), txn_man->txn_stats.starttime); + } + // Get new timestamps + if(is_cc_new_timestamp()) { +#if WORKLOAD==DA //mvcc use timestamp + if(da_stamp_tab.count(txn_man->get_txn_id())==0) { - da_stamp_tab[txn_man->get_txn_id()]=get_next_ts(); - txn_man->set_timestamp(da_stamp_tab[txn_man->get_txn_id()]); + da_stamp_tab[txn_man->get_txn_id()]=get_next_ts(); + txn_man->set_timestamp(da_stamp_tab[txn_man->get_txn_id()]); } else - txn_man->set_timestamp(da_stamp_tab[txn_man->get_txn_id()]); - #else + txn_man->set_timestamp(da_stamp_tab[txn_man->get_txn_id()]); +#else txn_man->set_timestamp(get_next_ts()); - #endif - } - txn_man->txn_stats.starttime = get_sys_clock(); - txn_man->txn_stats.restart_starttime = txn_man->txn_stats.starttime; - msg->copy_to_txn(txn_man); - DEBUG("START %ld %f %lu\n", txn_man->get_txn_id(), - simulation->seconds_from_start(get_sys_clock()), txn_man->txn_stats.starttime); - #if WORKLOAD==DA - if(da_start_trans_tab.count(txn_man->get_txn_id())==0) - { - da_start_trans_tab.insert(txn_man->get_txn_id()); - INC_STATS(get_thd_id(),local_txn_start_cnt,1); - } - #else - INC_STATS(get_thd_id(), local_txn_start_cnt, 1); - #endif - - } else { - txn_man->txn_stats.restart_starttime = get_sys_clock(); - DEBUG("RESTART %ld %f %lu\n", txn_man->get_txn_id(), - simulation->seconds_from_start(get_sys_clock()), txn_man->txn_stats.starttime); - } - // Get new timestamps - if(is_cc_new_timestamp()) { - #if WORKLOAD==DA //mvcc use timestamp - if(da_stamp_tab.count(txn_man->get_txn_id())==0) - { - da_stamp_tab[txn_man->get_txn_id()]=get_next_ts(); - txn_man->set_timestamp(da_stamp_tab[txn_man->get_txn_id()]); +#endif } - else - txn_man->set_timestamp(da_stamp_tab[txn_man->get_txn_id()]); - #else - txn_man->set_timestamp(get_next_ts()); - #endif - } #if CC_ALG == MVCC - txn_table.update_min_ts(get_thd_id(),txn_id,0,txn_man->get_timestamp()); + txn_table.update_min_ts(get_thd_id(),txn_man->get_txn_id(),0,txn_man->get_timestamp()); #endif #if CC_ALG == WSI || CC_ALG == SSI - txn_table.update_min_ts(get_thd_id(),txn_id,0,txn_man->get_start_timestamp()); + txn_table.update_min_ts(get_thd_id(),txn_man->get_txn_id(),0,txn_man->get_start_timestamp()); #endif #if CC_ALG == OCC || CC_ALG == FOCC || CC_ALG == BOCC || CC_ALG == SSI || CC_ALG == WSI - #if WORKLOAD==DA - if(da_start_stamp_tab.count(txn_man->get_txn_id())==0) - { - da_start_stamp_tab[txn_man->get_txn_id()]=get_next_ts(); - txn_man->set_start_timestamp(da_start_stamp_tab[txn_man->get_txn_id()]); - } - else - txn_man->set_start_timestamp(da_start_stamp_tab[txn_man->get_txn_id()]); - #else - txn_man->set_start_timestamp(get_next_ts()); - #endif +#if WORKLOAD==DA + if(da_start_stamp_tab.count(txn_man->get_txn_id())==0) + { + da_start_stamp_tab[txn_man->get_txn_id()]=get_next_ts(); + txn_man->set_start_timestamp(da_start_stamp_tab[txn_man->get_txn_id()]); + } + else + txn_man->set_start_timestamp(da_start_stamp_tab[txn_man->get_txn_id()]); +#else + txn_man->set_start_timestamp(get_next_ts()); +#endif #endif #if CC_ALG == MAAT - #if WORKLOAD==DA - if(da_start_stamp_tab.count(txn_man->get_txn_id())==0) - { - da_start_stamp_tab[txn_man->get_txn_id()]=1; - time_table.init(get_thd_id(), txn_man->get_txn_id()); - assert(time_table.get_lower(get_thd_id(), txn_man->get_txn_id()) == 0); - assert(time_table.get_upper(get_thd_id(), txn_man->get_txn_id()) == UINT64_MAX); - assert(time_table.get_state(get_thd_id(), txn_man->get_txn_id()) == MAAT_RUNNING); - } - #else - time_table.init(get_thd_id(),txn_man->get_txn_id()); - assert(time_table.get_lower(get_thd_id(),txn_man->get_txn_id()) == 0); - assert(time_table.get_upper(get_thd_id(),txn_man->get_txn_id()) == UINT64_MAX); - assert(time_table.get_state(get_thd_id(),txn_man->get_txn_id()) == MAAT_RUNNING); - #endif +#if WORKLOAD==DA + if(da_start_stamp_tab.count(txn_man->get_txn_id())==0) + { + da_start_stamp_tab[txn_man->get_txn_id()]=1; + time_table.init(get_thd_id(), txn_man->get_txn_id()); + assert(time_table.get_lower(get_thd_id(), txn_man->get_txn_id()) == 0); + assert(time_table.get_upper(get_thd_id(), txn_man->get_txn_id()) == UINT64_MAX); + assert(time_table.get_state(get_thd_id(), txn_man->get_txn_id()) == MAAT_RUNNING); + } +#else + time_table.init(get_thd_id(),txn_man->get_txn_id()); + assert(time_table.get_lower(get_thd_id(),txn_man->get_txn_id()) == 0); + assert(time_table.get_upper(get_thd_id(),txn_man->get_txn_id()) == UINT64_MAX); + assert(time_table.get_state(get_thd_id(),txn_man->get_txn_id()) == MAAT_RUNNING); +#endif #endif +#if CC_ALG == DTA + txn_table.update_min_ts(get_thd_id(), txn_man->get_txn_id(), 0, txn_man->get_timestamp()); + dta_time_table.init(get_thd_id(), txn_man->get_txn_id(), txn_man->get_timestamp()); + // assert(dta_time_table.get_lower(get_thd_id(),txn_man->get_txn_id()) == 0); + assert(dta_time_table.get_upper(get_thd_id(), txn_man->get_txn_id()) == UINT64_MAX); + assert(dta_time_table.get_state(get_thd_id(), txn_man->get_txn_id()) == DTA_RUNNING); +#endif + rc = init_phase(); + if (rc != RCOK) return rc; - rc = init_phase(); - if (rc != RCOK) return rc; + // Execute transaction + rc = txn_man->run_txn(); + check_if_done(rc); +#if WORKLOAD == DA + } - // Execute transaction - rc = txn_man->run_txn(); - check_if_done(rc); + auto tmp_txn_man = txn_man; + if (!DA_delayed_operations.empty()) { + std::vector delayed_operations; + std::swap(DA_delayed_operations, delayed_operations); + for (auto msg : delayed_operations) { + txn_man = get_transaction_manager(msg); + const bool is_current_trans = txn_man == tmp_txn_man; + static_cast(txn_man)->set_not_waiting(); + process_rtxn_cont(msg); + if (is_current_trans && txn_man == nullptr) { + // current txn is over + // such as: C1 C2 *R2x *C2, current trans is 2. + // before process cont, trans 2 is not over because C2 is delayed, so tmp_txn_man is not null + // after process cont, trans is over because *C2 finished + // in this time, we need finally set txn_man NULL + tmp_txn_man = nullptr; + } + } + } + txn_man = tmp_txn_man; + + da_wl->nextstate = ((DAClientQueryMessage*)msg)->next_state; + if (da_wl->nextstate == 0) { + auto& output_file = abort_history ? abort_file : commit_file; + if (!DA_delayed_operations.empty()) { + output_file << "[ERROR] remain delayed operations: "; + DA_delayed_operations.clear(); + } +#if SUPPORT_ANOMALY_IDENTIFY + if (g_da_cycle.has_value()) { + const auto anomaly_type = UniAlgManager::IdentifyAnomaly(g_da_cycle->Preces()); + output_file << DA_history_mem << " | " << anomaly_type << " | " << *g_da_cycle << std::endl; + ++(g_anomaly_counts.at(static_cast(anomaly_type))); + g_da_cycle.reset(); + } else { + output_file << DA_history_mem << std::endl; + ++g_no_anomaly_count; + } + uni_alg_man.CheckConcurrencyTxnEmpty(); +#else + output_file << DA_history_mem << std::endl; +#endif + string().swap(DA_history_mem); + abort_history = false; + da_start_stamp_tab.clear(); + da_wl->reset_tab_idx(); + already_abort_tab.clear(); + da_start_trans_tab.clear(); + } +#endif return rc; } @@ -876,6 +1017,7 @@ RC WorkerNumThread::run() { while(!simulation->is_done()) { progress_stats(); +/* heap-buffer-overflow uint64_t wq_size = work_queue.get_wq_cnt(); uint64_t tx_size = work_queue.get_txn_cnt(); uint64_t ewq_size = work_queue.get_enwq_cnt(); @@ -897,6 +1039,7 @@ RC WorkerNumThread::run() { INC_STATS(_thd_id,work_queue_etx_cnt[i],etx_size); INC_STATS(_thd_id,work_queue_dtx_cnt[i],dtx_size); +*/ i++; sleep(1); diff --git a/contrib/deneva/system/worker_thread.h b/contrib/deneva/system/worker_thread.h index b2dafdef..bdd5da13 100644 --- a/contrib/deneva/system/worker_thread.h +++ b/contrib/deneva/system/worker_thread.h @@ -1,9 +1,7 @@ /* Tencent is pleased to support the open source community by making 3TS available. - Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - Tencent Modifications are Copyright (C) THL A29 Limited. + Copyright (C) 2020 Tencent. All rights reserved. Author: hongyaozhao@ruc.edu.cn diff --git a/contrib/deneva/transport/message.cpp b/contrib/deneva/transport/message.cpp index 3c071019..e1fdf2b8 100644 --- a/contrib/deneva/transport/message.cpp +++ b/contrib/deneva/transport/message.cpp @@ -25,6 +25,7 @@ #include "global.h" #include "message.h" #include "maat.h" +#include "dta.h" #include "da.h" #include "da_query.h" #include "sundial.h" @@ -492,8 +493,8 @@ void YCSBClientQueryMessage::copy_from_buf(char * buf) { for(uint64_t i = 0 ; i < size;i++) { DEBUG_M("YCSBClientQueryMessage::copy ycsb_request alloc\n"); ycsb_request * req = (ycsb_request*)mem_allocator.alloc(sizeof(ycsb_request)); - COPY_VAL(*req,buf,ptr); - + *req = *reinterpret_cast(buf + ptr); + ptr += sizeof(ycsb_request); assert(req->key < g_synth_table_size); requests.add(req); } @@ -1126,6 +1127,10 @@ void AckMessage::copy_from_txn(TxnManager * txn) { lower = time_table.get_lower(txn->get_thd_id(),txn->get_txn_id()); upper = time_table.get_upper(txn->get_thd_id(),txn->get_txn_id()); #endif +#if CC_ALG == DTA || CC_ALG == DLI_DTA || CC_ALG == DLI_DTA2 || CC_ALG == DLI_DTA3 + lower = dta_time_table.get_lower(txn->get_thd_id(), txn->get_txn_id()); + upper = dta_time_table.get_upper(txn->get_thd_id(), txn->get_txn_id()); +#endif #if CC_ALG == SILO max_tid = txn->max_tid; #endif @@ -1406,7 +1411,8 @@ void YCSBQueryMessage::copy_from_buf(char * buf) { for(uint64_t i = 0 ; i < size;i++) { DEBUG_M("YCSBQueryMessage::copy ycsb_request alloc\n"); ycsb_request * req = (ycsb_request*)mem_allocator.alloc(sizeof(ycsb_request)); - COPY_VAL(*req,buf,ptr); + *req = *reinterpret_cast(buf + ptr); + ptr += sizeof(ycsb_request); ASSERT(req->key < g_synth_table_size); requests.add(req); } diff --git a/contrib/deneva/transport/message.h b/contrib/deneva/transport/message.h index f7276d1c..d8df4f8b 100644 --- a/contrib/deneva/transport/message.h +++ b/contrib/deneva/transport/message.h @@ -173,7 +173,7 @@ class AckMessage : public Message { void release() {} RC rc; -#if CC_ALG == MAAT +#if CC_ALG == MAAT || CC_ALG == DTA || CC_ALG == DLI_DTA || CC_ALG == DLI_DTA2 || CC_ALG == DLI_DTA3 uint64_t lower; uint64_t upper; #endif diff --git a/contrib/deneva/transport/msg_thread.cpp b/contrib/deneva/transport/msg_thread.cpp index ba656b97..0bb99c20 100644 --- a/contrib/deneva/transport/msg_thread.cpp +++ b/contrib/deneva/transport/msg_thread.cpp @@ -118,13 +118,14 @@ void MessageThread::run() { #if WORKLOAD == DA if(!is_server&&false) - printf("cl seq_id:%lu type:%c trans_id:%lu item:%c state:%lu next_state:%lu\n", + printf("cl seq_id:%lu type:%c trans_id:%lu item:%c state:%lu next_state:%lu write_version:%lu\n", ((DAClientQueryMessage*)msg)->seq_id, type2char1(((DAClientQueryMessage*)msg)->txn_type), ((DAClientQueryMessage*)msg)->trans_id, static_cast('x'+((DAClientQueryMessage*)msg)->item_id), ((DAClientQueryMessage*)msg)->state, - (((DAClientQueryMessage*)msg)->next_state)); + (((DAClientQueryMessage*)msg)->next_state), + ((DAClientQueryMessage*)msg)->write_version); fflush(stdout); #endif diff --git a/contrib/deneva/transport/transport.cpp b/contrib/deneva/transport/transport.cpp index 8fb9c241..3ceb8039 100644 --- a/contrib/deneva/transport/transport.cpp +++ b/contrib/deneva/transport/transport.cpp @@ -253,8 +253,19 @@ void Transport::send_msg(uint64_t send_thread_id, uint64_t dest_node_id, void * int rc = -1; while (rc < 0 && (!simulation->is_setup_done() || - (simulation->is_setup_done() && !simulation->is_done()))) { - rc= socket->sock.send(&buf,NN_MSG,NN_DONTWAIT); + (simulation->is_setup_done() && !simulation->is_done() +#if WORKLOAD == DA + && !simulation->da_server_iothread_timeout() +#endif + ))) { +#if WORKLOAD == DA + try { +#endif + rc= socket->sock.send(&buf,NN_MSG,NN_DONTWAIT); +#if WORKLOAD == DA + } catch (const nn::exception&) { + } +#endif } //nn_freemsg(sbuf); DEBUG("%ld Batch of %d bytes sent to node %ld\n",send_thread_id,size,dest_node_id); @@ -287,22 +298,29 @@ std::vector * Transport::recv_msg(uint64_t thd_id) { while (bytes <= 0 && (!simulation->is_setup_done() || (simulation->is_setup_done() && !simulation->is_done()))) { Socket * socket = recv_sockets[ctr]; - bytes = socket->sock.recv(&buf, NN_MSG, NN_DONTWAIT); +#if WORKLOAD == DA + try { +#endif + bytes = socket->sock.recv(&buf, NN_MSG, NN_DONTWAIT); +#if WORKLOAD == DA + } catch (const nn::exception&) { + } +#endif //++ctr; ctr = (ctr + g_this_rem_thread_cnt); if (ctr >= recv_sockets.size()) ctr = (thd_id % g_this_rem_thread_cnt) % recv_sockets.size(); if (ctr == start_ctr) break; - if(bytes <= 0 && errno != 11) { + if (bytes <= 0 && errno != 11) { printf("Recv Error %d %s\n",errno,strerror(errno)); nn::freemsg(buf); } if (bytes > 0) break; - } + } - if(bytes <= 0 ) { + if (bytes <= 0 ) { INC_STATS(thd_id,msg_recv_idle_time, get_sys_clock() - starttime); return msgs; } diff --git a/contrib/deneva/transport/transport.h b/contrib/deneva/transport/transport.h index 2fe0c37b..b4d6341c 100644 --- a/contrib/deneva/transport/transport.h +++ b/contrib/deneva/transport/transport.h @@ -38,7 +38,7 @@ Data: MSG_SIZE - HDR_SIZE bytes class Socket { public: Socket () : sock(AF_SP,NN_PAIR) {} - ~Socket () { delete &sock;} + ~Socket () {} char _pad1[CL_SIZE]; nn::socket sock; char _pad[CL_SIZE - sizeof(nn::socket)]; diff --git a/contrib/deneva/unified_concurrency_control/alg_dli_identify_chain.h b/contrib/deneva/unified_concurrency_control/alg_dli_identify_chain.h new file mode 100644 index 00000000..81062272 --- /dev/null +++ b/contrib/deneva/unified_concurrency_control/alg_dli_identify_chain.h @@ -0,0 +1,136 @@ +#pragma once + +#include "dli_identify_util.h" +#include "row_prece.h" +#include +#include + +namespace ttts { + +template +class AlgManager> +{ + public: + using Txn = TxnManager; + + bool Validate(Txn& txn) + { + txn.node_->state() = TxnNode::State::PREPARING; + + { + std::scoped_lock l(m_); + cc_txns_.emplace_back(txn.node_); + } + + Path cycle_part = DirtyCycle(*txn.node_); + + if (!cycle_part.Passable()) { + cycle_part = CyclePart_(txn); + } + + if (!cycle_part.Passable()) { + return true; + } else { + txn.cycle_ = std::make_unique(std::move(cycle_part)); + return false; + } + } + + void Commit(Txn& txn) + { + txn.node_->Commit(); + } + + void Abort(Txn& txn) + { + if (!txn.cycle_) { + // we can only identify the dirty write/read anomaly rather than avoiding it + Path cycle = DirtyCycle(*txn.node_); + if (cycle.Passable()) { + txn.cycle_ = std::make_unique(std::move(cycle)); + } + } + if (const std::unique_ptr& cycle = txn.cycle_) { + txn.node_->Abort(true /*clear_to_txns*/); // break cycles to prevent memory leak + } else { + txn.node_->Abort(false /*clear_to_txns*/); + } + } + + void CheckConcurrencyTxnEmpty() { + std::scoped_lock l(m_); + bool is_empty = true; + for (const auto& weak_txn : cc_txns_) { + if (const auto txn = weak_txn.lock()) { + std::cerr << "** Txn Leak ** " << *txn; + is_empty = false; + } + } + // assert failed here means there is actually a cycle but we miss it + assert(is_empty); + cc_txns_.clear(); + } + + static AnomalyType IdentifyAnomaly(const std::vector& preces) + { + assert(preces.size() == 2); + const auto& p1 = preces.front(); + const auto& p2 = preces.back(); + if (std::any_of(preces.begin(), preces.end(), + [](const PreceInfo& prece) { return prece.type() == PreceType::WA || + prece.type() == PreceType::WC; })) { + return AnomalyType::WAT_1_DIRTY_WRITE; + } else if (std::any_of(preces.begin(), preces.end(), + [](const PreceInfo& prece) { return prece.type() == PreceType::RA; })) { + return AnomalyType::RAT_1_DIRTY_READ; + } else if (p1.from_txn_id() != p2.to_txn_id()) { + return IdentifyAnomalyMultiple(preces); + // [Note] When build path, later happened precedence is sorted to back, which is DIFFERENT from 3TS-DA + } else if (p1.row_id() != p2.row_id()) { + return IdentifyAnomalyDouble(p1.type(), p2.type()); + } else if (p1.to_ver_id() < p2.to_ver_id() || + (p1.to_ver_id() == p2.to_ver_id() && p1.from_ver_id() < p2.from_ver_id())) { + return IdentifyAnomalySingle(p1.type(), p2.type()); + } else { + return IdentifyAnomalySingle(p2.type(), p1.type()); + } + } + + private: + Path CyclePart_(Txn& txn) + { + // Validate failed if has a from_txn and a to_txn which are both finished but not released. + std::scoped_lock l(txn.node_->mutex()); + std::shared_ptr from_prece; + for (const auto& [_, weak_from_prece] : txn.node_->UnsafeGetFromPreces()) { + from_prece = weak_from_prece.lock(); + if (from_prece == nullptr) { + continue; + } + const auto& from_txn = from_prece->from_txn(); + if (from_txn == nullptr || from_txn->state() == TxnNode::State::ACTIVE) { + from_prece = nullptr; + continue; + } + break; + } + + std::shared_ptr to_prece; + for (const auto& [_, to_prece_tmp] : txn.node_->UnsafeGetToPreces()) { + if (to_prece_tmp->to_txn()->state() != TxnNode::State::ACTIVE) { + to_prece = to_prece_tmp; + break; + } + } + + if (from_prece && to_prece) { + return Path(std::vector{*from_prece, *to_prece}); + } + return {}; + } + + std::mutex m_; + std::vector> cc_txns_; // only for debug +}; + +} diff --git a/contrib/deneva/unified_concurrency_control/alg_dli_identify_cycle.h b/contrib/deneva/unified_concurrency_control/alg_dli_identify_cycle.h new file mode 100644 index 00000000..e97b6230 --- /dev/null +++ b/contrib/deneva/unified_concurrency_control/alg_dli_identify_cycle.h @@ -0,0 +1,199 @@ +/* Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ + +#pragma once + +#include "dli_identify_util.h" +#include "row_prece.h" +#include +#include +#include +#include + +namespace ttts { + +template +class AlgManager> +{ + public: + using Txn = TxnManager; + + bool Validate(Txn& txn) + { + txn.node_->state() = TxnNode::State::PREPARING; + + //txn-.lock(m_); // release after build WC, WA or RA precedence + const auto cc_txns = [this, &txn] { + std::scoped_lock l(m_); + cc_txns_.emplace_back(txn.node_); + return RefreshAndLockTxns_(); + }(); + const auto txn_num = cc_txns.size(); + const auto this_idx = txn_num - 1; // this txn node is just emplaced to the end of cc_txns_ + + auto matrix = InitPathMatrix_(cc_txns); + UpdatePathMatrix_(matrix); + + Path cycle = MinCycle_(matrix, this_idx); + + if (!cycle.Passable()) { + cycle = DirtyCycle(*txn.node_); + } + + if (!cycle.Passable()) { + return true; + } else { + txn.cycle_ = std::make_unique(std::move(cycle)); + return false; + } + } + + void Commit(Txn& txn) + { + txn.node_->Commit(); + } + + void Abort(Txn& txn) + { + if (!txn.cycle_) { + // we can only identify the dirty write/read anomaly rather than avoiding it + Path cycle = DirtyCycle(*txn.node_); + if (cycle.Passable()) { + txn.cycle_ = std::make_unique(std::move(cycle)); + } + } + if (const std::unique_ptr& cycle = txn.cycle_) { + txn.node_->Abort(true /*clear_to_txns*/); // break cycles to prevent memory leak + } else { + txn.node_->Abort(false /*clear_to_txns*/); + } + } + + void CheckConcurrencyTxnEmpty() + { + std::scoped_lock l(m_); + for (auto it = cc_txns_.begin(); it != cc_txns_.end(); ) { + if (const auto txn = it->lock()) { + std::cerr << "** Txn Leak ** " << *txn; + ++it; + } else { + it = cc_txns_.erase(it); + } + } + // assert failed here means there is actually a cycle but we miss it + assert(cc_txns_.empty()); + } + + static AnomalyType IdentifyAnomaly(const std::vector& preces) + { + assert(preces.size() >= 2); + const auto& p1 = preces.front(); + const auto& p2 = preces.back(); + if (std::any_of(preces.begin(), preces.end(), + [](const PreceInfo& prece) { return prece.type() == PreceType::WA || + prece.type() == PreceType::WC; })) { + return AnomalyType::WAT_1_DIRTY_WRITE; + } else if (std::any_of(preces.begin(), preces.end(), + [](const PreceInfo& prece) { return prece.type() == PreceType::RA; })) { + return AnomalyType::RAT_1_DIRTY_READ; + } else if (preces.size() >= 3) { + return IdentifyAnomalyMultiple(preces); + // [Note] When build path, later happened precedence is sorted to back, which is DIFFERENT from 3TS-DA + } else if (p1.row_id() != p2.row_id()) { + return IdentifyAnomalyDouble(p1.type(), p2.type()); + } else { + return IdentifyAnomalySingle(p1.type(), p2.type()); + } + } + + private: + std::vector> RefreshAndLockTxns_() + { + std::vector> txns; + for (auto it = cc_txns_.begin(); it != cc_txns_.end(); ) { + if (auto txn = it->lock()) { + txns.emplace_back(std::move(txn)); + ++it; + } else { + it = cc_txns_.erase(it); + } + } + return txns; + } + + std::vector> InitPathMatrix_( + const std::vector>& cc_txns) const + { + const uint64_t txn_num = cc_txns.size(); + std::vector> matrix(txn_num); + for (uint64_t from_idx = 0; from_idx < txn_num; ++from_idx) { + matrix[from_idx].resize(txn_num); + TxnNode& from_txn_node = *cc_txns[from_idx]; + std::lock_guard l(from_txn_node.mutex()); + const auto& to_preces = from_txn_node.UnsafeGetToPreces(); + for (uint64_t to_idx = 0; to_idx < txn_num; ++to_idx) { + const uint64_t to_txn_id = cc_txns[to_idx]->txn_id(); + if (const auto it = to_preces.find(to_txn_id); it != to_preces.end()) { + matrix[from_idx][to_idx] = *it->second; + } + } + } + return matrix; + } + + static void UpdatePath_(Path& path, Path&& new_path) { + if (new_path < path) { + path = std::move(new_path); // do not use std::min because there is a copy cost when assign self + } + }; + + void UpdatePathMatrix_(std::vector>& matrix) const + { + const uint64_t txn_num = matrix.size(); + for (uint64_t mid = 0; mid < txn_num; ++mid) { + for (uint64_t start = 0; start < txn_num; ++start) { + if (start == mid) { + continue; + } + for (uint64_t end = 0; end < txn_num; ++end) { + if (end != mid && start != end) { + UpdatePath_(matrix[start][end], matrix[start][mid] + matrix[mid][end]); + } + } + } + } + } + + Path MinCycle_(std::vector>& matrix, const uint64_t this_idx) const + { + const uint64_t txn_num = matrix.size(); + Path min_cycle; + for (uint64_t start = 0; start < txn_num; ++start) { + if (start == this_idx) { + continue; + } + // We should try different order because the path may be imPassable when order is wrong. + UpdatePath_(min_cycle, matrix[start][this_idx] + matrix[this_idx][start]); + UpdatePath_(min_cycle, matrix[this_idx][start] + matrix[start][this_idx]); + assert(!min_cycle.Passable() || min_cycle.IsCycle()); + for (uint64_t end = 0; end < txn_num; ++end) { + if (end != this_idx && start != end) { + // Here try once is ok because start and end will exchange later. + UpdatePath_(min_cycle, matrix[start][end] + matrix[end][this_idx] + matrix[this_idx][start]); + assert(!min_cycle.Passable() || min_cycle.IsCycle()); + } + } + } + return min_cycle; + } + + std::mutex m_; + std::list> cc_txns_; +}; + +} diff --git a/contrib/deneva/unified_concurrency_control/alg_dli_identify_merge.h b/contrib/deneva/unified_concurrency_control/alg_dli_identify_merge.h new file mode 100644 index 00000000..2113079d --- /dev/null +++ b/contrib/deneva/unified_concurrency_control/alg_dli_identify_merge.h @@ -0,0 +1,143 @@ +/* Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ + +#pragma once + +#include "dli_identify_util.h" +#include "row_prece.h" +#include +#include + +namespace ttts { + +template +class AlgManager> +{ + public: + using Txn = TxnManager; + + bool Validate(Txn& txn) + { + txn.node_->state() = TxnNode::State::PREPARING; + + { + std::scoped_lock l(m_); + cc_txns_.emplace_back(txn.node_); + } + + Path cycle_part = DirtyCycle(*txn.node_); + + if (!cycle_part.Passable()) { + cycle_part = CyclePart_(txn); + } + + if (!cycle_part.Passable()) { + return true; + } else { + txn.cycle_ = std::make_unique(std::move(cycle_part)); + return false; + } + } + + void Commit(Txn& txn) + { + txn.node_->Commit(); + } + + void Abort(Txn& txn) + { + if (!txn.cycle_) { + // we can only identify the dirty write/read anomaly rather than avoiding it + Path cycle = DirtyCycle(*txn.node_); + txn.cycle_ = std::make_unique(std::move(cycle)); + } + if (const std::unique_ptr& cycle = txn.cycle_) { + txn.node_->Abort(true /*clear_to_txns*/); // break cycles to prevent memory leak + } else { + txn.node_->Abort(false /*clear_to_txns*/); + } + } + + void CheckConcurrencyTxnEmpty() { + std::scoped_lock l(m_); + bool is_empty = true; + for (const auto& weak_txn : cc_txns_) { + if (const auto txn = weak_txn.lock()) { + std::cerr << "** Txn Leak ** " << *txn; + is_empty = false; + } + } + // assert failed here means there is actually a cycle but we miss it + assert(is_empty); + cc_txns_.clear(); + } + + static AnomalyType IdentifyAnomaly(const std::vector& preces) + { + assert(preces.size() == 2); + const auto& p1 = preces.front(); + const auto& p2 = preces.back(); + if (std::any_of(preces.begin(), preces.end(), + [](const PreceInfo& prece) { return prece.type() == PreceType::WA || + prece.type() == PreceType::WC; })) { + return AnomalyType::WAT_1_DIRTY_WRITE; + } else if (std::any_of(preces.begin(), preces.end(), + [](const PreceInfo& prece) { return prece.type() == PreceType::RA; })) { + return AnomalyType::RAT_1_DIRTY_READ; + } else if (p1.from_txn_id() != p2.to_txn_id()) { + return IdentifyAnomalyMultiple(preces); + // [Note] When build path, later happened precedence is sorted to back, which is DIFFERENT from 3TS-DA + } else if (p1.row_id() != p2.row_id()) { + return IdentifyAnomalyDouble(p1.type(), p2.type()); + } else if (p1.to_ver_id() < p2.to_ver_id() || + (p1.to_ver_id() == p2.to_ver_id() && p1.from_ver_id() < p2.from_ver_id())) { + return IdentifyAnomalySingle(p1.type(), p2.type()); + } else { + return IdentifyAnomalySingle(p2.type(), p1.type()); + } + } + + private: + Path CyclePart_(Txn& txn) + { + // Validate failed if has a from_txn and a to_txn which are both finished but not released. + std::scoped_lock l(txn.node_->mutex()); + std::shared_ptr from_prece; + for (const auto& [_, weak_from_prece] : txn.node_->UnsafeGetFromPreces()) { + from_prece = weak_from_prece.lock(); + if (from_prece == nullptr) { + continue; + } + const auto& from_txn = from_prece->from_txn(); + if (from_txn == nullptr || from_txn->state() == TxnNode::State::ACTIVE) { + from_prece = nullptr; + continue; + } + break; + } + + std::shared_ptr to_prece; + for (const auto& [_, to_prece_tmp] : txn.node_->UnsafeGetToPreces()) { + if (to_prece_tmp->to_txn()->state() != TxnNode::State::ACTIVE) { + to_prece = to_prece_tmp; + break; + } + } + + if (from_prece && to_prece) { + return Path(std::vector{*from_prece, *to_prece}); + } + return {}; + } + + std::mutex m_; + std::vector> cc_txns_; // only for debug +}; + +} + diff --git a/contrib/deneva/unified_concurrency_control/dli_identify_util.h b/contrib/deneva/unified_concurrency_control/dli_identify_util.h new file mode 100644 index 00000000..c0d59203 --- /dev/null +++ b/contrib/deneva/unified_concurrency_control/dli_identify_util.h @@ -0,0 +1,247 @@ +/* Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ + +#pragma once + +#include "row_prece.h" +#include +#include +#include +#include +#include +#include + +#define CYCLE_ORDER 1 + +namespace ttts { + +class Path { + public: + inline Path(); + inline Path(std::vector&& preces); + inline Path(const PreceInfo& prece); + inline Path(PreceInfo&& prece); + + Path(const Path&) = default; + Path(Path&&) = default; + Path& operator=(const Path&) = default; + Path& operator=(Path&&) = default; + + inline bool operator<(const Path& p) const; + inline Path& operator+=(const Path& p); + inline Path operator+(const Path& p) const; + inline friend std::ostream& operator<<(std::ostream& os, const Path& path); + + inline std::string ToString() const; + inline bool Passable() const { return !preces_.empty(); } + inline const std::vector& Preces() const { return preces_; } + bool IsCycle() const { + return preces_.size() >= 2 && preces_.front().from_txn_id() == preces_.back().to_txn_id(); + } + + private: + std::vector preces_; +}; + +static Path DirtyPath_(const PreceInfo& rw_prece, TxnNode& txn_to_finish, const PreceType type) { + PreceInfo dirty_prece(rw_prece.to_txn(), txn_to_finish.shared_from_this(), type, rw_prece.row_id(), + rw_prece.to_ver_id(), UINT64_MAX); + return Path(std::vector{rw_prece, std::move(dirty_prece)}); +} + +template +inline Path DirtyCycle(TxnNode& txn_to_finish) { + std::lock_guard l(txn_to_finish.mutex()); + const auto& preces = txn_to_finish.UnsafeGetDirtyToPreces(); + for (const auto& prece : preces) { + if (!prece || prece->to_txn()->state() == TxnNode::State::ABORTED) { + // do nothing and continue + } else if (prece->type() == PreceType::WW) { + return DirtyPath_(*prece, txn_to_finish, IS_COMMIT ? PreceType::WC : PreceType::WA); + } else if (prece->type() == PreceType::WR) { + if (!IS_COMMIT) { + return DirtyPath_(*prece, txn_to_finish, PreceType::RA); + } + } else { + assert(false); + return {}; + } + } + return {}; // no dirty cycle +} + +Path::Path() {} + +Path::Path(std::vector&& preces) : preces_( +#if CYCLE_ORDER + std::move(preces)) +#else + (sort(preces, std::greater()), std::move(preces))) +#endif +{} + +Path::Path(const PreceInfo& prece) : preces_{prece} {} + +Path::Path(PreceInfo&& prece) : preces_{std::move(prece)} {} + +bool Path::operator<(const Path& p) const { + // imPassable has the greatest weight + if (!Passable()) { + return false; + } + if (!p.Passable()) { + return true; + } +#if CYCLE_ORDER + return preces_.size() < p.preces_.size(); +#else + return std::lexicographical_compare(preces_.begin(), preces_.end(), p.preces_.begin(), p.preces_.end()); +#endif +} + +Path& Path::operator+=(const Path& p) { + if (!Passable() || !p.Passable()) { + preces_.clear(); + return *this; + } +#if CYCLE_ORDER + const auto& current_back = preces_.back(); + const auto& append_front = p.preces_.front(); + assert(current_back.to_txn_id() == append_front.from_txn_id()); + + const auto cat_preces = [&](auto&& begin) { + for (auto it = begin; it != p.preces_.end(); ++it) { + preces_.emplace_back(*it); + } + }; + + const uint64_t new_from_ver_id = current_back.from_ver_id(); + const uint64_t new_to_ver_id = append_front.to_ver_id(); + const OperationType new_from_op_type = current_back.from_op_type(); + const OperationType new_to_op_type = append_front.to_op_type(); + bool merged = false; + if (current_back.row_id() == append_front.row_id()) { + if (new_from_ver_id >= new_to_ver_id) { + // it breaks the operation order on the same row + preces_.clear(); + return *this; + } + // merge two precedence on the same row + if (current_back.from_txn_id() != append_front.to_txn_id() && + (new_from_op_type != OperationType::R || new_to_op_type != OperationType::R)) { + preces_.pop_back(); + const auto new_prece_type = MergeOperationType(new_from_op_type, new_to_op_type); + preces_.emplace_back(current_back.from_txn(), append_front.to_txn(), new_prece_type, + current_back.row_id(), new_from_ver_id, new_to_ver_id); + // the first precedence in p.prece has already merged + cat_preces(std::next(p.preces_.begin())); + merged = true; + } + } + if (!merged) { + cat_preces(p.preces_.begin()); + } +#else + std::vector preces; + std::merge(preces_.begin(), preces_.end(), p.preces_.begin(), p.preces_.end(), std::back_inserter(preces), std::greater()); +#endif + return *this; +} + +Path Path::operator+(const Path& p) const { + return Path(*this) += p; +} + +std::string Path::ToString() const { + std::stringstream ss; + ss << *this; + return ss.str(); +} + +std::ostream& operator<<(std::ostream& os, const Path& path) { + if (path.preces_.empty()) { + os << "Empty path"; + } else { + std::copy(path.preces_.begin(), path.preces_.end(), std::ostream_iterator(os, ", ")); + } + return os; +} + +// require type1 precedence happens before type2 precedence +static AnomalyType IdentifyAnomalySingle(const PreceType early_type, const PreceType later_type) { + if ((early_type == PreceType::WW || early_type == PreceType::WR) && (later_type == PreceType::WW || later_type == PreceType::WCW)) { + return AnomalyType::WAT_1_FULL_WRITE; // WW-WW | WR-WW = WWW + } else if (early_type == PreceType::WR && early_type == PreceType::WW) { + return AnomalyType::WAT_1_FULL_WRITE; // WR-WW = WWW + } else if ((early_type == PreceType::WW || early_type == PreceType::WR) && (later_type == PreceType::WR || later_type == PreceType::WCR)) { + return AnomalyType::WAT_1_LOST_SELF_UPDATE; // WW-WR = WWR + } else if (early_type == PreceType::RW && later_type == PreceType::WW) { + return AnomalyType::WAT_1_LOST_UPDATE; // RW-WW | RW-RW = RWW + } else if (early_type == PreceType::RW && later_type == PreceType::RW) { + return AnomalyType::WAT_1_LOST_UPDATE; // RW-WW | RW-RW = RWW + } else if (early_type == PreceType::WR && later_type == PreceType::RW) { + return AnomalyType::RAT_1_INTERMEDIATE_READ; // WR-RW = WRW + } else if (early_type == PreceType::RW && (later_type == PreceType::WR || later_type == PreceType::WCR)) { + return AnomalyType::RAT_1_NON_REPEATABLE_READ; // RW-WR = RWR + } else if (early_type == PreceType::RW && later_type == PreceType::WCW) { + return AnomalyType::IAT_1_LOST_UPDATE_COMMITTED; // RW-WW(WCW) = RWW + } else { + return AnomalyType::UNKNOWN_1; + } +} + +static AnomalyType IdentifyAnomalyDouble(const PreceType early_type, const PreceType later_type) { + const auto any_order = [early_type, later_type](const PreceType type1, const PreceType type2) -> std::optional { + if (early_type == type1 && later_type == type2) { + return true; + } else if (early_type == type2 && later_type == type1) { + return false; + } else { + return {}; + } + }; + if (const auto order = any_order(PreceType::WR, PreceType::WW); order.has_value()) { + return *order ? AnomalyType::WAT_2_DOUBLE_WRITE_SKEW_1 : AnomalyType::WAT_2_DOUBLE_WRITE_SKEW_2; + } else if (any_order(PreceType::WW, PreceType::WCR)) { + return AnomalyType::WAT_2_DOUBLE_WRITE_SKEW_2; + } else if (const auto order = any_order(PreceType::RW, PreceType::WW); order.has_value()) { + return *order ? AnomalyType::WAT_2_READ_WRITE_SKEW_1 : AnomalyType::WAT_2_READ_WRITE_SKEW_2; + } else if (any_order(PreceType::WW, PreceType::WW)) { + return AnomalyType::WAT_2_FULL_WRITE_SKEW; + } else if (any_order(PreceType::WW, PreceType::WCW)) { + return AnomalyType::WAT_2_FULL_WRITE_SKEW; + } else if (any_order(PreceType::WR, PreceType::WR)) { + return AnomalyType::RAT_2_WRITE_READ_SKEW; + } else if (any_order(PreceType::WR, PreceType::WCR)) { + return AnomalyType::RAT_2_WRITE_READ_SKEW; + } else if (any_order(PreceType::WR, PreceType::WCW)) { + return AnomalyType::RAT_2_DOUBLE_WRITE_SKEW_COMMITTED; + } else if (const auto order = any_order(PreceType::RW, PreceType::WR); order.has_value()) { + return *order ? AnomalyType::RAT_2_READ_SKEW : AnomalyType::RAT_2_READ_SKEW_2; + } else if (any_order(PreceType::RW, PreceType::WCR)) { + return AnomalyType::RAT_2_READ_SKEW; + } else if (any_order(PreceType::RW, PreceType::WCW)) { + return AnomalyType::IAT_2_READ_WRITE_SKEW_COMMITTED; + } else if (any_order(PreceType::RW, PreceType::RW)) { + return AnomalyType::IAT_2_WRITE_SKEW; + } else { + return AnomalyType::UNKNOWN_2; + } +} + +static AnomalyType IdentifyAnomalyMultiple(const std::vector& preces) { + if (std::any_of(preces.begin(), preces.end(), [](const PreceInfo& prece) { return prece.type() == PreceType::WW; })) { + return AnomalyType::WAT_STEP; + } + if (std::any_of(preces.begin(), preces.end(), [](const PreceInfo& prece) { return prece.type() == PreceType::WR || prece.type() == PreceType::WCR; })) { + return AnomalyType::RAT_STEP; + } + return AnomalyType::IAT_STEP; +} + +} diff --git a/contrib/deneva/unified_concurrency_control/operation_type.h b/contrib/deneva/unified_concurrency_control/operation_type.h new file mode 100644 index 00000000..f2393ab8 --- /dev/null +++ b/contrib/deneva/unified_concurrency_control/operation_type.h @@ -0,0 +1,24 @@ +#ifdef ENUM_BEGIN +#ifdef ENUM_MEMBER +#ifdef ENUM_END + +ENUM_BEGIN(OperationType) +ENUM_MEMBER(OperationType, W) +ENUM_MEMBER(OperationType, R) +ENUM_MEMBER(OperationType, C) +ENUM_MEMBER(OperationType, A) +ENUM_END(OperationType) + +#endif +#endif +#endif + +#ifndef TTTS_DENEVA_OPERATION_TYPE_H_ +#define TTTS_DENEVA_OPERATION_TYPE_H_ + +namespace ttts { +#define ENUM_FILE "../../../../contrib/deneva/unified_concurrency_control/operation_type.h" +#include "../../../src/3ts/backend/util/extend_enum.h" +} + +#endif // TTTS_DENEVA_OPERATION_TYPE_H_ diff --git a/contrib/deneva/unified_concurrency_control/row_prece.h b/contrib/deneva/unified_concurrency_control/row_prece.h new file mode 100644 index 00000000..ec69da5d --- /dev/null +++ b/contrib/deneva/unified_concurrency_control/row_prece.h @@ -0,0 +1,358 @@ +/* Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include "util.h" +#include "../../../src/3ts/backend/cca/anomaly_type.h" +#include "../../../src/3ts/backend/cca/prece_type.h" +#include "operation_type.h" + +namespace ttts { + +class TxnNode; + +inline std::pair DividePreceType(const PreceType prece) +{ + if (prece == PreceType::WR) { + return {OperationType::W, OperationType::R}; + } else if (prece == PreceType::WCR) { + return {OperationType::C, OperationType::R}; + } else if (prece == PreceType::WW) { + return {OperationType::W, OperationType::W}; + } else if (prece == PreceType::WCW) { + return {OperationType::C, OperationType::W}; + } else if (prece == PreceType::RW) { + return {OperationType::R, OperationType::W}; + } else { + assert(false); + return {}; + } +} + +inline PreceType MergeOperationType(const OperationType o1, const OperationType o2) +{ + if (o1 == OperationType::W && o2 == OperationType::R) { + return PreceType::WR; + } else if (o1 == OperationType::C && o2 == OperationType::R) { + return PreceType::WCR; + } else if (o1 == OperationType::W && o2 == OperationType::W) { + return PreceType::WW; + } else if (o1 == OperationType::C && o2 == OperationType::W) { + return PreceType::WCW; + } else if (o1 == OperationType::R && o2 == OperationType::W) { + return PreceType::RW; + } else { + assert(false); + return {}; + } +} + +class PreceInfo { + public: + inline PreceInfo(std::shared_ptr from_txn, std::shared_ptr to_txn, const PreceType type, + const uint64_t row_id, const uint64_t from_ver_id, const uint64_t to_ver_id); + PreceInfo(const PreceInfo&) = default; + PreceInfo(PreceInfo&&) = default; + + friend std::ostream& operator<<(std::ostream& os, const PreceInfo prece) { + return os << 'T' << prece.from_txn_id() << "--" << prece.type_ << "(row=" << prece.row_id_ << ")->T" << prece.to_txn_id(); + } + + uint64_t from_txn_id() const { return from_txn_id_; } + inline uint64_t to_txn_id() const; + uint64_t from_ver_id() const { return from_ver_id_; } + uint64_t to_ver_id() const { return to_ver_id_; } + OperationType from_op_type() const { return DividePreceType(type_).first; } + OperationType to_op_type() const { return DividePreceType(type_).second; } + uint64_t row_id() const { return row_id_; } + PreceType type() const { return type_; } + std::shared_ptr from_txn() const { return from_txn_.lock(); } + std::shared_ptr to_txn() const { return to_txn_; } + + private: + const uint64_t from_txn_id_; + const std::weak_ptr from_txn_; + const std::shared_ptr to_txn_; // release condition (2) for TxnNode + const PreceType type_; + const uint64_t row_id_; + const uint64_t from_ver_id_; + const uint64_t to_ver_id_; +}; + +// A TxnNode can be released only when no more transactions build precedence before it. In this case, the +// transaction cannot be a part of cycle anymore. +// +// For latest read, it should satisfies: +// (1) The transaction is already finished (for latest read, no future transactions build RW precedence before +// it). +// (2) There are no transactions built precedence before it. +// We use std::shared_ptr to realize it. When the pointer expired, the two conditions are satisified. +// +// For snapshot read, it should also satisfies: +// (3) Minimum active transaction snapshot timestamp (start timestamp) > this transaction's largest write +// timestamp (commit timestamp). (no future transactions build RW precedence before it) +class TxnNode : public std::enable_shared_from_this +{ + public: + enum class State { ACTIVE, PREPARING, COMMITTED, ABORTED }; + + TxnNode(const uint64_t txn_id) : txn_id_(txn_id), state_(State::ACTIVE) {} + + // We use ver_id but not the count of operation to support snapshot read. + // [Note] Should ensure to_txn_node thread safe. + template + void AddToTxn(std::shared_ptr to_txn_node, const uint64_t row_id, const uint64_t from_ver_id, + const uint64_t to_ver_id) + { + const uint64_t to_txn_id = to_txn_node->txn_id(); + if (const auto& type = RealPreceType_(); txn_id_ != to_txn_id && type.has_value()) { + std::lock_guard l(m_); + std::shared_ptr prece = std::make_shared(shared_from_this(), to_txn_node, + *type, row_id, from_ver_id, to_ver_id); + // For read/write precedence, only record the first precedence between the two transactions + to_preces_.try_emplace(to_txn_id, prece); + to_txn_node->from_preces_.try_emplace(txn_id_, prece); + // For dirty precedence, W1W2 has higher priority than W1R2 because W1R2C1 is not dirty read + if (type == PreceType::WR || type == PreceType::WW) { + dirty_to_preces_.emplace_back(prece); + } + } + } + + uint64_t txn_id() const { return txn_id_; } + + const auto& UnsafeGetToPreces() const { return to_preces_; } + const auto& UnsafeGetFromPreces() const { return from_preces_; } + const auto& UnsafeGetDirtyToPreces() const { return dirty_to_preces_; } + + std::mutex& mutex() const { return m_; } + + void Commit() + { + std::lock_guard l(m_); + state_ = State::COMMITTED; + dirty_to_preces_.clear(); + } + + void Abort(const bool clear_to_preces) + { + std::lock_guard l(m_); + state_ = State::ABORTED; + dirty_to_preces_.clear(); + if (clear_to_preces) { + to_preces_.clear(); + } + } + + const auto& state() const { return state_; } + auto& state() { return state_; } + + friend std::ostream& operator<<(std::ostream& os, const TxnNode& txn) + { + os << "[Leak Txn] T" << txn.txn_id(); + std::scoped_lock l(txn.mutex()); + os << " [From Txns] T"; + for (const auto& [from_txn_id, from_prece_weak] : txn.UnsafeGetFromPreces()) { + if (const auto from_prece = from_prece_weak.lock()) { + os << from_txn_id << "(" << *from_prece << ") "; + } else { + os << from_txn_id << "(null prece) "; + } + } + os << " [To Txns] T"; + for (const auto& [to_txn_id, to_prece] : txn.UnsafeGetToPreces()) { + os << to_txn_id << "(" << *to_prece << ") "; + } + return os; + } + + private: + std::optional RealPreceType_(const std::optional& active_prece_type, + const std::optional& committed_prece_type, + const std::optional& aborted_prece_type) const + { + const auto state = state_.load(); + if (state == State::ACTIVE || state == State::PREPARING) { + return active_prece_type; + } else if (state == State::COMMITTED) { + return committed_prece_type; + } else { + assert(state == State::ABORTED); + return aborted_prece_type; + } + } + + template > + std::optional RealPreceType_() const + { + if constexpr (TYPE == PreceType::WW) { + return RealPreceType_(PreceType::WW, PreceType::WCW, {}); + } else if constexpr (TYPE == PreceType::WR) { + return RealPreceType_(PreceType::WR, PreceType::WCR, {}); + } else { + return RealPreceType_(TYPE, TYPE, {}); + } + } + + mutable std::mutex m_; + uint64_t txn_id_; + std::atomic state_; + // Key is txn_id. + std::unordered_map> to_preces_; + // We can not use std::shared_ptr because PreceInfo has a use count to this which will cause cycle + // reference. Key is txn_id; + std::unordered_map> from_preces_; + // The prece may also stored in to_preces_. + std::vector> dirty_to_preces_; +}; + +inline uint64_t PreceInfo::to_txn_id() const { return to_txn_->txn_id(); } +inline PreceInfo::PreceInfo(std::shared_ptr from_txn, std::shared_ptr to_txn, + const PreceType type, const uint64_t row_id, const uint64_t from_ver_id, const uint64_t to_ver_id) + : from_txn_id_(from_txn->txn_id()), from_txn_(std::move(from_txn)), to_txn_(std::move(to_txn)), type_(type), row_id_(row_id), + from_ver_id_(from_ver_id), to_ver_id_(to_ver_id) {} +// Not thread safe. Protected by RowManager::m_ +// +// A VersionInfo can be released when it will not be read anymore. +// +// For latest read, it should satisfies: +// (1) It is not the latest version. +// (2) The later version's write transaction is finished. (to prevent version revoke) +// +// For snapshot read, it should also satisfies: +// (3) Minimum active transaction snapshot timestamp (start timestamp) > the later version's write timestamp. +template +class VersionInfo +{ + public: + VersionInfo(std::weak_ptr w_txn, Data&& data, const uint64_t ver_id) + : w_txn_(std::move(w_txn)), data_(std::move(data)), ver_id_(ver_id) {} + VersionInfo(const VersionInfo&) = default; + VersionInfo(VersionInfo&&) = default; + ~VersionInfo() {} + + std::shared_ptr w_txn() const { return w_txn_.lock(); } + + template + void foreach_r_txn(Task&& task) const + { + for (const auto& r_txn_weak : r_txns_) { + if (const auto r_txn = r_txn_weak.lock()) { + task(*r_txn); + } + } + } + + void add_r_txn(std::weak_ptr txn) { r_txns_.emplace_back(std::move(txn)); } + + const Data& data() const { return data_; } + + uint64_t ver_id() const { return ver_id_; } + + private: + const std::weak_ptr w_txn_; + Data const data_; + // There may be two versions on same the row which have the same ver_id due to version revoking + const uint64_t ver_id_; + std::vector> r_txns_; +}; + +template +class RowManager> +{ + public: + using Txn = TxnManager; + + RowManager(const uint64_t row_id, Data init_data) + : row_id_(row_id), cur_ver_id_(0), + latest_version_(std::make_shared>(std::weak_ptr(), std::move(init_data), + 0)) + {} + + RowManager() {} + std::optional Read(Txn& txn) + { + std::lock_guard l(m_); + const uint64_t to_ver_id = latest_version_->ver_id(); + build_prece_from_w_txn_(*latest_version_, txn, to_ver_id); + latest_version_->add_r_txn(txn.node_); + return latest_version_->data(); + } + + bool Prewrite(Data data, Txn& txn) + { + std::lock_guard l(m_); + const uint64_t to_ver_id = (++cur_ver_id_); + const auto pre_version = std::exchange(latest_version_, + std::make_shared>(txn.node_, std::move(data), + to_ver_id)); + build_prece_from_w_txn_(*pre_version, txn, to_ver_id); + build_preces_from_r_txns_(*pre_version, txn, to_ver_id); + // If transaction writes multiple versions for the same row, only record the first pre_version + txn.pre_versions_.emplace(row_id_, std::move(pre_version)); + return true; + } + + void Write(Data data, Txn& txn) + { + // do nothing + } + + void Revoke(Data data, Txn& txn) + { + std::lock_guard l(m_); + const auto it = txn.pre_versions_.find(row_id_); + assert(it != txn.pre_versions_.end()); + latest_version_ = it->second; // revoke version + } + + private: + template + void build_prece_from_w_txn_(VersionInfo& version, const Txn& to_txn, + const uint64_t to_ver_id) const + { + if (const std::shared_ptr from_txn = version.w_txn()) { + build_prece(*from_txn, to_txn, version.ver_id(), to_ver_id); + } + } + + template + void build_preces_from_r_txns_(VersionInfo& version, const Txn& to_txn, + const uint64_t to_ver_id) const + { + version.foreach_r_txn([&to_txn, to_ver_id, from_ver_id = version.ver_id(), this](TxnNode& from_txn) { + build_prece(from_txn, to_txn, from_ver_id, to_ver_id); + }); + } + + template + void build_prece(TxnNode& from_txn, const Txn& to_txn, const uint64_t from_ver_id, + const uint64_t to_ver_id) const + { + from_txn.AddToTxn(to_txn.node_, row_id_, from_ver_id, to_ver_id); + } + + private: + const uint64_t row_id_; + std::mutex m_; + uint64_t cur_ver_id_; + //std::map> versions_; // key is write timestamp (snapshot read) + std::shared_ptr> latest_version_; +}; + +} diff --git a/contrib/deneva/unified_concurrency_control/row_ssi.h b/contrib/deneva/unified_concurrency_control/row_ssi.h new file mode 100644 index 00000000..9991d0f0 --- /dev/null +++ b/contrib/deneva/unified_concurrency_control/row_ssi.h @@ -0,0 +1,89 @@ +/* Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ + +#pragma once + +#include "util.h" +#include +#include +#include + +template +class TxnNode> +{ + public: + TxnNode() : txn_id_(txn_id), start_time_(start_time), state_(State::ACTIVE), w_side_conf_(false), + r_side_conf_(false) {} + + private: + const uint64_t txn_id_; + const uint64_t start_ts_; + std::atomic state_; + uint64_t commit_ts_; + std::atomic w_side_conf_; + std::atomic r_side_conf_; +}; + +template +class Version +{ + + const std::weak_ptr w_txn_; + const uint64_t w_ts_; + Data const data_; + std::vector> r_txns_; +}; + +template +class RowManager> +{ + public: + using Txn = TxnManager; + + RowManager(const uint64_t row_id, Data init_data) {} + + std::optional Read(Txn& txn) + { + std::lock_guard l(m_); + const auto& [latest_w_ts, latest_version] = versions_.back(); + if (latest_w_ts > l) + } + + bool Prewrite(Data data, Txn& txn) + { + std::lock_guard l(m_); + assert(!version_.empty()); + const auto& [latest_w_ts, latest_version] = versions_.back(); + if (latest_w_ts > txn.start_ts_ || w_lock_txn_ != nullptr) { // prevent S1W2 precedence + return false; + } + version.foreach_r_txn([](TxnNode& from_txn) { + if (from_txn.state() == from_txnNode::State::ACTIVE || + from_txn.state() == from_txnNode::State::PREPARING || + (from_txn.state() == from_txnNode::State::COMMITTED && from_txn.commit_ts_ > txn.start_ts_)) { + build_preces_from_r_txns_(from_txn, txn); + } + }); + + } + + void Write(Data _, Txn& txn) + { + } + + void Revoke(Data _, Txn& txn) + { + // do nothing + } + + private: + const uint64_t row_id_; + std::mutex m_; + std::deque>> versions_; // first is w_ts + std::shared_ptr w_lock_txn_; +}; diff --git a/contrib/deneva/unified_concurrency_control/txn_dli_identify.h b/contrib/deneva/unified_concurrency_control/txn_dli_identify.h new file mode 100644 index 00000000..5f721379 --- /dev/null +++ b/contrib/deneva/unified_concurrency_control/txn_dli_identify.h @@ -0,0 +1,35 @@ +/* Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ + +#pragma once + +#include +#include +#include +#include +#include "util.h" +#include "dli_identify_util.h" + +namespace ttts { + +template +class TxnManager> +{ + public: + TxnManager(const uint64_t txn_id) : node_(std::make_shared(txn_id)) {} + TxnManager() {} + const uint64_t txn_id() const { return node_->txn_id(); } + + std::unique_lock l_; + std::shared_ptr node_; // release condition (1) for TxnNode + std::unique_ptr cycle_; + std::unordered_map>> pre_versions_; // record for revoke +}; + +} diff --git a/contrib/deneva/unified_concurrency_control/txn_ssi.h b/contrib/deneva/unified_concurrency_control/txn_ssi.h new file mode 100644 index 00000000..671054e4 --- /dev/null +++ b/contrib/deneva/unified_concurrency_control/txn_ssi.h @@ -0,0 +1,32 @@ +/* Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ + +#pragma once + +#include +#include +#include +#include +#include "util.h" + +namespace ttts { + + +template +class TxnManager> +{ + public: + enum class State { ACTIVE, PREPARING, COMMITTED, ABORTED }; + + TxnManager(const uint64_t txn_id, const uint64_t start_time) : node_(std::make_shared(txn_id)) {} + + private: + std::shared_ptr> node_; +}; + +} diff --git a/contrib/deneva/unified_concurrency_control/util.h b/contrib/deneva/unified_concurrency_control/util.h new file mode 100644 index 00000000..a3fd1164 --- /dev/null +++ b/contrib/deneva/unified_concurrency_control/util.h @@ -0,0 +1,37 @@ +/* Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ + +#ifdef ENUM_BEGIN +#ifdef ENUM_MEMBER +#ifdef ENUM_END + +ENUM_BEGIN(UniAlgs) +ENUM_MEMBER(UniAlgs, UNI_DLI_IDENTIFY_CHAIN) +ENUM_MEMBER(UniAlgs, UNI_DLI_IDENTIFY_CYCLE) +ENUM_MEMBER(UniAlgs, ALL) +ENUM_END(UniAlgs) + +#endif +#endif +#endif + +#ifndef TTTS_DENEVA_UNI_H_ +#define TTTS_DENEVA_UNI_H_ + +namespace ttts { + +#define ENUM_FILE "../../../../contrib/deneva/unified_concurrency_control/util.h" +#include "../../../src/3ts/backend/util/extend_enum.h" + +template class AlgManager; +template class RowManager; +template class TxnManager; + +} + +#endif // TTTS_DENEVA_UNI_ALGS_H_ diff --git a/doc/en/3TS Distributed Transaction Processing Framework--Part I--V3.md b/doc/en/3TS Distributed Transaction Processing Framework--Part I--V3.md new file mode 100644 index 00000000..be43c91a --- /dev/null +++ b/doc/en/3TS Distributed Transaction Processing Framework--Part I--V3.md @@ -0,0 +1,246 @@ +![logo](../../assets/logo.png) + +# 3TS Tencent Transaction Processing Testbed System(Part I) + +###### Author: Haixiang Li, Translator: Jingyao Li + +​ Tencent Transaction Processing Testbed System (3TS) is a **database transaction processing-oriented testbed system** jointly developed by the TDSQL team of Tencent and the Key Laboratory of Data Engineering and Knowledge Engineering of Renmin University of China. The system aims to facilitate users’ quickly building of a new concurrency control algorithm through designing and building a unified framework for transaction processing(including distributed transactions), and through the accessible API provided by the framework. By using the testbed provided by the system, it is convenient for users to impartially compare the performance of mainstream concurrency control algorithms in the same test environment, and select an optimal concurrency control algorithm according to the needs of specific application scenarios. By far, the verification system has integrated 13 different kinds of mainstream concurrent control algorithms and common benchmarks such as TPC-C, Sysbench and YCSB. In addition, 3TS further provides a consistency level benchmark to determine the level of consistency and compare result of performance test in face of the difficult system selection problem created by the rapid development of distributed database system nowadays. + +​ 3TS system aims at exploring the theories and implementation technologies related to database transaction processing in depth, with core concepts as **openness, deepness and evolution**. Openness is to share knowledge and technology while insisting on open source coding. Deepness requires incisive study of the fundamental problem in transaction processing technology with a spirit of systematic lucubration. Evolution stands for endless pursuit and constant explore along the long way of ascetic practices. + +## 1. 3TS Overall Description + +​ As a framework of transaction processing technology, 3TS is committed to exploring essential issues including: + +1. How many kinds of data anomalies exist? How to establish systematic research method of data anomalies? +2. Why exist different kinds of concurrent access control algorithms? Among them, is there any essential correlation? +3. After the transactional database changes from single-machine to distributed, which aspects will be affected?(availability? Reliability? Security? Consistency? Scalability? Function? Performance? Architecture? . . . ) +4. What new technologies will affect and how will they affect distributed transactional database systems? +5. How should the evaluation system of distributed transactional database system be established? + +​ For each of above research problems, corresponding subsystems exist at code level. Such as 3TS-DA subsystem and 3TS-Deneva subsystem open sourced earlier. + +## 2. 3TS-DA, Data Anomaly Subsystem + +​ 3TS-DA Data anomaly subsystem, located in path src/3ts/da, has a project architecture as follows: + +![figure1_2](../images/3ts_desc_image/figure1_2.png) + +1. History Creator: responsible for generating history and exporting it to the algorithm for verification. +2. CCA Group: CCA(Concurrent access control algorithm) can perform anomaly detection on incoming history and return anomaly detection results. +3. Outputter: export anomaly detection results of current history obtained by different CCA to the output file. + +**Current functions of the 3TS-DA subsystem:** + +1. **Test data generation:** support three generation modes of history, including traversal generation, random generation and reading from text. +2. **Algorithm addition:** provides a unified algorithm interface, which can easily add new concurrent algorithms. At the same time, the framework itself has a variety of built-in algorithms, including: **Serializable, Conflict-serializable, SSI, WSI, BOCC, FOCC, etc.** +3. **Test indicators:** the framework provides a variety of test indicators, including: algorithm rollback rate, true rollback rate, fake rollback rate, execution time, etc. +4. **Anomaly extension:** The framework implements data anomaly extension algorithm, which can generate unlimited data anomaly histories for the test of the algorithm. + +## 3. 3TS-Deneva, Concurrent Algorithm Framework + +​ Devena[1] is a evaluation framework of distributed in-memory database concurrent access control algorithm open sourced by MIT, the original project can be found at https://github.com/mitdbg/deneva. It can study the performance characteristics of concurrency control protocol in a controlled environment. Mainstream algorithms such as **Maat, OCC, TO, Locking(No Wait, WaitDie), Calvin** are provided in this framework. 3TS-Deneva is Tencent's improvement on the original Deneva system, including improvement in multiple aspects. At the algorithm level, more concurrent access control algorithms are added, including: **Serializable, Conflict-serializable, SSI, WSI, BOCC, FOCC, Sundial, Silo, etc.** + +### 3.1 Infrastructure + +​ Devena uses a custom engine and some other settings to deploy and implement different concurrency control protocols on the platform, so as to make an assessment as fair as possible. The architecture of the system, as shown in Figure 1, mainly includes two modules: + +1. Client instance, which acts as the initiator of the transaction. Each thread in the client is responsible for initiating the transaction request, puts the initiated transaction request into the MsgQueue, and sends it to the server in sequence for specific execution. The client and server instances exhibit fully connected topology, and are generally deployed on different machines. + +2. Server instance, which performs various operations in the transaction. Data stored in different server instances is indexed using consistent hash, so as to form a global partition mapping table between server IP and stored data. By ensuring non-modifiability of partition mapping during test period, availability of this mapping table to all nodes is guaranteed. The communication between client and server, server and server instances use TCP/IP protocol. Each server instance can be internally subdivided into four modules: + + 1. Input message queue, which stores messages sent by clients/other servers temporarily. + 2. Execution engine, which allocates multiple worker threads to parse and execute messages in the message queue, using a resource scheduling method of binding one thread with one core. + 3. Concurrency control module. While a worker thread is executing transactions operations, it shall maintain information required by specific concurrency control protocol as well as conform to the procedure stipulated by the protocol, so as to guarantee the effectiveness of the specified concurrency control protocol. + 4. Data storage module, which is responsible for managing the data in this instance and keeping data in memory. + + ![figure1_3](../images/3ts_desc_image/figure1_3.png) + +​ In 3TS, Deneva is improved, with the improved code located in path contrib/deneva and its internal project architecture as follows: + +![figure1_4](../images/3ts_desc_image/figure1_4.png) + +- Deneva has two types of node: server and client. + +- Client generate workload query and send it to server, while server executes the workload query received from client in coordination. + +- Modules shared by client and server: + + ◼ MsgQueue: message queue, store the *msg* to send. + + ◼ MsgThread: message sending thread, take and send *msg* from MsgQueue in loop. + + ◼ InputThread: message receiving thread, receive *msg* from server/client. + +- Exclusive module owned by client: + + ◼ ClientQueryQueue: query queue in client, store query lists generated before test begin. + + ◼ ClientThread: take out *msg* from ClientQueryQueue in loop, and send it to server by MsgThread and MsgQueue. + +- Exclusive module owned by server: + + ◼ WorkQueue: message queue to be processed in server, stores *msg* received by InputThread. + + ◼ WorkThread: execution thread in server, take *msg* from WorkQueue and process. Generate return *msg* after execution, which will be send by MsgThread and MsgQueue as well. + +### 3.2 Transaction execution procedure + +​ In Deneva, as shown in Figure 3, the execution procedure of a transaction can be described as: + +1. Client initiates a transaction request composed of multiple operations, and put it in the ClientQueryQueue. ClientThread take out the transaction request from the queue and store it in the MsgQueue. Afterwards, one transaction request is taken out from the MsgQueue, encapsulated as a request and sent to its coordination server by the MsgThread, in which the coordinator is usually chosen as the server with the first visited data. +2. Server first parses a request after its arriving, and put it in the WorkQueue with all of its operations as one element. New transactions received from client as well as remote pending transactions which have already started exist in the WorkQueue, with the latter has higher priority than the former. The threads in the worker thread pool poll the work queue and process the operations in the transaction. When a worker thread processes the operation of current transaction, it first initializes the transaction, then executes read and write operations in sequence, and finally commits or rolls back. + 1. In the process of executing a transaction, there are two situations that will cause the transaction to wait. One is to wait for the release of the exclusive lock on a resource; The second is the need to access data in the remote server. When a WAIT is required for remote access to data in another server, the remote server will return a WAIT instruction to the current coordinator node. The coordinator node will temporarily store the WAIT state of the current transaction and schedule the current worker thread to execute other transactions, thus avoiding worker thread blocking. When a waiting transaction is ready to continue, based on a priority scheduling policy implemented by the work queue, it will be continued by the first available worker thread. + 2. Additional operations required by the concurrency control protocol will be embedded in the transaction execution process, including read and write operations, validation operations, commit/rollback operations, etc. +3. After the coordination node completes the operation of a certain transaction, it will put the transaction execution result into the MsgQueue, and then the MsgThread will inform the client of the execution result of current transaction. + +![figure1_5](../images/3ts_desc_image/figure1_5.png) + +## 4. Distributed Transaction Overview + +​ Reference [15] identifies distributed transaction as: + +![figure1_6](../images/3ts_desc_image/figure1_6.png) + +​ Distributed transactions base on distributed system to realize the semantic requirements of transaction processing, that is, they must also satisfy ACID characteristics on the distributed system. Therefore, distributed transaction processing of distributed database should also follow the transaction-related theory of stand-alone database system to ensure that each transaction meets the requirements of ACID. Distributed concurrent access control technology is adopted to deal with data anomalies in distributed system and realize distributed transaction’s ACID characteristics. + +​ Basic technology of distributed transaction processing mechanism, is build upon transaction processing mechanism in stand-alone database system. However, there are also some differences, such as how to deal with distributed data anomalies, how to achieve serializability in distributed architecture, how to achieve across-nodes atomic commit, how to shorten transaction response in the presence of network partition or high latency, etc. + +​ Serving as a distributed environment, all these will be implemented and verified in 3TS framework. + +## 5. Concurrent Access Control Algorithms Provided By 3TS + +​ By far, 3TS has integrated 13 different kinds of concurrent access control algorithms, mainly including the following: + +1. Two-phase locking protocol（2PL: No-Wait, Wait-Die） +2. Timestamp Ordering Protocol（T/O） +3. Multi-Version Concurrency Control Protocol（MVCC） +4. Optimistic Concurrency Control Protocol（OCC、FOCC、BOCC） +5. Optimized Optimistic Concurrency Control Protocol（MaaT、Sundial、Silo） +6. Deterministic Concurrency Control Protocol（Calvin） +7. Concurrency control protocol based on snapshot isolation（SSI、WSI） + +​ These concurrency control protocols are briefly described below: + +### 5.1 Two-phase locking protocol（2PL） + +​ Two-phase Locking protocol（2PL）is the most widely used concurrency control protocol at present. 2PL synchronizes conflicting operations between transactions by acquiring shared locks or exclusive locks when executing read or write operations. According to Bernstein and Goodman [2], 2PL has the following two rules for acquiring locks: 1) No conflicting locks can exist on the same data item at the same time; 2) After a transaction has firstly released a lock, it cannot acquire any locks afterwards. The second rule separates the locking phase of a transaction as two phase: growing phase and shrinking phase. During the growing phase, a transaction acquires locks for all records it needs to access. A shared lock is acquired by a read operation while a write operation acquires a mutex lock. While shared locks do not conflict with each other, conflict exists between mutex and shared locks or multiple mutex locks. Once the transaction releases either of the locks, it will enter the second phase known as shrinking phase in 2PL. After entering into shrinking phase, no new lock is allowed to be acquired for this transaction. + +​ 3TS implements the Strict 2PL protocol that does not release locks until the transaction commits or aborts. Depending on how to avoid deadlock, there are two implementations of 2PL in 3TS: 2PL (no-wait) and 2PL (wait-die). The implementations of these two protocols follow the descriptions in [2, 3]. + +#### 5.1.1 2PL（No-Wait） + +​ 2PL(No-Wait) protocol states that when a lock conflict occurs when a transaction tries to lock, the transaction that is currently requesting the lock shall rollback. Any locks held by the rollback transaction will be released, allowing other transactions to acquire the lock. The No_Wait mechanism avoids deadlocks by preventing ring among wait transaction. However, not every lock conflict results in deadlock, therefore the rollback rate can be comparatively high. + +#### 5.1.2 2PL（Wait-Die） + +​ Rosenkrantz[3] proposed 2PL(Wait-Die), prioritize transactions by their corresponding start timestamp, so as to ensure lock wait order is consistent with the timestamp order. As long as the transaction timestamp is smaller(older) than any transaction currently holding the lock, the current transaction waits. Otherwise, the current transaction needs to rollback. For any two conflicting transactions *Ti* and *Tj*, the protocol relays on the timestamp priority to decide whether *Ti* should wait for *Tj*. **If *Ti* has a lower priority (smaller timestamp), then *Ti* needs to wait, otherwise *Ti* should rollback. Therefore, no loop in the lock wait graph can exist, thus deadlock can be prevented.** 2PL(Wait-Die) is a combination of TO and locking technology. + +​ However, distributed transaction processing mechanisms based on the principles of 2PL either avoid deadlocks (high rollback rate) or need to solve the problem of deadlocks(resource and communication deadlocks). The cost of resolving deadlocks in a distributed system will be very high (the cost of resolving deadlocks on a single-machine system is already high, and modern database systems based on multi-process or multi-threaded architectures may cause the system to almost stop service. In MySQL version 5. 6 and 5. 7, the same data item is updated concurrently, and the deadlock detection operation may cause the system to almost stop service). + +​ Not only deadlock detection consumes huge resources, but also the disadvantages of the lock mechanism itself have been criticized. Reference [5] believes that the disadvantages of the locking mechanism are as follows (a clear understanding of these disadvantages prompted the author of reference [5] to design OCC, that is, Optimistic Concurrency Control): + +1. **The locking mechanism is expensive:** In order to ensure serializability, for read-only transactions that comply with the integrity constraints of the database, read locks needs to be added by the locking protocol to exclude potential concurrent write operations and prevent others from modifying. For locking protocol with possible deadlocks, overhead caused by mechanisms such as deadlock prevention/deadlock detection also needs to be endured. +2. **The locking mechanism is complex:** In order to avoid deadlocks, various complex locking protocols need to be customized (such as when to lock, when to release lock, how to guarantee strictness, etc. ). +3. **Reduce the concurrent throughput of the system:** + 1. If a transaction waiting for I/O operation holds locks, the overall concurrent throughput of the system will be greatly reduced. + 2. Locks must be held by the transaction until the end of transaction rollback, which will also reduce the overall concurrent throughput of the system. + +​ In addition, the use of the lock mechanism for mutually exclusion operations will cause time-consuming kernel-mode operations for the operating system and make the lock mechanism inefficient. This means that the 2PL technology with transaction processing semantics based on the lock mechanism of the operating system is even more unusable (nevertheless, there are also some technologies that are constantly improving the concurrent access control algorithm based on locking protocol). + +### 5.2 Timestamp ordering protocol（T/O） + +​ Timestamp Ordering(T/O) protocol allocates a timestamp at the beginning of a transaction, and sorts the transactions in the order of the timestamp[2]. When the execution of an operation violates the order that has been specified between transactions, corresponding transaction to current operation will rollback or enter a waiting state. + +​ The implementation of the T/O algorithm in 3TS follows the description in Section 4.1 in [2]. You can refer to this document for more details. The following figure shows the implementation of a basic T/O algorithm. + +![figure1_7](../images/3ts_desc_image/figure1_7.png) + +### 5.3 Multi-version concurrency control protocol（MVCC） + +​ Multi-Version Concurrency Control(MVCC) is a concurrent access control technology commonly used in current database management systems. It was first proposed by David Patrick Reed [4] in 1978. The main idea is to expand a logical data item into multiple physical versions, and transaction’s operation on data item is converted to operation on physical versions, thereby improving the concurrency rate of transaction processing, as well as providing the ability to read and write without blocking each other. + +​ The MVCC technology was firstly proposed in 1970, and was further described in 1978 in "Naming and synchronization in a decentralized computer system"[4]. Later, reference [16] described the MVCC technology in detail in 1981, while the MVCC technology it described is based on timestamp. + +![figure1_8](../images/3ts_desc_image/figure1_8.png) + +​ After that, MVCC technology was widely used with multiple versions been developed. + +​ In 2008, reference [13] was published, achieving serializable isolation level based on MVCC by proposing “Serializable Snapshot Isolation” technology. Which makes PostgreSQL V9.1 achieves serializable isolation level by implementing this technology. + +​ In 2012, reference [14] was published, and proposed the “Write-Snapshot Isolation” technology to achieve a serializable isolation level based on MVCC by verifying read-write conflicts. Compared with the method of detecting write-write conflicts, it improves the concurrency rate(a certain kind of write-write conflict can be serializable). The author of this article made a system implementation based on HBase. + +​ In 2012, reference [17] implements SSI technology on PostgreSQL. This article not only tells the theoretical basis of serialized snapshot, PostgreSQL’s implementation technology of SSI, but also proposed important topics such as "Safe Snapshots" and "Deferable Transaction" to support read only transactions, “Safe Retry” strategy to avoid potential effect caused by transaction rollback after read-write conflict, as well as 2PC’s influence on choosing rollback transaction after read-write conflict. + +​ Reference [19] systematically discussed the four aspects involved in MVCC technology, namely: concurrent access control protocol, multi-version storage, old version garbage collection, and index management. In addition, the principle of various variants of MVCC was discussed, and the effect of each variant ( MV2PL , MVOCC, MVTO, etc. ) was tested and evaluated on OLTP workload after their implementation. [18] discussed the old version garbage collection problem of MVCC in detail. + + + +​ In 3TS, MVCC is realized based on the description in Section 4. 3 in [2], combined with T/O algorithm. Therefore, the transaction still uses the start timestamp for sorting. Unlike the traditional T/O algorithm, MVCC uses the feature of multiple versions to reduce the operation waiting overhead in T/O. The operation execution mechanism in MVCC is as follows (use ts to represent the timestamp of the current transaction): + +1. Read operation + 1. If *ts* is greater than the timestamp of all transactions in *prereq*, and there is a version in *writehis* (the version chain of current data item) that its *wts* is greater than *ts* and less than *pts*, then the current version can be returned, and the timestamp of the current transaction can be stored into *readhis*. If there is no such timestamp in *writehis*, the timestamp of the current transaction is stored in *readreq*. The main reasons are: + 1. If there is a committed write between the pre-write transaction and the read operation, it means that the data read by the current read operation has been written by the write transaction. As the timestamp order is satisfied, the version can be read; + 2. If the timestamp of the read operation is larger than the timestamp of the current unfinished write transaction, new data should be read, so wait is needed; + 2. Otherwise, the current read operation reads the latest visible version to its timestamp, and stores the timestamp of current transaction in *readreq*. +2. Write operation + 1. If *ts* is less than the timestamp of all transactions in *readhis*, and there is a timestamp in *writehis* between *rts* and *ts*, data can be pre-written normally. If there is no such timestamp in *writehis*, then current transaction will rollback. + 2. Temporarily store the current write operation in *prereq_mvcc*; +3. Submit operation + 1. Insert the current transaction timestamp and the new version written into *writehis*; + 2. Delete the write operation of the current transaction from *prereq*; + 3. Continue to execute read transactions that comply with the timestamp ordering in *readreq*; + +More concurrent access control algorithms, to be continued. . . + +## Acknowledgement + +​ Special thanks to the Tencent TDSQL team and the Key Laboratory of Data Engineering and Knowledge Engineering of Renmin University of China for their support and help in this work, and thank Zhao Zhanhao, Liu Chang, Zhao Hongyao and other students for their contributions to this article. + +## Reference + +[1] Rachael Harding, Dana Van Aken, Andrew Pavlo, Michael Stonebraker: An Evaluation of Distributed Concurrency Control. Proc. VLDB Endow. 10(5): 553-564 (2017) + +[2] Philip A. Bernstein, Nathan Goodman: Concurrency Control in Distributed Database Systems. ACM Comput. Surv. 13(2): 185-221 (1981) + +[3] Daniel J. Rosenkrantz, Richard Edwin Stearns, Philip M. Lewis II: System Level Concurrency Control for Distributed Database Systems. ACM Trans. Database Syst. 3(2): 178-198 (1978) + +[4] D. P. Reed. Naming and synchronization in a decentralized computer system. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, USA, 1978. + +[5] H. T. Kung, John T. Robinson: On Optimistic Methods for Concurrency Control. ACM Trans. Database Syst. 6(2): 213-226 (1981) + +[6] Theo Härder: Observations on optimistic concurrency control schemes. Inf. Syst. 9(2): 111-120 (1984) + +[7] Hatem A. Mahmoud, Vaibhav Arora, Faisal Nawab, Divyakant Agrawal, Amr El Abbadi: MaaT: Effective and scalable coordination of distributed transactions in the cloud. Proc. VLDB Endow. 7(5): 329-340 (2014) + +[8] Xiangyao Yu, Yu Xia, Andrew Pavlo, Daniel Sánchez, Larry Rudolph, Srinivas Devadas: Sundial: Harmonizing Concurrency Control and Caching in a Distributed OLTP Database Management System. Proc. VLDB Endow. 11(10): 1289-1302 (2018) + +[9] Stephen Tu, Wenting Zheng, Eddie Kohler, Barbara Liskov, Samuel Madden: Speedy transactions in multicore in-memory databases. SOSP 2013: 18-32 + +[10] Alexander Thomson, Thaddeus Diamond, Shu-Chun Weng, Kun Ren, Philip Shao, Daniel J. Abadi: Calvin: fast distributed transactions for partitioned database systems. SIGMOD Conference 2012: 1-12 + +[11] Hal Berenson, Philip A. Bernstein, Jim Gray, Jim Melton, Elizabeth J. O'Neil, Patrick E. O'Neil: A Critique of ANSI SQL Isolation Levels. SIGMOD Conference 1995: 1-10 + +[12] Alan D. Fekete, Dimitrios Liarokapis, Elizabeth J. O'Neil, Patrick E. O'Neil, Dennis E. Shasha: Making snapshot isolation serializable. ACM Trans. Database Syst. 30(2): 492-528 (2005) + +[13] Michael J. Cahill, Uwe Röhm, Alan D. Fekete: Serializable isolation for snapshot databases. SIGMOD Conference 2008: 729-738 + +[14] Maysam Yabandeh, Daniel Gómez Ferro: A critique of snapshot isolation. EuroSys 2012: 155-168 + +[15] https://en.wikipedia.org/wiki/Distributed_transaction + +[16] P. Bernstein, V. Hadzilacos, and N. Goodman. Concurrency Control and Recovery in Database Systems. Addison–Wesley, 1987. + +[17] D. R. Ports and K. Grittner, “Serializable snapshot isolation in postgresql, ” PVLDB, vol. 5, no. 12, pp. 1850–1861, 2012. + +[18] J. Böttcher, et al. , Scalable Garbage Collection for In-Memory MVCC Systems, in VLDB, 2019 + +[19] Yingjun Wu, Joy Arulraj, Jiexi Lin, Ran Xian, Andrew Pavlo:An Empirical Evaluation of In-Memory Multi-Version Concurrency Control. Proc. VLDB Endow. 10(7): 781-792 (2017) + +[20] D. Lomet and M. F. Mokbel, “Locking key ranges with unbundled transaction services, ” VLDB, pp. 265–276, 2009. + +[21] Jinwei Guo, Peng Cai, Jiahao Wang, Weining Qian, Aoying Zhou: Adaptive Optimistic Concurrency Control for Heterogeneous Workloads. PVLDB 12(5): 584-596 (2019) + +[22] X. Yu, A. avlo, D. Sanchez, and S. Devadas, “Tictoc: Time traveling optimistic concurrency control, ” in Proceedings of SIGMOD, vol. 8, 2016, pp. 209–220. + +[23] Rudolf Bayer, Klaus Elhardt, Johannes Heigert, Angelika Reiser:Dynamic Timestamp Allocation for Transactions in Database Systems. DDB 1982: 9-20. \ No newline at end of file diff --git a/doc/en/3TS Distributed Transaction Processing Framework--Part II--V3.md b/doc/en/3TS Distributed Transaction Processing Framework--Part II--V3.md new file mode 100644 index 00000000..c9e728e6 --- /dev/null +++ b/doc/en/3TS Distributed Transaction Processing Framework--Part II--V3.md @@ -0,0 +1,250 @@ +![logo](../../assets/logo.png) + +# 3TS Tencent Transaction Processing Testbed System(Part II) + +###### Author: Haixiang Li, Translator: Jingyao Li + +​ Tencent Transaction Processing Testbed System (3TS) is a **database transaction processing-oriented testbed system** jointly developed by the TDSQL team of Tencent and the Key Laboratory of Data Engineering and Knowledge Engineering of Renmin University of China. The system aims to facilitate users’ quickly building of a new concurrency control algorithm through designing and building a unified framework for transaction processing(including distributed transactions), and through the accessible API provided by the framework. By using the testbed provided by the system, it is convenient for users to impartially compare the performance of mainstream concurrency control algorithms in the same test environment, and select an optimal concurrency control algorithm according to the needs of specific application scenarios. By far, the validation system has integrated 13 different kinds of mainstream concurrent control algorithms and common benchmarks such as TPC-C, Sysbench and YCSB. In addition, 3TS further provides a consistency level benchmark to determine the level of consistency and compare result of performance test in face of the difficult system selection problem created by the rapid development of distributed database system nowadays. + +​ 3TS system aims at exploring the theories and implementation technologies related to database transaction processing in depth, with core concepts as **openness, deepness and evolution**. Openness is to share knowledge and technology while insisting on open source coding. Deepness requires incisive study of the fundamental problem in transaction processing technology with a spirit of systematic lucubration. Evolution stands for endless pursuit and constant explore along the long way of ascetic practices. + +## 5. Concurrent Access Control Algorithm Provided By 3TS + +​ Part I introduced the framework and basic content of 3TS. This section continues to introduce a variety of concurrent access control algorithms in depth. + +### 5.4 Optimistic concurrency control protocol（OCC、FOCC、BOCC） + +​ Under the optimistic concurrency control protocol, the execution procedure of transaction is divided into three phases: read, verify, and write[5], as shown in Figure 5. + +![figure2_1](../images/3ts_desc_image/figure2_1.png) + +​ The advantages brought by the division of these three phases are obvious: + +1. **High transaction processing performance:** The improvement of transaction efficiency is mainly ensured by the non-blocking of reading and writing in the first phase, which greatly improves the concurrency and fully utilizes multi-core hardware resources. It is friendly especially to read-only transactions because of non-blocking read. + +2. **Avoidance of deadlock:** OCC can avoid deadlock by sorting the read and write objects in the first stage and locking in order in the second stage. This wins in comparison with the locking based concurrent access control algorithms in solving the deadlock problem. These two advantages enable OCC to still support high transaction throughput in scenarios such as distributed transactions, high data hotspots, and high communication delays. There is no obvious system performance jitter phenomenon in high concurrency scenarios (reference [169] shows through experiments that OCC algorithm performance is not high under high competition). + +3. **The correctness of data consistency is guaranteed:** the correctness is guaranteed in the validation phase. A directed graph is constructed through the transaction conflict relationship. Through loop detection and transaction rollback, loop-free is ensured to resolve transaction conflicts, while write-write conflicts are prevented through the locking mechanism in validation phase. However, there are different ways to implement in practice. For example, reference [9] improves the OCC algorithm. In the validation phase, it checks the read set of current transaction and rollback if modification by other concurrent transactions is found to avoid data inconsistency. In this way, construction of directed graph and detection of loop is no longer needed. + + + +​ In 3TS, according to different validation mechanisms, three different optimistic concurrency control protocols are implemented: (1) OCC: implementation of Parallel Validation algorithm in [5]; (2) BOCC: implementation of Backward Validation algorithm in [6]; (3) FOCC: implementation of Forward Validation algorithm in [6]. It should be noted that in 3TS, since there is no global timestamp mechanism (global clock will be added according to subsequent plan), the read and write sets that need to be compared in the validation phase may deviate due to clock synchronization, which may result in different degree of efficiency influence concerning different algorithms. + +1. The three protocols deal with the read phase in the same way, mainly: + 1. For read operation, first saved into read set, then reads data as required. + 2. For write operation, saved into write set. +2. When it comes to the validation phase, the main idea of all three protocols is to sort transactions according to the order in which they entering the validation phase, then read-write set is checked to ensure the processing results of these transactions meet this order. However, there is a difference in read-write set checking method concerning different protocol. + +#### 5.4.1 OCC + +​ The main procedure of the validation phase is (denote the start timestamp of current transaction to be validated as *start_ts*, the timestamp acquires when entering the validation phase as *finish_ts*): + +- Obtain the set of transactions committed in the time period (*start_ts*, *finish_ts*], record it as *History*, traverse the write set of transactions in *History*, if there is an intersection with the read set of current transaction, then the validation fails; +- Obtain the set of transactions currently in validation phase, record it as *Active*, traverse the write set of transactions in *Active*, if there is an intersection with the read set of current transaction, then the validation fails; + +#### 5.4.2 BOCC + +​ The validation phase and the write phase are required to be executed in the same critical region. The process is to obtain the set of transactions submitted in the time period (*start_ts*, *finish_ts*] and record it as *History*. Traverse the write set of transactions in *History*, if there is an intersection with the read set of current transaction, then the validation fails. BOCC has obvious disadvantages, including read-only transactions also need to be validated, large read set of current transaction to be validated has a great impact on validation efficiency, and a large number of write set of committed transactions will need to be retained for long transactions, etc. + +#### 5.4.3 FOCC + +​ The validation phase and the write phase are required to be executed in the same critical region. Traverse the read set of transactions currently in read phase, if there is an intersection with the write set of current transaction, then the validation fails. Compared with BOCC, FOCC has the advantages that read-only transactions can skip the validation phase, and the cost of validation for active transactions is less. + +3. The three protocols deal with the write phase in the same way, mainly: get the submission timestamp, write data in the write set into the database, and set the submission timestamp of the data to the obtained submission timestamp. + +### 5.5 Optimized optimistic concurrency control protocol（MaaT、Sundial、Silo） + +​ Traditional optimistic concurrency control protocols determine whether a transaction can be submitted according to the order of validation. Compared with traditional OCC, some optimized optimistic concurrency control protocols relax this requirement and reduce unnecessary rollbacks. Now, there are many improved versions based on OCC, such as ROCC [20], adaptive OCC [21] and so on. + +​ In 3TS, we have integrated three state of the art optimistic concurrency control algorithms, including MaaT, Sunidal and Silo. More concurrent algorithms are expected to be integrated into 3TS. + +#### 5.5.1 MaaT + +​ MaaT [6] uses a dynamic time stamp range adjustment method to reduce the transaction rollback rate. The main idea is to determine the sequence of transactions through the relationship formed in read and write operations among transactions, thereby determining the sequence of transactions in the equivalent serialized sequence. For example, if transaction *Tj* update *x* after transaction *Ti*’s read of *x*, then *Ti* needs to be ranked before *Tj*. + +​ MaaT needs to maintain additional metadata on each data item, including: (1) ID set of Transactions that have read the data item but not yet committed, denote as readers; (2)ID set of Transactions that are about to write the data item but not yet committed, denote as writers; (3)The largest commit timestamp of transactions that has read the data item, record as *Rts*; (4)The largest commit timestamp of transactions that has written the data item, record as *wts*; Each transaction has a timestamp range *[lower, upper)*, and is initialized to *[0, +)*. The procedure of each operation in the transaction mainly includes: + +1. Read operation + 1. Save the write transaction list of the data item to the *uncommitted_writes* of the transaction; + 2. Update current transaction: + *greatest_write_timestamp = Max{greatest_write_timestamp, wts}*; + 3. Write the current transaction ID into the read transaction list of the read data item; + 4. Read required data item and store the read data in the read set. +2. Write operation + 1. Save the write transaction list of the data item to the *uncommitted_writes_y* of the transaction. + 2. Save the read transaction list of the data item to the *uncommitted_reads* of the transaction. + 3. Update current transaction: + *greatest_write_timestamp=Max{greatest_write_timestamp, wts}*, *greatest_read_timestamp=Max{greatest_red_timestamp, rts}*; + 4. Write the current transaction ID into the write transaction list of the data item to be written; + 5. Save the new value of the data item to be written into the write set; +3. Validation phase(the transaction coordinator determines *lower* and *upper* based on the intersection of *lower* and *upper* returned by all participants, the following operations are all performed on participants) + 1. Update *lower=Max{greatest_write_timestamp+1, lower}*； + 2. Ensure that the *lower* of all transactions in *uncommitted_writes* is greater than the *upper* of current transaction; + 1. If a transaction in *uncommitted_writes* has validated successfully, modify the *upper* of current transaction; + 2. Otherwise, put the transaction in *uncommitted_writes* into the after queue of current transaction (transactions in the queue needs to be submitted after the current transaction); + 3. Update *lower=Max{greatest_read_timestamp+1, lower}*; + 4. Ensure that the *upper* of all transactions in *uncommitted_reads* is smaller than the *lower* of current transaction; + 1. If a transaction in *uncommitted_reads* has validated successfully, modify the *lower* of current transaction; + 2. Otherwise, put the transaction in *uncommitted_reads* into the before queue of current transaction (transactions in the queue needs to be submitted before the current transaction); + 5. Adjust the order between *uncommitted_writes_y* (list of uncommitted transactions with write-write conflicts) and current transaction; + 1. If a transaction in *uncommitted_writes_y* has validated successfully, modify the *lower* of current transaction so that it is greater than the *upper* of the validated transaction; + 2. Otherwise, put the transaction in *uncommitted_writes_y* into the after queue of current transaction; + 6. Check if true: *lower "; + std::string text = ""; + std::getline(std::cin, text); + if ("help" == text || "h" == text) { + Printer::PrintHelpInfo(); + } else if ("q" == text || "quit" == text) { + break; + } else if (text.find("\\d") != text.npos || text.find("definition") != text.npos) { + const auto index_space = text.find_first_of(" "); + if (index_space != text.npos) { + std::string input = text.substr(index_space); + Printer::TrimSpace(input); + if (printer.InfoMap().count(input) == 0) { + Printer::Print("Unknown Definition"); + } else { + Printer::Print(printer.InfoMap()[input]); + } + } else { + Printer::Print("Please check input format, such as \\d History"); + } + } else if (text.find("\\a") != text.npos || text.find("anomaly") != text.npos) { + const auto index_space = text.find_first_of(" "); + if (index_space != text.npos) { + std::string input = text.substr(index_space); + Printer::TrimSpace(input); + if (printer.AnomalyMap().count(input) == 0) { + Printer::Print("Unknown Anomaly"); + } else { + std::cout << "Type SubType History Example Definition" << std::endl; + std::cout << "-------------------------------------------------------------" << std::endl; + Printer::Print(printer.AnomalyMap()[input]); + } + } else { + Printer::Print("Please check input format, such as \\a Dirty Write"); + } + } else if (text.find("\\g") != text.npos || text.find("algorithm") != text.npos) { + const auto index_space = text.find_first_of(" "); + bool ret = true; + if (index_space != text.npos) { + std::string input = text.substr(index_space); + Printer::TrimSpace(input); + + std::vector alg_list; + std::vector alg_type_list; + Checker::split(input, alg_list, ","); + // Check the validity of the input + if (input.find("DLI_IDENTIFY_CYCLE") == input.npos && input.find("DLI_IDENTIFY_CHAIN") == input.npos) { + ret = false; + Printer::Print("Unknown Algorithm, please check algorithm's name"); + } + // fill and set alg_type_list + for (auto& alg : alg_list) { + if ("DLI_IDENTIFY_CYCLE" == alg) { + alg_type_list.emplace_back(ttts::UniAlgs::UNI_DLI_IDENTIFY_CYCLE); + } else if ("DLI_IDENTIFY_CHAIN" == alg) { + alg_type_list.emplace_back(ttts::UniAlgs::UNI_DLI_IDENTIFY_CHAIN); + } + } + printer.SetAlgs(alg_type_list); + // print set success info + if (ret) { + std::cout << "Set algorithm success" << std::endl; + } + } else { + Printer::Print("Please check input format, such as \\g DLI_IDENTIFY_CYCLE"); + } + } else if (text.find("\\t") != text.npos || text.find("table") != text.npos) { + std::vector anomaly_list = Printer::InitAnomalyList(); + draw(); + std::cout << "Please tell me how you feel now:" << std::endl; + std::string feel; + std::cin >> feel; + std::cout << "I knonw you are so " << feel << ", but i want to ask you a question:" << std::endl; + std::cout << "Do you want to join the TDSQL team? yes/no" << std::endl; + std::string y = "no"; + while ("yes" != y) { + std::cin >> y; + if ("yes" == y) { + break; + } else { + std::cout << "Please think it over before you answer!" << std::endl; + } + } + std::cout << "Welcome to our team! You will receive a book of martial arts secrets!" << std::endl; + std::cout << "Whether to get it now? yes/no" << std::endl; + std::string ret; + std::cin >> ret; + if ("yes" == ret) { + Printer::PrintAnomalyTableInfo(anomaly_list); + } else { + std::cout << "You've lost a chance to get stronger." << std::endl; + } + } else if (text.find("\\A") != text.npos || text.find("authors") != text.npos) { + Printer::PrintAuthorInfo(); + } else if (text.find("R") != text.npos || text.find("W") != text.npos) { + checker.ExecAnomalyIdentify(text, printer.Algs()); + } + } + return 0; +} diff --git a/src/3ts/backend/anomaly_identify.h b/src/3ts/backend/anomaly_identify.h new file mode 100644 index 00000000..b75795d2 --- /dev/null +++ b/src/3ts/backend/anomaly_identify.h @@ -0,0 +1,276 @@ +/* + * Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: farrisli@tencent.com + * + */ +//#include +#include +#include +#include "util/generic.h" +#include "cca/conflict_serializable_algorithm.h" +#include "cca/unified_history_algorithm.h" +#include "shape.h" +#include "../../../contrib/deneva/unified_concurrency_control/util.h" + +class Printer { +public: + Printer() : anomaly_map_{ + {"DirtyWrite", "WAT SDA 'R0a W0a W1a R1a W1a R1a C0 C1' Wi[xm]...Wj[xm+1]"}, + {"LostUpdate", "WAT SDA 'R0a R1a W0a R0a W0a W1a A1 C0' Ri[xm]...Wj[xm+1]...Wi[xm+2]"}, + {"LostSelfUpdate", "WAT SDA 'R0a W0a R0a W1a R0a W1a C0 C1' Wi[xm]...Wj[xm+1]...Ri[xm+1]"}, + {"Full-Write", "WAT SDA 'R0a W0a R1a W1a W0a C0 W1a C1' Wi[xm]...Wj[xm+1]...Wi[xm+2]"}, + {"Read-WriteSkew1", "WAT DDA 'R0a W0a R0a R1b W0b W1a C0 C1' Ri[xm]...Wj[xm+1]...Wj[yn]...Wi[yn+1]"}, + {"Read-WriteSkew2", "WAT DDA 'R0a W0a W0b W1a R1b W0b C0 C1' Wi[xm]...Wj[xm+1]...Wj[yn]...Ri[yn]"}, + {"Double-WriteSkew1", "WAT DDA 'R0a W0a R1a W1a W1b W0b C1 C0' Wi[xm]...Rj[xm]...Wj[yn]...Wi[yn+1]"}, + {"Double-WriteSkew2", "WAT DDA 'R0a W0a R0a W1a W1b R0b C1 C0' Wi[xm]...Wj[xm+1]...Rj[yn]...Wi[yn+1]"}, + {"Full-WriteSkew", "WAT DDA 'R0a W0a W1a W1b W0b R1a C0 C1' Wi[xm]...Wj[xm+1]...Wj[yn]...Wi[yn+1]"}, + {"StepWAT", "WAT MDA 'R0a R0b W1b W2c W0c C0 C1 W2b C2' ...Wi[xm]...Wj[xm+1]..."}, + {"DirtyRead", "RAT SDA 'R0a W0a R1a R0a R1a R0a C1 A0' Wi[xm]...Rj[xm+1]"}, + {"Non-RepeatableRead", "RAT SDA 'R0a R1a R0a W1a R0a R1a C1 C0' Ri[xm]...Wj[xm+1]...Ri[xm+1]"}, + {"IntermediateRead", "RAT SDA 'R0a W0a R1a W0a R1a W0a C1 C0' Wi[xm]...Rj[xm+1]...Wi[xm+2]"}, + {"ReadSkew", "RAT DDA 'R0a W0a R1b R0b W0b R1a C0 C1' Ri[xm]...Wj[xm+1]...Wj[yn]...Ri[yn]"}, + {"ReadSkew2", "RAT DDA 'R0a W0a W0b R1b R1a C1 W0a C0' Wi[xm]...Rj[xm]...Rj[yn]...Wi[yn+1]"}, + {"Write-ReadSkew", "RAT DDA 'R0a W0a R1a W1b R0b R1a C0 C1' Wi[xm]...Rj[xm]...Wj[yn]...Ri[yn]"}, + {"StepRAT", "RAT MDA 'R0a R0b W1a R2a R2c W0c C0 C1 C2' ...Wi[xm]...Rj[xm]..., and not include (...Wii[xm]...Wjj[xm+1]...)"}, + {"WriteSkew", "IAT DDA 'R0a R0b R1a W0a R1b W1b C1 C0' Ri[xm]...Wj[xm+1]...Rj[yn]...Wi[yn+1]"}, + {"StepIAT", "IAT MDA 'R0a R0b R1c W1a W2c A1 C2 W0c C0' ...Ri[xm]...Wj[xm+1]..., and not include (...Wii[xm]...Rjj[xm]...and ...Wiii[xm]...Wjjj[xm+1]...)"} + }, info_map_{ + {"History", "The sequence of operations that produces the data anomaly, one history contains several operations."}, + {"Operation", "One operation contains 3 character, such as R0a, first character is operation type, second character is transaction id, third character is data item.\n Operation Type -> Such as R W C A(R: Read, W: Write, C: Commit, A: Aort)\n Transaction ID -> Such as 0 1 2 ...(must be a number and less than 10)\n Data Item -> Such as a b c ...(must be lowercase letter)"}, + {"WAT", "WAT is a kind of Data Anomalies, which has 'WW' partial order in the cycle."}, + {"RAT", "RAT is a kind of Data Anomalies, which has one or more 'WR' partial orders in the cycle but no 'WW' partial order."}, + {"IAT", "IAT is a kind of Data Anomalies in addition to 'WAT' and 'RAT'."}, + {"SDA", "SDA is a subdivision type of Data Anomalies, which occur on two transactions in single variable."}, + {"DDA", "DDA is a subdivision type of Data Anomalies, which occur on two transactions in double variables."}, + {"MDA", "MDA is a subdivision type of Data Anomalies in addition to 'SDA' and 'DDA'"} + } {}; + + static std::vector InitAnomalyList() { + return std::vector { + "Dirty Write WAT SDA 'R0a W0a W1a R1a W1a R1a C0 C1' Wi[xm]...Wj[xm+1]", + "Lost Update WAT SDA 'R0a R1a W0a R0a W0a W1a A1 C0' Ri[xm]...Wj[xm+1]...Wi[xm+2]", + "Lost Self Update WAT SDA 'R0a W0a R0a W1a R0a W1a C0 C1' Wi[xm]...Wj[xm+1]...Ri[xm+1]", + "Full-Write WAT SDA 'R0a W0a R1a W1a W0a C0 W1a C1' Wi[xm]...Wj[xm+1]...Wi[xm+2]", + "Read-Write Skew 1 WAT DDA 'R0a W0a R0a R1b W0b W1a A0 C1' Ri[xm]...Wj[xm+1]...Wj[yn]...Wi[yn+1]", + "Read-Write Skew 2 WAT DDA 'R0a W0a W0b W1a R1b W0b C0 C1' Wi[xm]...Wj[xm+1]...Wj[yn]...Ri[yn]", + "Double-Write Skew 1 WAT DDA 'R0a W0a R1a W1a W1b W0b A1 C0' Wi[xm]...Rj[xm]...Wj[yn]...Wi[yn+1]", + "Double-Write Skew 2 WAT DDA 'R0a W0a R0a W1a W1b R0b C1 C0' Wi[xm]...Wj[xm+1]...Rj[yn]...Wi[yn+1]", + "Full-Write Skew WAT DDA 'R0a W0a W1a W1b W0b R1a A0 C1' Wi[xm]...Wj[xm+1]...Wj[yn]...Wi[yn+1]", + "Step WAT WAT MDA 'R0a R0b W1b W2c W0c A0 C1 W2b C2' ...Wi[xm]...Wj[xm+1]...", + "", + "Dirty Read RAT SDA 'R0a W0a R1a R0a R1a R0a C1 A0' Wi[xm]...Rj[xm+1]", + "Unrepeatable Read RAT SDA 'R0a R1a R0a W1a R0a R1a C1 C0' Ri[xm]...Wj[xm+1]...Ri[xm+1]", + "Intermediate Read RAT SDA 'R0a W0a R1a W0a R1a W0a C1 C0' Wi[xm]...Rj[xm+1]...Wi[xm+2]", + "Read Skew RAT DDA 'R0a W0a R1b R0b W0b R1a C0 A1' Ri[xm]...Wj[xm+1]...Wj[yn]...Ri[yn]", + "Read Skew 2 RAT DDA 'R0a W0a W0b R1b R1a C1 W0a C0' Wi[xm]...Rj[xm]...Rj[yn]...Wi[yn+1]", + "Write-Read Skew RAT DDA 'R0a W0a R1a W1b R0b R1a A0 C1' Wi[xm]...Rj[xm]...Wj[yn]...Ri[yn]", + "Step RAT RAT MDA 'R0a R0b W1a R2a R2c W0c C0 C1 C2' ...Wi[xm]...Rj[xm]..., and not include (...Wii[xm]...Wjj[xm+1]...)", + "", + "Write Skew IAT DDA 'R0a R0b R1a W0a R1b W1b C1 C0' Ri[xm]...Wj[xm+1]...Rj[yn]...Wi[yn+1]", + "Step IAT IAT MDA 'R0a R0b R1c W1a W2c A1 C2 W0c C0' ...Ri[xm]...Wj[xm+1]..., and not include (...Wii[xm]...Rjj[xm]...and ...Wiii[xm]...Wjjj[xm+1]...)"}; + } + + static void Print(const std::string& info) { + std::cout << info << std::endl; + std::cout << std::endl; + } + + static void PrintAnomalyTableInfo(std::vector& anomaly_list) { + std::cout << "\n --------------------------- " << std::endl; + std::cout << " Data Anomaly Reduction Mode " << std::endl; + std::cout << " --------------------------- " << std::endl; + std::cout << "\nDA Name Type SubType History Example Definition" << std::endl; + std::cout << "-------------------------------------------------------------------------------------------------------------" << std::endl; + for (auto info : anomaly_list) { + std::cout << info << std::endl; + } + std::cout << "-------------------------------------------------------------------------------------------------------------\n" << std::endl; + } + + static void PrintStartInfo() { + std::cout << "Welcome to the 3TS-DAI." << std::endl; + std::cout << "version: 1.0.0" << std::endl; + std::cout << "Tencent is pleased to support the open source community by making 3TS available.\n" << std::endl; + std::cout << "For information about 3TS-DAI(Tencent Transaction Processing Tested System-Data Anomaly Identify) products and services, visit:\n https://github.com/Tencent/3TS\n" << std::endl; + std::cout << "Type 'help' or 'h' for help." << std::endl; + std::cout << "Type 'quit' or 'q' to quit the program.\n" << std::endl; + } + + static void PrintAuthorInfo() { + std::cout << "Authors:" << std::endl; + std::cout << "Chang Liu" << std::endl; + std::cout << "Yu Li" << std::endl; + std::cout << "HaiXiang Li" << std::endl; + std::cout << std::endl; + } + + static void PrintHelpInfo() { + std::cout << "List of all 3TS-DAI commands:" << std::endl; + std::cout << "definition (\\d) Output precise definitions of History and Anomaly, including History Operation WAT RAT IAT SDA DDA MDA, such as '\\d WAT'" << std::endl; + std::cout << "algorithm (\\g) Select the algorithm that identifies the exception, including DLI_IDENTIFY_CYCLE DLI_IDENTIFY_CHAIN, such as '\\g DLI_IDENTIFY_CYCLE'. If you want to compare different algorithms, you can type such as '\\g DLI_IDENTIFY_CYCLE, DLI_IDENTIFY_CHAIN'" << std::endl; + std::cout << "anomaly (\\a) Output history sequence of anomaly, including " << std::endl; + std::cout << " WAT: Dirty Write, Lost Update, Lost Self Update, Full-Write, Read-Write Skew 1, Read-Write Skew 2, Double-Write Skew 1, Double-Write Skew 2, Full-Write Skew, Step WAT" << std::endl; + std::cout << " RAT: Dirty Read, Non-Repeatable Read, Intermediate Read, Read Skew, Read Skew 2, Write-Read Skew, Step RAT" << std::endl; + std::cout << " IAT: Write Skew, Step IAT" << std::endl; + std::cout << " such as '\\a Dirty Write'" << std::endl; + std::cout << " We do not support anomaly Identification for predicate classes for now, such as Phantom Read" << std::endl; + std::cout << "table (\\t)" << std::endl; + std::cout << "authors (\\A)" << std::endl; + std::cout << std::endl; + } + + static void TrimSpace(std::string& str) { + auto itor = remove_if(str.begin(), str.end(), ::isspace); + str.erase(itor, str.end()); + } + + std::vector Algs() const { return alg_type_list_; }; + void SetAlgs(std::vector alg_type_list) { alg_type_list_ = alg_type_list; }; + + std::unordered_map InfoMap() const { return info_map_; }; + std::unordered_map AnomalyMap() const { return anomaly_map_; }; +private: + std::vector alg_type_list_ = {ttts::UniAlgs::UNI_DLI_IDENTIFY_CYCLE}; + std::unordered_map info_map_; + std::unordered_map anomaly_map_; +}; + +class Checker { +public: + static void split(const std::string& str, std::vector& tokens, const std::string delim) { + tokens.clear(); + auto start = str.find_first_not_of(delim, 0); + auto position = str.find_first_of(delim, start); + while (position != std::string::npos || start != std::string::npos) { + tokens.emplace_back(std::move(str.substr(start, position - start))); + start = str.find_first_not_of(delim, position); + position = str.find_first_of(delim, start); + } + } + + void ExecAnomalyIdentify(const std::string& text, std::vector alg_type_list) { + ttts::History history; + std::istringstream is(text); + + const auto get_and_print_anomaly = [&] (auto&& alg, auto&& alg_type) { + const std::optional anomaly = alg.GetAnomaly(history, nullptr); + PrintAnomalyInfo(anomaly, alg_type); + }; + + if ((is >> history)) { + for (const auto& alg_type : alg_type_list) { + if (alg_type == ttts::UniAlgs::UNI_DLI_IDENTIFY_CYCLE) { + get_and_print_anomaly(ttts::ConflictSerializableAlgorithm(), alg_type); + } else if (alg_type == ttts::UniAlgs::UNI_DLI_IDENTIFY_CHAIN) { + ttts::UnifiedHistoryAlgorithm alg; + get_and_print_anomaly(ttts::UnifiedHistoryAlgorithm(), alg_type); + } + } + } + } + + void PrintAnomalyInfo(const std::optional anomaly, ttts::UniAlgs alg_type) { + if (!anomaly.has_value()) { + std::cout << "No Data Anomaly\n" << std::endl; + } else { + const std::vector anomaly_info = AnomalyInfo(ttts::ToString(anomaly.value())); + std::cout << ttts::ToString(alg_type) << "'s Identification Result:" << std::endl; + std::cout << "Anomaly Type: " << anomaly_info[0] << "\nAnomaly SubType: " << anomaly_info[1] << "\nAnomaly Name: " << anomaly_info[2] << "\nAnomaly Format: " << anomaly_info[3] << "\n" << std::endl; + } + } + + std::vector AnomalyInfo(const std::string& anomaly) { + auto index = anomaly.find_first_of("_"); + std::vector anomaly_info; + if (index != anomaly.npos) { + // get anomaly_type + anomaly_info.emplace_back(anomaly.substr(0, index)); + // get anomaly_subtype + std::string anomaly_subtype = ""; + std::string m = anomaly.substr(index + 1, 1); + if ("1" == m) { + anomaly_subtype = "SDA"; + } else if ("2" == m) { + anomaly_subtype = "DDA"; + } else { + anomaly_subtype = "MDA"; + } + anomaly_info.emplace_back(anomaly_subtype); + // get anomaly_name + std::string name = anomaly.substr(index + 3); + if (anomaly.find("STEP") == anomaly.npos) { + bool is_head = false; + for (size_t i = 0; i < name.size(); i++) { + if (i == 0) { + continue; + } else if (name[i] == '_') { + name[i] = ' '; + is_head = true; + } else if (is_head == true) { + is_head = false; + } else if (name[i] >= 'A' && name[i] <= 'Z') { + name[i] += 'a' - 'A'; // Convert to lowercase + } + } + } else { + name = "Step " + anomaly_info[0]; + } + anomaly_info.emplace_back(name); + // get anomaly_format + std::string format = ""; + if ("Dirty Write" == name) { + format = "Wi[xm]...Wj[xm+1]"; + } else if ("Full Write" == name) { + format = "Wi[xm]...Wj[xm+1]...Wi[xm+2]"; + } else if ("Lost Self Update" == name) { + format = "Wi[xm]...Wj[xm+1]...Ri[xm+1]"; + } else if ("Lost Update" == name) { + format = "Ri[xm]...Wj[xm+1]...Wi[xm+2]"; + } else if ("Double Write Skew 1" == name) { + format = "Wi[xm]...Rj[xm]...Wj[yn]...Wi[yn+1]"; + } else if ("Double Write Skew 2" == name) { + format = "Wi[xm]...Wj[xm+1]...Rj[yn]...Wi[yn+1]"; + } else if ("Read Write Skew 1" == name) { + format = "Ri[xm]...Wj[xm+1]...Wj[yn]...Wi[yn+1]"; + } else if ("Read Write Skew 2" == name) { + format = "Wi[xm]...Wj[xm+1]...Wj[yn]...Ri[yn]"; + } else if ("Full Write Skew" == name) { + format = "Wi[xm]...Wj[xm+1]...Wj[yn]...Wi[yn+1]"; + } else if ("WAT_STEP" == anomaly) { + format = "...Wi[xm]...Wj[xm+1]..."; + } else if ("Dirty Read" == name) { + format = "Wi[xm]...Rj[xm+1]"; + } else if ("Non Repeatable Read" == name) { + format = "Ri[xm]...Wj[xm+1]...Ri[xm+1]"; + } else if ("Intermediate Read" == name) { + format = "Wi[xm]...Rj[xm+1]...Wi[xm+2]"; + } else if ("Write Read Skew" == name) { + format = "Wi[xm]...Rj[xm]...Wj[yn]...Ri[yn]"; + } else if ("DOUBLE_WRITE_SKEW_COMMITTED" == name) { + format = "Wi[xm]...Rj[xm+1]...Wj[yn]...Wi[yn+1]"; + } else if ("Read Skew" == name) { + format = "Ri[xm]...Wj[xm+1]...Wj[yn]...Ri[yn]"; + } else if ("Read Skew 2" == name) { + format = "Wi[xm]...Rj[xm]...Rj[yn]...Wi[yn+1]"; + } else if ("RAT_STEP" == anomaly) { + format = "...Wi[xm]...Rj[xm]..., and not include (...Wii[xm]...Wjj[xm+1]...)"; + } else if ("LOST_UPDATE_COMMITTED" == name) { + format = "Ri[xm]...Wj[xm]...Wj[yn]...Wi[yn+1]"; + } else if ("READ_WRITE_SKEW_COMMITTED" == name) { + format = "Ri[xm]...Wj[xm]...Wj[yn]...Wi[yn+1]"; + } else if ("Write Skew" == name) { + format = "Ri[xm]...Wj[xm+1]...Rj[yn]...Wi[yn+1]"; + } else if ("IAT_STEP" == anomaly) { + format = "...Ri[xm]...Wj[xm+1]..., and not include (...Wii[xm]...Rjj[xm]...and ...Wiii[xm]...Wjjj[xm+1]...)"; + } + anomaly_info.emplace_back(format); + } else { + std::cerr << "get AnomalyType failed" << std::endl; + } + return anomaly_info; + } +}; + diff --git a/src/3ts/backend/cca/algorithm.h b/src/3ts/backend/cca/algorithm.h index a5ccda60..cc12ab62 100644 --- a/src/3ts/backend/cca/algorithm.h +++ b/src/3ts/backend/cca/algorithm.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * @@ -24,6 +22,7 @@ class HistoryAlgorithm { virtual ~HistoryAlgorithm() {} virtual bool Check(const History& history, std::ostream* const os = nullptr) const = 0; + virtual void Statistics() const {}; std::string name() const { return name_; } const std::string name_; diff --git a/src/3ts/backend/cca/anomaly_type.h b/src/3ts/backend/cca/anomaly_type.h new file mode 100644 index 00000000..2feff859 --- /dev/null +++ b/src/3ts/backend/cca/anomaly_type.h @@ -0,0 +1,54 @@ +#ifdef ENUM_BEGIN +#ifdef ENUM_MEMBER +#ifdef ENUM_END + +ENUM_BEGIN(AnomalyType) +// ======== WAT - 1 ========= +ENUM_MEMBER(AnomalyType, WAT_1_DIRTY_WRITE) +ENUM_MEMBER(AnomalyType, WAT_1_FULL_WRITE) +ENUM_MEMBER(AnomalyType, WAT_1_LOST_SELF_UPDATE) +ENUM_MEMBER(AnomalyType, WAT_1_LOST_UPDATE) +// ======== WAT - 2 ========= +ENUM_MEMBER(AnomalyType, WAT_2_DOUBLE_WRITE_SKEW_1) +ENUM_MEMBER(AnomalyType, WAT_2_DOUBLE_WRITE_SKEW_2) +ENUM_MEMBER(AnomalyType, WAT_2_READ_WRITE_SKEW_1) +ENUM_MEMBER(AnomalyType, WAT_2_READ_WRITE_SKEW_2) +ENUM_MEMBER(AnomalyType, WAT_2_FULL_WRITE_SKEW) +// ======== WAT - 3 ========= +ENUM_MEMBER(AnomalyType, WAT_STEP) +// ======== RAT - 1 ========= +ENUM_MEMBER(AnomalyType, RAT_1_DIRTY_READ) +ENUM_MEMBER(AnomalyType, RAT_1_INTERMEDIATE_READ) +ENUM_MEMBER(AnomalyType, RAT_1_NON_REPEATABLE_READ) +// ======== RAT - 2 ========= +ENUM_MEMBER(AnomalyType, RAT_2_WRITE_READ_SKEW) +ENUM_MEMBER(AnomalyType, RAT_2_DOUBLE_WRITE_SKEW_COMMITTED) +ENUM_MEMBER(AnomalyType, RAT_2_READ_SKEW) +ENUM_MEMBER(AnomalyType, RAT_2_READ_SKEW_2) +// ======== RAT - 3 ========= +ENUM_MEMBER(AnomalyType, RAT_STEP) +// ======== IAT - 1 ========= +ENUM_MEMBER(AnomalyType, IAT_1_LOST_UPDATE_COMMITTED) +// ======== IAT - 2 ========= +ENUM_MEMBER(AnomalyType, IAT_2_READ_WRITE_SKEW_COMMITTED) +ENUM_MEMBER(AnomalyType, IAT_2_WRITE_SKEW) +// ======== IAT - 3 ========= +ENUM_MEMBER(AnomalyType, IAT_STEP) +// ======== Unknown ========= +ENUM_MEMBER(AnomalyType, UNKNOWN_1) +ENUM_MEMBER(AnomalyType, UNKNOWN_2) +ENUM_END(AnomalyType) + +#endif +#endif +#endif + +#ifndef TTTS_CCA_ANOMALY_TYPE_H_ +#define TTTS_CCA_ANOMALY_TYPE_H_ + +namespace ttts { +#define ENUM_FILE "../cca/anomaly_type.h" +#include "../util/extend_enum.h" +} + +#endif // TTTS_CCA_ANOMALY_TYPE_H_ diff --git a/src/3ts/backend/cca/conflict_serializable_algorithm.h b/src/3ts/backend/cca/conflict_serializable_algorithm.h index 276223b2..85f240b2 100644 --- a/src/3ts/backend/cca/conflict_serializable_algorithm.h +++ b/src/3ts/backend/cca/conflict_serializable_algorithm.h @@ -1,53 +1,158 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * */ -#pragma once + + +#ifndef CCA_CONFLICT_SERIALIZABLE_ALGORITHM +#define CCA_CONFLICT_SERIALIZABLE_ALGORITHM + +#include +#include +#include +#include #include "algorithm.h" +#include "anomaly_type.h" +#include "prece_type.h" namespace ttts { +class DAPreceInfo { + public: + DAPreceInfo(const uint64_t pre_trans_id, const uint64_t trans_id, const uint64_t item_id, const PreceType type, const uint32_t order) + : pre_trans_id_(pre_trans_id), trans_id_(trans_id), item_id_(item_id), type_(type), order_(order) {} + DAPreceInfo(const DAPreceInfo&) = default; + + friend std::ostream& operator<<(std::ostream& os, const DAPreceInfo prece) { + return os << 'T' << static_cast('0' + prece.pre_trans_id_) << "-[" << prece.type_ << "-" + << static_cast('a' + prece.item_id_) << "]->" << 'T' << static_cast('0' + prece.trans_id_); + } + bool operator>(const DAPreceInfo& p) const { return order_ > p.order_; } + bool operator<(const DAPreceInfo& p) const { return order_ < p.order_; } + + uint64_t item_id() const { return item_id_; } + PreceType type() const { return type_; } + + private: + uint64_t pre_trans_id_; + uint64_t trans_id_; + uint64_t item_id_; + PreceType type_; + uint32_t order_; +}; + class ConflictGraphNode { public: - ConflictGraphNode() : removed_(false) {} + ConflictGraphNode(const uint64_t trans_id) : trans_id_(trans_id), removed_(false) {} ~ConflictGraphNode() {} bool HasNoPreTrans() const { return pre_trans_set_.empty(); } - void AddPreTrans(const uint64_t pre_trans_id) { pre_trans_set_.insert(pre_trans_id); } + void AddPreTrans(const uint64_t pre_trans_id, const uint64_t item_id, const PreceType type, const uint32_t order) { + // we only record the first precedence between the two specific transactions + pre_trans_set_.try_emplace(pre_trans_id, DAPreceInfo{pre_trans_id, trans_id_, item_id, type, order}); + } void RemovePreTrans(const uint64_t pre_trans_id) { pre_trans_set_.erase(pre_trans_id); } void Remove() { removed_ = true; } bool IsRemoved() const { return removed_; } + std::optional& is_committed() { return is_committed_; } + const std::optional& is_committed() const { return is_committed_; } + const std::map& pre_trans_set() const { return pre_trans_set_; } + uint64_t trans_id() const { return trans_id_; } private: - std::set pre_trans_set_; + const uint64_t trans_id_; + std::map pre_trans_set_; bool removed_; + std::optional is_committed_; +}; + +template +static void sort(Container&& container, Compare&& comp) { + if (!std::is_sorted(container.begin(), container.end(), comp)) { + std::sort(container.begin(), container.end(), comp); + } +} + +class DAPath { + public: + DAPath() {} + DAPath(std::vector&& preces) : preces_((sort(preces, std::greater()), std::move(preces))) {} + DAPath(const DAPreceInfo& prece) : preces_{prece} {} + DAPath(const DAPath&) = default; + DAPath(DAPath&&) = default; + DAPath& operator=(const DAPath&) = default; + DAPath& operator=(DAPath&&) = default; + + bool operator<(const DAPath& p) const { + // impassable has the greatest weight + if (!passable()) { + return false; + } + if (!p.passable()) { + return true; + } + return std::lexicographical_compare(preces_.begin(), preces_.end(), p.preces_.begin(), p.preces_.end()); + } + + DAPath operator+(const DAPath& p) const { + if (!passable() || !p.passable()) { + return {}; + } + std::vector preces; + std::merge(preces_.begin(), preces_.end(), p.preces_.begin(), p.preces_.end(), std::back_inserter(preces), std::greater()); + return preces; + } + + friend std::ostream& operator<<(std::ostream& os, const DAPath& path) { + if (path.preces_.empty()) { + os << "Empty path"; + } else { + std::copy(path.preces_.begin(), path.preces_.end(), std::ostream_iterator(os, ", ")); + } + return os; + } + + bool passable() const { return !preces_.empty(); } + + const std::vector& preces() const { return preces_; } + + private: + std::vector preces_; }; class ConflictGraph { public: - ConflictGraph(const uint64_t trans_num) : nodes_(trans_num) {} + ConflictGraph(const uint64_t trans_num) : nodes_() { + nodes_.reserve(trans_num); + for (uint64_t trans_id = 0; trans_id < trans_num; ++trans_id) { + nodes_.emplace_back(trans_id); + } + } - void Insert(const uint64_t pre_trans_id, const uint64_t trans_id) { + template + void Insert(const uint64_t pre_trans_id, const uint64_t trans_id, const uint64_t item_id, const uint32_t order) { if (pre_trans_id == trans_id) { return; } - nodes_[trans_id].AddPreTrans(pre_trans_id); + const auto type = RealPreceType_(pre_trans_id); + if (type.has_value()) { + nodes_[trans_id].AddPreTrans(pre_trans_id, item_id, *type, order); + } } - void Insert(const std::set& pre_trans_set, const uint64_t cur_trans_id) { - for (const uint64_t pre_trans : pre_trans_set) { - Insert(pre_trans, cur_trans_id); + template + void Insert(const std::set& pre_trans_id_set, const uint64_t trans_id, const uint64_t item_id, const uint32_t order) { + for (const uint64_t pre_trans_id : pre_trans_id_set) { + Insert(pre_trans_id, trans_id, item_id, order); } } bool HasCycle() { - RemoveNodesNotInCycle(); + RemoveNodesNotInCycle_(); for (const ConflictGraphNode& node : nodes_) { if (!node.HasNoPreTrans()) { return true; @@ -56,70 +161,284 @@ class ConflictGraph { return false; } + std::optional& is_committed(const uint64_t trans_id) { return nodes_[trans_id].is_committed(); } + + // Find the first conflict cycle in history. The latest precedence is at the head of return path. + DAPath MinCycleByFloyd() const { + const size_t trans_num = nodes_.size(); + DAPath matrix[trans_num][trans_num]; + + // init matrix + for (uint64_t pre_trans_id = 0; pre_trans_id < trans_num; ++pre_trans_id) { + for (uint64_t trans_id = 0; trans_id < trans_num; ++trans_id) { + auto& path = matrix[pre_trans_id][trans_id]; + auto& pre_trans_set = nodes_[trans_id].pre_trans_set(); + if (const auto it = pre_trans_set.find(pre_trans_id); it != nodes_[trans_id].pre_trans_set().end()) { + path = it->second; + } + } + } + + static auto update_path = [](DAPath& path, DAPath&& new_path) { + if (new_path < path) { + path = std::move(new_path); // do not use std::min because there is a copy cost when assign self + } + }; + + DAPath min_cycle; + for (uint64_t mid = 0; mid < trans_num; ++mid) { + // find mini cycle when pass mid node + for (uint64_t start = 0; start < mid; ++start) { + update_path(min_cycle, matrix[start][mid] + matrix[mid][start]); + for (uint64_t end = 0; end < mid; ++end) { + if (start != end) { + update_path(min_cycle, matrix[start][end] + matrix[end][mid] + matrix[mid][start]); + } + } + } + + // update direct path + for (uint64_t start = 0; start < trans_num; ++start) { + for (uint64_t end = 0; end < trans_num; ++end) { + update_path(matrix[start][end], matrix[start][mid] + matrix[mid][end]); + } + } + } + return min_cycle; + } + private: - void RemoveNodesNotInCycle() { - bool found_no_pre_trans = false; - // A trans which did not have pretrans can not be a part of cycle. + std::optional RealPreceType_(const uint64_t pre_trans_id, const std::optional& active_prece_type, + const std::optional& committed_prece_type, const std::optional& aborted_prece_type) const { + if (!nodes_[pre_trans_id].is_committed().has_value()) { + return active_prece_type; + } else if (nodes_[pre_trans_id].is_committed().value()) { + return committed_prece_type; + } else { + return aborted_prece_type; + } + } + + template + std::optional RealPreceType_(const uint64_t pre_trans_id) const { + if constexpr (TYPE == PreceType::WW) { + return RealPreceType_(pre_trans_id, PreceType::WW, PreceType::WCW, {}); + } else if constexpr (TYPE == PreceType::WR) { + return RealPreceType_(pre_trans_id, PreceType::WR, PreceType::WCR, {}); + } else { + return RealPreceType_(pre_trans_id, TYPE, TYPE, {}); + } + } + + // keep remove nodes of which indegree is 0 + void RemoveNodesNotInCycle_() { + bool removed_node = false; do { - found_no_pre_trans = false; - for (uint64_t trans_id = 0; trans_id < nodes_.size(); ++trans_id) { - if (nodes_[trans_id].HasNoPreTrans() && !nodes_[trans_id].IsRemoved()) { - found_no_pre_trans = true; - nodes_[trans_id].Remove(); - for (ConflictGraphNode& node : nodes_) { - node.RemovePreTrans(trans_id); - } + removed_node = false; + for (ConflictGraphNode& node_to_remove : nodes_) { + if (!node_to_remove.HasNoPreTrans() || node_to_remove.IsRemoved()) { + continue; + } + removed_node = true; + node_to_remove.Remove(); + for (ConflictGraphNode& node : nodes_) { + node.RemovePreTrans(node_to_remove.trans_id()); } } - } while (found_no_pre_trans); + } while (removed_node); } private: std::vector nodes_; }; +template class ConflictSerializableAlgorithm : public HistoryAlgorithm { public: - ConflictSerializableAlgorithm() : HistoryAlgorithm("Conflict Serializable") {} + ConflictSerializableAlgorithm() : HistoryAlgorithm(IDENTIFY_ANOMALY ? "DLI_IDENTIFY OK" : "Conflict Serializable"), anomaly_counts_{0}, no_anomaly_count_(0) {} virtual ~ConflictSerializableAlgorithm() {} - virtual bool Check(const History& history, std::ostream* const os) const override { + void Statistics() const override { + if (!IDENTIFY_ANOMALY) { return; } + std::cout.setf(std::ios::right); + std::cout.precision(4); + + static auto print_percent = [](auto&& value, auto&& total) { + std::cout << std::setw(10) << static_cast(value) / total * 100 << "% = " << std::setw(5) << value << " / " << std::setw(5) << total; + }; + + const auto anomaly_count = std::accumulate(anomaly_counts_.begin(), anomaly_counts_.end(), 0); + std::cout << "=== DLI_IDENTIFY ===" << std::endl; + + std::cout << std::setw(40) << "True Rollback: "; + print_percent(anomaly_count, anomaly_count + no_anomaly_count_); + std::cout << std::endl; + + std::vector> sorted_anomaly_counts_; + for (const auto anomaly : Members()) { + sorted_anomaly_counts_.emplace_back(anomaly, anomaly_counts_.at(static_cast(anomaly))); + } + std::sort(sorted_anomaly_counts_.begin(), sorted_anomaly_counts_.end(), [](auto&& _1, auto&& _2) { return _1.second > _2.second; }); + for (const auto& [anomaly, count] : sorted_anomaly_counts_) { + std::cout << std::setw(40) << (std::string("[") + ToString(anomaly) + "] "); + print_percent(count, anomaly_count); + std::cout << std::endl; + } + std::cout << "=== DLI_IDENTIFY END ===" << std::endl; + } + + std::optional GetAnomaly(const History& history, std::ostream* const os) const { ConflictGraph graph(history.trans_num()); std::vector> read_trans_set_for_items(history.item_num()); std::vector> write_trans_set_for_items(history.item_num()); std::vector> write_item_set_for_transs(history.trans_num()); - for (const Operation& operation : history.operations()) { + for (size_t i = 0, size = history.size(); i < size; ++i) { + const Operation& operation = history.operations()[i]; const uint64_t trans_id = operation.trans_id(); if (operation.IsPointDML()) { - std::set& read_trans_set = read_trans_set_for_items[operation.item_id()]; - std::set& write_trans_set = write_trans_set_for_items[operation.item_id()]; + const uint64_t item_id = operation.item_id(); + std::set& read_trans_set = read_trans_set_for_items[item_id]; + std::set& write_trans_set = write_trans_set_for_items[item_id]; if (Operation::Type::READ == operation.type()) { - graph.Insert(write_trans_set, trans_id); + graph.Insert(write_trans_set, trans_id, item_id, i); // WR or WCR is both possible read_trans_set.insert(trans_id); } else if (Operation::Type::WRITE == operation.type()) { - graph.Insert(read_trans_set, trans_id); - graph.Insert(write_trans_set, trans_id); + // WW precedence's priority is higher than RW precedence + graph.Insert(write_trans_set, trans_id, item_id, i); // WW or WCW is both possible + graph.Insert(read_trans_set, trans_id, item_id, i); write_trans_set.insert(trans_id); - write_item_set_for_transs[trans_id].insert(operation.item_id()); // record for abort + write_item_set_for_transs[trans_id].insert(item_id); // record for check WA or WC } } else if (Operation::Type::SCAN_ODD == operation.type()) { // TODO: realize scan odd } else if (Operation::Type::ABORT == operation.type()) { + graph.is_committed(trans_id) = false; for (const uint64_t write_item : write_item_set_for_transs[trans_id]) { - // restore all items has been written - graph.Insert(read_trans_set_for_items[write_item], trans_id); - graph.Insert(write_trans_set_for_items[write_item], trans_id); + // WA precedence's priority is higher than RA precedence + graph.Insert(write_trans_set_for_items[write_item], trans_id, write_item, i); + graph.Insert(read_trans_set_for_items[write_item], trans_id, write_item, i); + } + } else if (Operation::Type::COMMIT == operation.type()) { + graph.is_committed(trans_id) = true; + for (const uint64_t write_item : write_item_set_for_transs[trans_id]) { + graph.Insert(write_trans_set_for_items[write_item], trans_id, write_item, i); } } } - return !graph.HasCycle(); + + const bool has_cycle = graph.HasCycle(); + if (IDENTIFY_ANOMALY && has_cycle) { + const auto cycle = graph.MinCycleByFloyd(); + const auto anomaly = IdentifyAnomaly_(cycle.preces()); + TRY_LOG(os) << "[" << anomaly << "] " << cycle; + ++(anomaly_counts_.at(static_cast(anomaly))); + return anomaly; + } else { + ++no_anomaly_count_; + return {}; + } + } + + virtual bool Check(const History& history, std::ostream* const os) const override { + return !(GetAnomaly(history, os).has_value()); } private: - struct ReadWriteSet { - std::set reads_; - std::set writes_; - }; + static uint64_t ItemCount_(const std::vector& preces) { + std::set item_ids; + for (const auto prece : preces) { + item_ids.emplace(prece.item_id()); + } + return item_ids.size(); + } + + static AnomalyType IdentifyAnomaly_(const std::vector& preces) { + assert(preces.size() >= 2); + if (std::any_of(preces.begin(), preces.end(), [](const DAPreceInfo& prece) { return prece.type() == PreceType::WA || prece.type() == PreceType::WC; })) { + // WA and WC precedence han only appear + return AnomalyType::WAT_1_DIRTY_WRITE; + } else if (std::any_of(preces.begin(), preces.end(), [](const DAPreceInfo& prece) { return prece.type() == PreceType::RA; })) { + return AnomalyType::RAT_1_DIRTY_READ; + } else if (preces.size() >= 3) { + return IdentifyAnomalyMultiple_(preces); + // when build path, later happened precedence is sorted to front + } else if (preces.back().item_id() != preces.front().item_id()) { + return IdentifyAnomalyDouble_(preces.back().type(), preces.front().type()); + } else { + return IdentifyAnomalySingle_(preces.back().type(), preces.front().type()); + } + } + + // require type1 precedence happens before type2 precedence + static AnomalyType IdentifyAnomalySingle_(const PreceType early_type, const PreceType later_type) { + if ((early_type == PreceType::WW || early_type == PreceType::WR) && (later_type == PreceType::WW || later_type == PreceType::WCW)) { + return AnomalyType::WAT_1_FULL_WRITE; // WW-WW | WR-WW = WWW + } else if (early_type == PreceType::WR && early_type == PreceType::WW) { + return AnomalyType::WAT_1_FULL_WRITE; // WR-WW = WWW + } else if ((early_type == PreceType::WW || early_type == PreceType::WR) && (later_type == PreceType::WR || later_type == PreceType::WCR)) { + return AnomalyType::WAT_1_LOST_SELF_UPDATE; // WW-WR = WWR + } else if (early_type == PreceType::RW && later_type == PreceType::WW) { + return AnomalyType::WAT_1_LOST_UPDATE; // RW-WW | RW-RW = RWW + } else if (early_type == PreceType::WR && later_type == PreceType::RW) { + return AnomalyType::RAT_1_INTERMEDIATE_READ; // WR-RW = WRW + } else if (early_type == PreceType::RW && (later_type == PreceType::WR || later_type == PreceType::WCR)) { + return AnomalyType::RAT_1_NON_REPEATABLE_READ; // RW-WR = RWR + } else if (early_type == PreceType::RW && later_type == PreceType::WCW) { + return AnomalyType::IAT_1_LOST_UPDATE_COMMITTED; // RW-WW(WCW) = RWW + } else { + return AnomalyType::UNKNOWN_1; + } + } + + static AnomalyType IdentifyAnomalyDouble_(const PreceType early_type, const PreceType later_type) { + const auto any_order = [early_type, later_type](const PreceType type1, const PreceType type2) -> std::optional { + if (early_type == type1 && later_type == type2) { + return true; + } else if (early_type == type2 && later_type == type1) { + return false; + } else { + return {}; + } + }; + if (const auto order = any_order(PreceType::WR, PreceType::WW); order.has_value()) { + return *order ? AnomalyType::WAT_2_DOUBLE_WRITE_SKEW_1 : AnomalyType::WAT_2_DOUBLE_WRITE_SKEW_2; + } else if (early_type == PreceType::WW && later_type == PreceType::WCR) { + return AnomalyType::WAT_2_DOUBLE_WRITE_SKEW_2; + } else if (const auto order = any_order(PreceType::RW, PreceType::WW); order.has_value()) { + return *order ? AnomalyType::WAT_2_READ_WRITE_SKEW_1 : AnomalyType::WAT_2_READ_WRITE_SKEW_2; + } else if (early_type == PreceType::WW && (later_type == PreceType::WW || later_type == PreceType::WCW)) { + return AnomalyType::WAT_2_FULL_WRITE_SKEW; + } else if (early_type == PreceType::WR && (later_type == PreceType::WR || later_type == PreceType::WCR)) { + return AnomalyType::RAT_2_WRITE_READ_SKEW; + } else if (early_type == PreceType::WR && later_type == PreceType::WCW) { + return AnomalyType::RAT_2_DOUBLE_WRITE_SKEW_COMMITTED; + } else if (const auto order = any_order(PreceType::RW, PreceType::WR); order.has_value()) { + return *order ? AnomalyType::RAT_2_READ_SKEW : AnomalyType::RAT_2_READ_SKEW_2; + } else if (early_type == PreceType::RW && later_type == PreceType::WCR) { + return AnomalyType::RAT_2_READ_SKEW; + } else if (early_type == PreceType::RW && later_type == PreceType::WCW) { + return AnomalyType::IAT_2_READ_WRITE_SKEW_COMMITTED; + } else if (early_type == PreceType::RW && later_type == PreceType::RW) { + return AnomalyType::IAT_2_WRITE_SKEW; + } else { + return AnomalyType::UNKNOWN_2; + } + } + + static AnomalyType IdentifyAnomalyMultiple_(const std::vector& preces) { + if (std::any_of(preces.begin(), preces.end(), [](const DAPreceInfo& prece) { return prece.type() == PreceType::WW; })) { + return AnomalyType::WAT_STEP; + } + if (std::any_of(preces.begin(), preces.end(), [](const DAPreceInfo& prece) { return prece.type() == PreceType::WR || prece.type() == PreceType::WCR; })) { + return AnomalyType::RAT_STEP; + } + return AnomalyType::IAT_STEP; + } + + mutable std::array, Count()> anomaly_counts_; + mutable std::atomic no_anomaly_count_; }; } // namespace ttts + +#endif diff --git a/src/3ts/backend/cca/occ_algorithm/env/rc_env.h b/src/3ts/backend/cca/occ_algorithm/env/rc_env.h index 8ac299fb..75d9bc17 100644 --- a/src/3ts/backend/cca/occ_algorithm/env/rc_env.h +++ b/src/3ts/backend/cca/occ_algorithm/env/rc_env.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * diff --git a/src/3ts/backend/cca/occ_algorithm/env/si_env.h b/src/3ts/backend/cca/occ_algorithm/env/si_env.h index 3476d36a..e43a3efd 100644 --- a/src/3ts/backend/cca/occ_algorithm/env/si_env.h +++ b/src/3ts/backend/cca/occ_algorithm/env/si_env.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * diff --git a/src/3ts/backend/cca/occ_algorithm/occ_algorithm.h b/src/3ts/backend/cca/occ_algorithm/occ_algorithm.h index 0bb75ff1..e19d42c1 100644 --- a/src/3ts/backend/cca/occ_algorithm/occ_algorithm.h +++ b/src/3ts/backend/cca/occ_algorithm/occ_algorithm.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * diff --git a/src/3ts/backend/cca/occ_algorithm/trans/bocc_trans.h b/src/3ts/backend/cca/occ_algorithm/trans/bocc_trans.h index f3cfa9f7..f460ed88 100644 --- a/src/3ts/backend/cca/occ_algorithm/trans/bocc_trans.h +++ b/src/3ts/backend/cca/occ_algorithm/trans/bocc_trans.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: elioyan@tencent.com * diff --git a/src/3ts/backend/cca/occ_algorithm/trans/dli_trans.h b/src/3ts/backend/cca/occ_algorithm/trans/dli_trans.h new file mode 100644 index 00000000..6c4d7f97 --- /dev/null +++ b/src/3ts/backend/cca/occ_algorithm/trans/dli_trans.h @@ -0,0 +1,136 @@ +/* + * Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ +#pragma once +#include "../env/si_env.h" +#include "../occ_algorithm.h" + +namespace ttts { +namespace occ_algorithm { + +class DLITransactionDesc + : public SITransactionDesc { + public: + static std::string name; + + DLITransactionDesc(const uint64_t trans_id, const uint64_t start_ts, + Snapshot&& snapshot, env_desc_type& env_desc) + : SITransactionDesc(trans_id, start_ts, std::move(snapshot), env_desc) {} + + virtual std::optional CheckConflict(const uint64_t) override { + // It is indeed unnecessary to do such a copy for each TransctionDesc. But in actual database + // system based on OCC, we usually copy the current environment to transaction with which + // transaction can do conflict check without considering of the thread safe problem when other + // transaction access the gloval environment concurrently. + std::list l; + const auto add_to_link = [trans_id = trans_id_, + &l](const std::unique_ptr& ptr) { + if (ptr->trans_id() != trans_id && !ptr->is_aborted()) { + l.emplace_back(ptr.get()); + } + }; + for (const auto& ptr : env_desc_.active_trans_) { + add_to_link(ptr.second); + } + for (const auto& ptr : env_desc_.commit_trans_) { + add_to_link(ptr.second); // add by ym: Q why we detect DLI with commit_trans_? + } + return CheckConflict_(l); + } + + private: + std::optional CheckConflict_( + const std::list& transs) { + DLITransactionDesc combined_trans = *this; + for (const auto& tl : transs) { + THROW_ANOMALY(combined_trans.CheckDynamicEdgeCross(*tl)); + combined_trans.MergeTransaction(*tl); + } + return {}; + } + + void MergeTransaction(const SITransactionDesc& tl) { + const auto merge = [](const set_type& src, set_type& dst) { + for (const auto& pair : src) { + if (!Has(dst, pair.first)) { + dst[pair.first] = pair.second; // if version is different, don't change it + } + } + }; + merge(tl.r_items(), r_items_); + merge(tl.w_items(), w_items_); + } + + struct IntersectionItem { + IntersectionItem(const uint64_t item_id, const uint64_t version_1, const uint64_t version_2) + : item_id_(item_id), version_1_(version_1), version_2_(version_2) {} + const uint64_t item_id_; + const uint64_t version_1_; + const uint64_t version_2_; + }; + + std::optional CheckDynamicEdgeCross(const DLITransactionDesc& tl) const { + bool upper = 0, tl_upper = 0; + THROW_ANOMALY(CheckWWOrRRIntersection(r_items_, upper, tl.r_items(), tl_upper)); + THROW_ANOMALY(CheckWWOrRRIntersection(w_items_, upper, tl.w_items(), tl_upper)); + THROW_ANOMALY(CheckWRIntersection(w_items_, upper, tl.r_items(), tl_upper)); + THROW_ANOMALY(CheckWRIntersection(tl.w_items(), tl_upper, r_items_, upper)); + return {}; + }; + + std::optional CheckWWOrRRIntersection(const set_type& s1, bool& upper1, + const set_type& s2, bool& upper2) const { + for (const auto& inter_item : GetIntersection(s1, s2)) { + if (inter_item.version_1_ > inter_item.version_2_) { + upper1 = true; + } else if (inter_item.version_1_ < inter_item.version_2_) { + upper2 = true; + } + } + if (upper1 && upper2) { + return Anomally::EDGE_CROESS; + } + return {}; + } + + std::optional CheckWRIntersection(const set_type& w_items, bool& w_upper, + const set_type& r_items, bool& r_upper) const { + for (const auto& inter_item : GetIntersection(w_items, r_items)) { + if (inter_item.version_1_ > inter_item.version_2_) { + w_upper = true; + } else { + r_upper = true; + } + } + if (w_upper && r_upper) { + return Anomally::EDGE_CROESS; + } + return {}; + } + + static std::vector GetIntersection(const set_type& s1, const set_type& s2) { + static const auto get_version = + [](const ItemVersionDesc* ver) { + return ver ? ver->version_ + : UINT64_MAX; // ver is empty only when it is written by current transction + }; + std::vector ret; + for (const auto& pair : s1) { + if (Has(s2, pair.first)) { + ret.emplace_back(pair.first, get_version(pair.second), + get_version(AssertGet(s2, pair.first))); + } + } + return ret; + } +}; + +std::string DLITransactionDesc::name = "DLI"; + +} // namespace occ_algorithm +} // namespace ttts diff --git a/src/3ts/backend/cca/occ_algorithm/trans/focc_trans.h b/src/3ts/backend/cca/occ_algorithm/trans/focc_trans.h index fe87de9b..cb82d144 100644 --- a/src/3ts/backend/cca/occ_algorithm/trans/focc_trans.h +++ b/src/3ts/backend/cca/occ_algorithm/trans/focc_trans.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: elioyan@tencent.com * diff --git a/src/3ts/backend/cca/occ_algorithm/trans/ssi_trans.h b/src/3ts/backend/cca/occ_algorithm/trans/ssi_trans.h index 25d59394..11c159e0 100644 --- a/src/3ts/backend/cca/occ_algorithm/trans/ssi_trans.h +++ b/src/3ts/backend/cca/occ_algorithm/trans/ssi_trans.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: moggma@tencent.com * diff --git a/src/3ts/backend/cca/occ_algorithm/trans/wsi_trans.h b/src/3ts/backend/cca/occ_algorithm/trans/wsi_trans.h index 7226e73e..47f0e5f8 100644 --- a/src/3ts/backend/cca/occ_algorithm/trans/wsi_trans.h +++ b/src/3ts/backend/cca/occ_algorithm/trans/wsi_trans.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: moggma@tencent.com * diff --git a/src/3ts/backend/cca/prece_type.h b/src/3ts/backend/cca/prece_type.h new file mode 100644 index 00000000..bae18e9c --- /dev/null +++ b/src/3ts/backend/cca/prece_type.h @@ -0,0 +1,28 @@ +#ifdef ENUM_BEGIN +#ifdef ENUM_MEMBER +#ifdef ENUM_END + +ENUM_BEGIN(PreceType) +ENUM_MEMBER(PreceType, RW) +ENUM_MEMBER(PreceType, WR) +ENUM_MEMBER(PreceType, WCR) +ENUM_MEMBER(PreceType, WW) +ENUM_MEMBER(PreceType, WCW) +ENUM_MEMBER(PreceType, RA) +ENUM_MEMBER(PreceType, WC) +ENUM_MEMBER(PreceType, WA) +ENUM_END(PreceType) + +#endif +#endif +#endif + +#ifndef TTTS_CCA_PRECE_TYPE_H_ +#define TTTS_CCA_PRECE_TYPE_H_ + +namespace ttts { +#define ENUM_FILE "../cca/prece_type.h" +#include "../util/extend_enum.h" +} + +#endif // TTTS_CCA_PRECE_TYPE_H_ diff --git a/src/3ts/backend/cca/serializable_algorithm.h b/src/3ts/backend/cca/serializable_algorithm.h index 4f7c28d4..0fffd8ce 100644 --- a/src/3ts/backend/cca/serializable_algorithm.h +++ b/src/3ts/backend/cca/serializable_algorithm.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * diff --git a/src/3ts/backend/cca/unified_history_algorithm.h b/src/3ts/backend/cca/unified_history_algorithm.h new file mode 100644 index 00000000..da1cc8f8 --- /dev/null +++ b/src/3ts/backend/cca/unified_history_algorithm.h @@ -0,0 +1,105 @@ +#include "algorithm.h" +#include "../../../../contrib/deneva/unified_concurrency_control/util.h" +#include "anomaly_type.h" +#include "prece_type.h" +#include "algorithm.h" +#include "../../../../contrib/deneva/unified_concurrency_control/dli_identify_util.h" +#include "../../../../contrib/deneva/unified_concurrency_control/txn_dli_identify.h" +#include "../../../../contrib/deneva/unified_concurrency_control/row_prece.h" +#include "../../../../contrib/deneva/unified_concurrency_control/alg_dli_identify_chain.h" +#include "../../../../contrib/deneva/unified_concurrency_control/alg_dli_identify_cycle.h" +#include + +namespace ttts { + +template +class UnifiedHistoryAlgorithm : public HistoryAlgorithm { +public: + UnifiedHistoryAlgorithm() : HistoryAlgorithm(ToString(ALG)), anomaly_counts_{0} {} + + virtual bool Check(const History& history, std::ostream* const os) const override { + return !(GetAnomaly(history, os).has_value()); + } + + std::optional GetAnomaly(const History& history, std::ostream* const os) const { + std::unique_ptr> alg_manager = std::make_unique>(); + std::unordered_map>> txn_map; + std::unordered_map>> row_map; + std::unordered_map row_value_map; + std::vector>> trans_write_set_list(history.trans_num()); + for (size_t i = 0, size = history.size(); i < size; ++i) { + const Operation& operation = history.operations()[i]; + const uint64_t trans_id = operation.trans_id(); + // init txn_map + if (txn_map.count(trans_id) == 0) { + std::unique_ptr> txn = std::make_unique>(trans_id); + txn_map.emplace(trans_id, std::move(txn)); + } + // check operation whether R or W + if (operation.IsPointDML()) { + const uint64_t item_id = operation.item_id(); + // If it is an unaccessed variable, the value is initialized to 0. + if (row_value_map.count(0) == 0) { + row_value_map.emplace(item_id, 0); + } + // If it is an unaccessed variable, row obj will be put into row_map. + if (row_map.count(item_id) == 0) { + std::unique_ptr> row = std::make_unique>(item_id, 0); + row_map.emplace(item_id, std::move(row)); + } + // Exec Read + if (Operation::Type::READ == operation.type()) { + row_map[item_id]->Read(*txn_map[trans_id]); + // Exec Prewrite + } else if (Operation::Type::WRITE == operation.type()) { + row_value_map[item_id] += 1; + row_map[item_id]->Prewrite(row_value_map[item_id], *txn_map[trans_id]); + trans_write_set_list[trans_id].emplace_back(std::pair(item_id, row_value_map[item_id])); + //std::cout << "W->tx_id:" << trans_id << " item_id:" << item_id << " value:" << row_value_map[item_id] << std::endl; + // Exec Abort + } + } else if (Operation::Type::ABORT == operation.type()) { + alg_manager->Abort(*txn_map[trans_id]); + // rollback written row + for (const auto& item_write : trans_write_set_list[trans_id]) { + row_map[item_write.first]->Revoke(item_write.second, *txn_map[trans_id]); + } + // check data anomaly in abort + if (txn_map[trans_id]->cycle_ != nullptr) { + //std::cout << "abort::preces_size:" << txn_map[trans_id]->cycle_->Preces().size() << std::endl; + const auto anomaly = AlgManager::IdentifyAnomaly(txn_map[trans_id]->cycle_->Preces()); + ++(anomaly_counts_.at(static_cast(anomaly))); + return anomaly; + } + } else if (Operation::Type::COMMIT == operation.type()) { + bool ret = alg_manager->Validate(*txn_map[trans_id]); + if (ret) { + alg_manager->Commit(*txn_map[trans_id]); + // data persistence + for (const auto& row : row_map) { + row.second->Write(row_value_map[row.first], *txn_map[trans_id]); + } + } else { + alg_manager->Abort(*txn_map[trans_id]); + // rollback written row + for (const auto& item_write : trans_write_set_list[trans_id]) { + row_map[item_write.first]->Revoke(item_write.second, *txn_map[trans_id]); + } + } + // check data anomaly in commit + if (txn_map[trans_id]->cycle_ != nullptr) { + //std::cout << "commit::preces_size" << txn_map[trans_id]->cycle_->Preces().size() << std::endl; + const auto anomaly = AlgManager::IdentifyAnomaly(txn_map[trans_id]->cycle_->Preces()); + ++(anomaly_counts_.at(static_cast(anomaly))); + return anomaly; + } + } + } + return {}; + } + +private: + mutable std::array, Count()> anomaly_counts_; +}; + +} diff --git a/src/3ts/backend/history/generator.h b/src/3ts/backend/history/generator.h index 0134d562..70d8f440 100644 --- a/src/3ts/backend/history/generator.h +++ b/src/3ts/backend/history/generator.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * @@ -49,7 +47,7 @@ class RandomHistoryGenerator : public HistoryGenerator { dml_operation_num_(opt.max_dml), history_num_(history_num), with_abort_(opt.with_abort), - tail_tcl_(opt.tail_tcl), + tcl_position_(opt.tcl_position), rd_(), gen_(rd_()), rand_trans_id_(0, trans_num_ - 1), @@ -60,10 +58,13 @@ class RandomHistoryGenerator : public HistoryGenerator { virtual void DeliverHistories(const std::function &handle) const { for (uint64_t history_no = 0; history_no < history_num_; ++history_no) { - if (tail_tcl_) { + if (tcl_position_ == TclPosition::TAIL) { handle(MakeDMLHistory() + MakeTCLHistory()); - } else { + } else if (tcl_position_ == TclPosition::ANYWHERE) { handle(ShuffleHistory(MakeDMLHistory() + MakeTCLHistory())); + } else { + assert(tcl_position_ == TclPosition::NOWHERE); + handle(ShuffleHistory(MakeDMLHistory())); } } } @@ -99,11 +100,10 @@ class RandomHistoryGenerator : public HistoryGenerator { return ((int64_t)rand_id <= cur_max_id) ? rand_id : (++cur_max_id); }; for (uint64_t dml_operation_no = 0; dml_operation_no < dml_operation_num_; ++dml_operation_no) { - const Operation::Type dml_type = - rand_bool_(gen_) ? Operation::Type::WRITE : Operation::Type::READ; const uint64_t trans_id = gen_rand_id(rand_trans_id_, max_trans_id); const uint64_t item_id = gen_rand_id(rand_item_id_, max_item_id); - dml_acts.emplace_back(dml_type, trans_id, item_id); + rand_bool_(gen_) ? dml_acts.emplace_back(Operation::ReadTypeConstant(), trans_id, item_id) : + dml_acts.emplace_back(Operation::WriteTypeConstant(), trans_id, item_id); } return History(trans_num_, item_num_, dml_acts); } @@ -112,9 +112,9 @@ class RandomHistoryGenerator : public HistoryGenerator { std::vector dtl_acts; uint64_t abort_trans_num; for (uint64_t trans_id = 0; trans_id < trans_num_; ++trans_id) { - dtl_acts.emplace_back( - (with_abort_ && rand_bool_(gen_)) ? Operation::Type::ABORT : Operation::Type::COMMIT, - trans_id); + with_abort_ && rand_bool_(gen_) ? + dtl_acts.emplace_back(Operation::AbortTypeConstant(), trans_id) : + dtl_acts.emplace_back(Operation::CommitTypeConstant(), trans_id); if (dtl_acts.back().type() == Operation::Type::ABORT) { abort_trans_num++; } @@ -130,7 +130,7 @@ class RandomHistoryGenerator : public HistoryGenerator { const uint64_t dml_operation_num_; const uint64_t history_num_; const bool with_abort_; - const bool tail_tcl_; + const TclPosition tcl_position_; mutable std::random_device rd_; mutable std::mt19937 gen_; mutable std::uniform_int_distribution rand_trans_id_; @@ -147,15 +147,17 @@ class TraversalHistoryGenerator : public HistoryGenerator { subtask_num_(opt.subtask_num), dfs_cnt_(opt.subtask_num - opt.subtask_id), with_abort_(opt.with_abort), - tail_tcl_(opt.tail_tcl), + tcl_position_(opt.tcl_position), allow_empty_trans_(opt.allow_empty_trans), - dynamic_history_len_(opt.dynamic_history_len) {} + dynamic_history_len_(opt.dynamic_history_len), + with_scan_(opt.with_scan), + with_write_(opt.with_write) {} void DeliverHistories(const std::function &handle) const { std::vector tmp_operations; RecursiveFillDMLHistory( - [this, &handle](const History &dml_history, const uint64_t max_trans_id) { - HandleDMLHistory(handle, dml_history, max_trans_id); + [this, &handle](History&& dml_history, const uint64_t max_trans_id) { + HandleDMLHistory(handle, std::move(dml_history), max_trans_id); }, tmp_operations, 0, 0); } @@ -163,29 +165,29 @@ class TraversalHistoryGenerator : public HistoryGenerator { static uint64_t cut_down_; private: - void HandleHistory(const std::function &handle, const History &history) const { - History history_copy = history; // copy History - handle(std::move(history_copy)); - } - - void HandleTCLHistory(const std::function &handle, const History &dml_history, - const History &dtl_history) const { - // Firstly, append every dml_history with dtl_history. + void HandleTCLHistory(const std::function &handle, History&& dml_history, + History&& dtl_history) const { History history_tot = dml_history + dtl_history; - // Then move dtl_history at a suitable location, forward. - RecursiveMoveForwardTCLOperation( - [this, &handle](const History &history) { HandleHistory(handle, history); }, history_tot, - dml_history.size()); + if (tcl_position_ == TclPosition::TAIL) { + handle(std::move(history_tot)); + } else { + assert(tcl_position_ == TclPosition::ANYWHERE); + RecursiveMoveForwardTCLOperation(handle, std::move(history_tot), dml_history.size()); + } } - void HandleDMLHistory(const std::function &handle, const History &dml_history, + void HandleDMLHistory(const std::function &handle, History&& dml_history, const uint64_t max_trans_id) const { - std::vector dtl_operation_types; - RecursiveFillTCLHistory( - [this, &handle, &dml_history](const History &dtl_history) { - HandleTCLHistory(handle, dml_history, dtl_history); - }, - dtl_operation_types, max_trans_id, 0); + std::vector is_commits; + if (tcl_position_ == TclPosition::NOWHERE) { + handle(std::move(dml_history)); + } else { + RecursiveFillTCLHistory( + [this, &handle, &dml_history](History&& dtl_history) { + HandleTCLHistory(handle, std::move(dml_history), std::move(dtl_history)); + }, + is_commits, dml_history.trans_num()); + } } bool OnlyOneTrans(const std::vector &operations) const { @@ -197,78 +199,126 @@ class TraversalHistoryGenerator : public HistoryGenerator { return true; } - void RecursiveFillDMLHistory(const std::function &handle, - std::vector &operations, uint64_t max_trans_id, - uint64_t max_item_id) const { - size_t cur = operations.size(); - if (dynamic_history_len_ || cur == dml_operation_num_) { - if (dfs_cnt_ == subtask_num_) { - if (max_trans_id == trans_num_ || allow_empty_trans_) { - handle(History(max_trans_id, item_num_, operations), max_trans_id); - } else { - cut_down_++; - } - dfs_cnt_ -= subtask_num_; + void RecursiveFillDMLHistoryOver(const std::function &handle, + std::vector &operations, uint64_t max_trans_id, + uint64_t max_item_id) const { + const auto check_has_operation = [this, &operations](const Operation::Type type) { + // check if allow no scan operations + for (const Operation operation : operations) { + if (operation.type() == type) { return true; } } - ++dfs_cnt_; + return false; + }; + + if (dfs_cnt_ == subtask_num_) { + if ((with_scan_ != Intensity::ALL_HAVE || check_has_operation(Operation::Type::SCAN_ODD)) && + (with_write_ != Intensity::ALL_HAVE || check_has_operation(Operation::Type::WRITE)) && // cannot all readonly transactions + (allow_empty_trans_ || max_trans_id == trans_num_)) { + handle(History(max_trans_id, max_item_id, operations), max_trans_id); + } else { + cut_down_++; + } + dfs_cnt_ -= subtask_num_; } - if (cur != dml_operation_num_) { - // Make sure trans id is increment - for (uint64_t trans_id = 0; trans_id < std::min(max_trans_id + 1, trans_num_); ++trans_id) { - for (uint64_t item_id = 0; item_id < std::min(max_item_id + 1, item_num_); ++item_id) { - for (Operation::Type dml_operation_type : - {Operation::Type::READ, Operation::Type::WRITE}) { - // Continuous read in same transaction is meaningless - if (cur > 0 && dml_operation_type == Operation::Type::READ && - dml_operation_type == operations[cur - 1].type() && + ++dfs_cnt_; + } + + void RecursiveFillDMLHistoryContinue(const std::function &handle, + std::vector &operations, uint64_t max_trans_id, + uint64_t max_item_id) const { + const size_t cur = operations.size(); + // Make sure trans id is increment + for (uint64_t trans_id = 0; trans_id < std::min(max_trans_id + 1, trans_num_); ++trans_id) { + for (uint64_t item_id = 0; item_id < std::min(max_item_id + 1, item_num_); ++item_id) { + const auto fill_dml = + [this, cur, trans_id, item_id, max_trans_id, max_item_id, &handle, &operations] + (auto&& type_constant) { + // Continuous same operation is meaningless + if (!(cur > 0 && operations[cur - 1].type() == type_constant.value && trans_id == operations[cur - 1].trans_id() && - item_id == operations[cur - 1].item_id()) { - continue; - } - operations.emplace_back(dml_operation_type, trans_id, item_id); + item_id == operations[cur - 1].item_id())) { + operations.emplace_back(type_constant, trans_id, item_id); RecursiveFillDMLHistory(handle, operations, std::max(trans_id + 1, max_trans_id), std::max(item_id + 1, max_item_id)); operations.pop_back(); } + }; + fill_dml(Operation::ReadTypeConstant()); + if (with_write_ != Intensity::NONE_HAVE) { + fill_dml(Operation::WriteTypeConstant()); } } + if (with_scan_ != Intensity::NONE_HAVE && + !(cur > 0 && operations[cur - 1].type() == Operation::Type::SCAN_ODD && + trans_id == operations[cur - 1].trans_id())) { + operations.emplace_back(Operation::ScanOddTypeConstant(), trans_id); + RecursiveFillDMLHistory(handle, operations, std::max(trans_id + 1, max_trans_id), max_item_id); + operations.pop_back(); + } } } - void RecursiveFillTCLHistory(const std::function &handle, - std::vector &dtl_operation_types, - const uint64_t max_trans_id, uint64_t abort_trans_num) const { - if (!with_abort_) { - dtl_operation_types.assign(max_trans_id, Operation::Type::COMMIT); + // Generate all possible DML histories and pass each history to handle. + void RecursiveFillDMLHistory(const std::function &handle, + std::vector &operations, uint64_t max_trans_id, + uint64_t max_item_id) const { + if (dynamic_history_len_ || operations.size() == dml_operation_num_) { + RecursiveFillDMLHistoryOver(handle, operations, max_trans_id, max_item_id); } - if (dtl_operation_types.size() == max_trans_id) { - std::vector trans_order; - for (uint64_t trans_id = 0; trans_id < max_trans_id; ++trans_id) { - trans_order.push_back(trans_id); + if (operations.size() != dml_operation_num_) { + RecursiveFillDMLHistoryContinue(handle, operations, max_trans_id, max_item_id); + } + } + + void RecursiveFillTCLHistoryOver(const std::function &handle, + const std::vector &is_commits) const { + std::vector dtl_operations; + uint64_t abort_trans_num = 0; + const uint64_t trans_num = is_commits.size(); + for (uint64_t trans_id = 0; trans_id < trans_num; ++trans_id) { + if (is_commits[trans_id]) { + dtl_operations.emplace_back(Operation::CommitTypeConstant(), trans_id); + } else { + ++ abort_trans_num; + dtl_operations.emplace_back(Operation::AbortTypeConstant(), trans_id); } - do { - std::vector dtl_operations; - for (uint64_t trans_id : trans_order) { - dtl_operations.emplace_back(dtl_operation_types[trans_id], trans_id); - } - handle(History(max_trans_id, item_num_, dtl_operations, abort_trans_num)); - } while (std::next_permutation(trans_order.begin(), trans_order.end())); + } + // traverse all possible transaction commit/abort order + do { + handle(History(trans_num, 0 /* DTL history has no items */, dtl_operations, abort_trans_num)); + } while (std::next_permutation(dtl_operations.begin(), dtl_operations.end())); + } + + void RecursiveFillTCLHistoryContinue(const std::function &handle, + std::vector &is_commits, const uint64_t trans_num) const { + // traverse all possible transaction commit/abort states + for (const bool is_commit : { true, false }) { + is_commits.emplace_back(is_commit); + RecursiveFillTCLHistory(handle, is_commits, trans_num); + is_commits.pop_back(); + } + } + + // Generate all possible TCL histories and pass each history to handle. + void RecursiveFillTCLHistory(const std::function &handle, + std::vector &is_commits, const uint64_t trans_num) const { + if (!with_abort_) { + is_commits.assign(trans_num, true); + } + if (is_commits.size() == trans_num) { + RecursiveFillTCLHistoryOver(handle, is_commits); } else { - for (Operation::Type dtl_operation_type : {Operation::Type::COMMIT, Operation::Type::ABORT}) { - dtl_operation_types.emplace_back(dtl_operation_type); - if (dtl_operation_type == Operation::Type::ABORT) { - ++abort_trans_num; - } - RecursiveFillTCLHistory(handle, dtl_operation_types, max_trans_id, abort_trans_num); - dtl_operation_types.pop_back(); - } + RecursiveFillTCLHistoryContinue(handle, is_commits, trans_num); } } - void RecursiveMoveForwardTCLOperation(const std::function &handle, - History &history, const size_t pos) const { - if (pos == history.size() || tail_tcl_) { - handle(history); + // Remix the DML operations and tailing TCL operations. + template + void RecursiveMoveForwardTCLOperation(const std::function &handle, + HistoryRef&& history, const size_t pos) const { + if (pos == history.size() || tcl_position_ == TclPosition::TAIL) { + // copy the history because we will go on moving forward tcl in this history + handle(History(history)); } else { RecursiveMoveForwardTCLOperation(handle, history, pos + 1); size_t i = pos; @@ -291,9 +341,11 @@ class TraversalHistoryGenerator : public HistoryGenerator { const uint64_t subtask_num_; mutable uint64_t dfs_cnt_; const bool with_abort_; - const bool tail_tcl_; + const TclPosition tcl_position_; const bool allow_empty_trans_; const bool dynamic_history_len_; + const Intensity with_scan_; + const Intensity with_write_; }; uint64_t TraversalHistoryGenerator::cut_down_ = 0; } // namespace ttts diff --git a/src/3ts/backend/history/outputter.h b/src/3ts/backend/history/outputter.h index 3f97c5f4..89e79e19 100644 --- a/src/3ts/backend/history/outputter.h +++ b/src/3ts/backend/history/outputter.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * diff --git a/src/3ts/backend/history/parse_config.h b/src/3ts/backend/history/parse_config.h index f0ecd4bd..e2db90c0 100644 --- a/src/3ts/backend/history/parse_config.h +++ b/src/3ts/backend/history/parse_config.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: elioyan@tencent.com * williamcliu@tencent.com @@ -17,12 +15,25 @@ #include "../cca/occ_algorithm/trans/focc_trans.h" #include "../cca/occ_algorithm/trans/ssi_trans.h" #include "../cca/occ_algorithm/trans/wsi_trans.h" +#include "../cca/occ_algorithm/trans/dli_trans.h" #include "../cca/serializable_algorithm.h" +#include "../cca/unified_history_algorithm.h" #include "../util/generic.h" #include "generator.h" #include "outputter.h" #include "run.h" +template +EnumType EnumParse(const std::string& s) +{ + for (const auto e : Members()) { + if (s == ToString(e)) { + return e; + } + } + throw std::string("Parse Enum failed: ") + s + " type:" + typeid(EnumType).name(); +} + std::shared_ptr GeneratorParse(const libconfig::Config &cfg, const std::string &name) { try { @@ -37,9 +48,11 @@ std::shared_ptr GeneratorParse(const libconfig::Config & opt.item_num = s.lookup("item_num"); opt.max_dml = s.lookup("max_dml"); opt.with_abort = s.lookup("with_abort"); - opt.tail_tcl = s.lookup("tail_tcl"); + opt.tcl_position = EnumParse(s.lookup("tcl_position")); opt.dynamic_history_len = s.lookup("dynamic_history_len"); opt.allow_empty_trans = s.lookup("allow_empty_trans"); + opt.with_scan = EnumParse(s.lookup("with_scan")); + opt.with_write = EnumParse(s.lookup("with_write")); if (name == "TraversalGenerator") { opt.subtask_num = s.lookup("subtask_num"); opt.subtask_id = s.lookup("subtask_id"); @@ -67,6 +80,8 @@ void AlgorithmParseInternal_(const libconfig::Config &cfg, const std::string &al add_algorithm(std::make_shared>()); } else if (algorithm_name == "FOCC") { add_algorithm(std::make_shared>()); + } else if (algorithm_name == "DLI") { + add_algorithm(std::make_shared>()); } else if constexpr (only_rollback_rate) { throw "Unknown algorithm name " + algorithm_name + " in algorithms supporting rollback rate statistics"; @@ -110,7 +125,13 @@ void AlgorithmParseInternal_(const libconfig::Config &cfg, const std::string &al std::make_shared>()); } else if (algorithm_name == "ConflictSerializableAlgorithm") { - add_algorithm(std::make_shared()); + add_algorithm(std::make_shared>()); + } else if (algorithm_name == "DLI_IDENTIFY") { + add_algorithm(std::make_shared>()); + } else if (algorithm_name == "DLI_IDENTIFY_CYCLE") { + add_algorithm(std::make_shared>()); + } else if (algorithm_name == "DLI_IDENTIFY_CHAIN") { + add_algorithm(std::make_shared>()); } else { throw "Unknown algorithm name " + algorithm_name; } @@ -250,12 +271,12 @@ void BenchmarkRunParse(const libconfig::Config &cfg) { const uint64_t history_num = s.lookup("history_num"); const std::string os = s.lookup("os"); const bool with_abort = s.lookup("with_abort"); - const bool tail_tcl = s.lookup("tail_tcl"); + const TclPosition tcl_position = EnumParse(s.lookup("tcl_position")); if (os == "cout") - BenchmarkRun(trans_nums, item_nums, history_num, algorithms, std::cout, with_abort, tail_tcl); + BenchmarkRun(trans_nums, item_nums, history_num, algorithms, std::cout, with_abort, tcl_position); else BenchmarkRun(trans_nums, item_nums, history_num, algorithms, std::ofstream(os), with_abort, - tail_tcl); + tcl_position); } catch (const libconfig::SettingNotFoundException &nfex) { throw "Func BenchmarkRun setting " + std::string(nfex.getPath()) + " no found"; } diff --git a/src/3ts/backend/history/run.h b/src/3ts/backend/history/run.h index f4ab037d..82af6cdf 100644 --- a/src/3ts/backend/history/run.h +++ b/src/3ts/backend/history/run.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * @@ -95,6 +93,11 @@ void FilterRun( signal(SIGTERM, handler); outs = outputters; ThreadRunBase(generator, task, thread_num); + for (const auto& [variant_alg, _] : algorithms) { + std::visit([](auto&& alg){ + alg->Statistics(); + }, variant_alg); + } } // Each algorithm check the histories and record the time cost. @@ -130,10 +133,10 @@ template void BenchmarkRun(const std::vector &trans_nums, const std::vector &item_nums, const uint64_t num, const std::vector> &algorithms, OS &&os, - const bool with_abort, const bool tail_tcl) { + const bool with_abort, const TclPosition tcl_position) { Options opts; opts.with_abort = with_abort; - opts.tail_tcl = tail_tcl; + opts.tcl_position = tcl_position; for (const uint64_t trans_num : trans_nums) { for (const uint64_t item_num : item_nums) { const uint64_t dml_operation_num = trans_num * item_num / 4; diff --git a/src/3ts/backend/main.cc b/src/3ts/backend/main.cc index e46eaa6b..f843f4e0 100644 --- a/src/3ts/backend/main.cc +++ b/src/3ts/backend/main.cc @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * diff --git a/src/3ts/backend/shape.h b/src/3ts/backend/shape.h new file mode 100644 index 00000000..4263ec0f --- /dev/null +++ b/src/3ts/backend/shape.h @@ -0,0 +1,38 @@ +#include +#include +#include + +float f(float x, float y, float z) { + float a = x * x + 9.0f / 4.0f * y * y + z * z - 1; + return a * a * a - x * x * z * z * z - 9.0f / 80.0f * y * y * z * z * z; +} + +float h(float x, float z) { + for (float y = 1.0f; y >= 0.0f; y -= 0.001f) { + if (f(x, y, z) <= 0.0f) { + return y; + } + } + return 0.0f; +} + +void draw() { + for (float z = 1.5f; z > -1.5f; z -= 0.05f) { + for (float x = -1.5f; x < 1.5f; x += 0.025f) { + float v = f(x, 0.0f, z); + if (v <= 0.0f) { + float y0 = h(x, z); + float ny = 0.01f; + float nx = h(x + ny, z) - y0; + float nz = h(x, z + ny) - y0; + float nd = 1.0f / sqrtf(nx * nx + ny * ny + nz * nz); + float d = (nx + ny - nz) * nd * 0.5f + 0.5f; + putchar("*********"[(int)(d * 5.0f)]); + } else { + putchar(' '); + } + } + putchar('\n'); + } + std::cout << "Congratulations on finding the Easter egg! Here's a gift from us for you!" << std::endl;; +} diff --git a/src/3ts/backend/util/extend_enum.h b/src/3ts/backend/util/extend_enum.h new file mode 100644 index 00000000..2d4b8dec --- /dev/null +++ b/src/3ts/backend/util/extend_enum.h @@ -0,0 +1,57 @@ +#ifdef ENUM_FILE + +template >> inline constexpr uint32_t Count(); +template >> inline const char* ToString(const EnumType e); +template >> inline const std::array()>& Members(); + +#define ENUM_BEGIN(name) enum class name : uint32_t { +#define ENUM_MEMBER(_, member) member, +#define ENUM_END(name) name##_MAX }; template <> constexpr uint32_t Count() { return static_cast(name::name##_MAX); } + +#include ENUM_FILE + +#undef ENUM_BEGIN +#undef ENUM_MEMBER +#undef ENUM_END + +#define ENUM_BEGIN(name)\ +template <> inline const char* ToString(const name e)\ +{\ + static std::array()> strings { +#define ENUM_MEMBER(_, member) #member, +#define ENUM_END(name)\ + };\ + return strings.at(static_cast(e));\ +}\ + +#include ENUM_FILE + +#undef ENUM_BEGIN +#undef ENUM_MEMBER +#undef ENUM_END + +#define ENUM_BEGIN(name)\ +template <> inline const std::array()>& Members()\ +{\ + static const std::array()> members { +#define ENUM_MEMBER(name, member) name::member, +#define ENUM_END(_)\ + };\ + return members;\ +} + +#include ENUM_FILE + +#undef ENUM_BEGIN +#undef ENUM_MEMBER +#undef ENUM_END + +#undef ENUM_FILE +#endif + +#ifndef EXTEND_ENUM_H_ +#define EXTEND_ENUM_H_ + +template >> inline std::ostream& operator<<(std::ostream& os, const EnumType e) { return os << ToString(e); } + +#endif diff --git a/src/3ts/backend/util/generic.h b/src/3ts/backend/util/generic.h index 75f2961a..61a2454f 100644 --- a/src/3ts/backend/util/generic.h +++ b/src/3ts/backend/util/generic.h @@ -1,14 +1,34 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * */ -#pragma once + +#ifdef ENUM_BEGIN +#ifdef ENUM_MEMBER +#ifdef ENUM_END + +ENUM_BEGIN(Intensity) +ENUM_MEMBER(Intensity, NONE_HAVE) +ENUM_MEMBER(Intensity, ALL_HAVE) +ENUM_MEMBER(Intensity, NO_LIMIT) +ENUM_END(Intensity) + +ENUM_BEGIN(TclPosition) +ENUM_MEMBER(TclPosition, TAIL) +ENUM_MEMBER(TclPosition, ANYWHERE) +ENUM_MEMBER(TclPosition, NOWHERE) +ENUM_END(TclPosition) + +#endif +#endif +#endif + +#ifndef UTIL_GENERIC_H +#define UTIL_GENERIC_H #include #include @@ -110,19 +130,24 @@ class Operation { ABORT = 'A', SCAN_ODD = 'S' }; + using ReadTypeConstant = std::integral_constant; + using WriteTypeConstant = std::integral_constant; + using CommitTypeConstant = std::integral_constant; + using AbortTypeConstant = std::integral_constant; + using ScanOddTypeConstant = std::integral_constant; Operation() : type_(Type::UNKNOWN), trans_id_(0) {} - Operation(const Type dtl_type, const uint64_t trans_id) : type_(dtl_type), trans_id_(trans_id) { - if (!IsTCL()) { - throw std::to_string(static_cast(dtl_type)) + " is not a TCL Operation"; - } - } - Operation(const Type dml_type, const uint64_t trans_id, const uint64_t item_id, - const std::optional version = {}) - : type_(dml_type), trans_id_(trans_id), item_id_(item_id), version_(version) { - if (!IsPointDML()) { - throw std::to_string(static_cast(dml_type)) + " is not a Point DML Operation"; - } - } + Operation(const std::integral_constant dtl_type, const uint64_t trans_id) + : type_(dtl_type.value), trans_id_(trans_id) {} + Operation(const std::integral_constant dtl_type, const uint64_t trans_id) + : type_(dtl_type.value), trans_id_(trans_id) {} + Operation(const std::integral_constant dtl_type, const uint64_t trans_id) + : type_(dtl_type.value), trans_id_(trans_id) {} + Operation(const std::integral_constant dml_type, const uint64_t trans_id, + const uint64_t item_id, const std::optional version = {}) + : type_(dml_type.value), trans_id_(trans_id), item_id_(item_id), version_(version) {} + Operation(const std::integral_constant dml_type, const uint64_t trans_id, + const uint64_t item_id, const std::optional version = {}) + : type_(dml_type.value), trans_id_(trans_id), item_id_(item_id), version_(version) {} Operation& operator=(const Operation& operation) = default; virtual ~Operation() {} @@ -177,18 +202,31 @@ class Operation { break; default: type = Operation::Type::UNKNOWN; - is.setstate(std::ios::failbit); + if (c != '\0') { + std::cerr << "Unknonw operation type character: " << c << ". Supported operations: R W C A" << std::endl; + } } return is; } friend std::istream& operator>>(std::istream& is, Operation& operation) { - if (!(is >> operation.type_) || !(is >> operation.trans_id_)) { + if (!(is >> operation.type_)) { + is.setstate(std::ios::failbit); return is; } + if (!(is >> operation.trans_id_)) { + is.setstate(std::ios::failbit); + std::cerr << "Transaction ID character must be a number" << std::endl; + return is; + } else if (operation.trans_id_ >= 10 || operation.trans_id_ < 0) { + is.setstate(std::ios::failbit); + std::cerr << "Transaction ID must be less than 10 and more than 0." << std::endl; + return is; + } if (char item_c; operation.type_ == Type::WRITE || operation.type_ == Type::READ) { if (!(is >> item_c) || !std::islower(item_c)) { is.setstate(std::ios::failbit); + std::cerr << "Data Item must be lowercase letter" << std::endl; return is; } operation.item_id_ = item_c - 'a'; @@ -253,15 +291,12 @@ class History { History& operator=(History&& history) = default; History operator+(const History& history) const { - if (trans_num_ != history.trans_num_ || item_num_ != history.item_num_) { - throw "History mismatch"; - } std::vector new_operations = operations_; for (const auto& operation : history.operations_) { new_operations.push_back(operation); } - return History(trans_num_, item_num_, std::move(new_operations), - abort_trans_num_ + history.abort_trans_num_); + return History(std::max(trans_num_, history.trans_num_), std::max(item_num_, history.item_num_), + std::move(new_operations), abort_trans_num_ + history.abort_trans_num_); } std::vector& operations() { return operations_; } const std::vector& operations() const { return operations_; } @@ -283,22 +318,28 @@ class History { if (std::getline(is, s)) { std::stringstream ss(s); std::vector operations; - std::set trans_num_set; - std::set item_num_set; uint64_t trans_num = 0; uint64_t item_num = 0; + std::unordered_map trans_num_map; + std::unordered_map item_num_map; for (std::stringstream ss(s); !ss.eof() && !ss.fail();) { Operation operation; if (Operation operation; ss >> operation) { - operations.emplace_back(operation); - trans_num_set.insert(operation.trans_id()); + if (trans_num_map.count(operation.trans_id()) == 0) { + trans_num_map[operation.trans_id()] = trans_num_map.size(); + } + operation.SetTransId(trans_num_map[operation.trans_id()]); if (operation.IsPointDML()) { - item_num_set.insert(operation.item_id()); + if (item_num_map.count(operation.item_id()) == 0) { + item_num_map[operation.item_id()] = item_num_map.size(); + } + operation.SetItemId(item_num_map[operation.item_id()]); } + operations.emplace_back(operation); } } - trans_num = trans_num_set.size(); - item_num = item_num_set.size(); + trans_num = trans_num_map.size(); + item_num = item_num_map.size(); if (ss.fail()) { std::cout << "Invalid history: \'" << s << "\'" << std::endl; } else { @@ -339,6 +380,9 @@ class History { std::string anomaly_name_; }; +#define ENUM_FILE "./generic.h" +#include "extend_enum.h" + struct Options { uint64_t trans_num; uint64_t item_num; @@ -348,9 +392,13 @@ struct Options { uint64_t max_dml; bool with_abort; - bool tail_tcl; + TclPosition tcl_position; bool allow_empty_trans; bool dynamic_history_len; + + Intensity with_scan; + Intensity with_write; }; } // namespace ttts +#endif diff --git a/src/3ts/backend/util/generic_backup.h b/src/3ts/backend/util/generic_backup.h new file mode 100644 index 00000000..9fd3130e --- /dev/null +++ b/src/3ts/backend/util/generic_backup.h @@ -0,0 +1,411 @@ +/* + * Tencent is pleased to support the open source community by making 3TS available. + * + * Copyright (C) 2020 Tencent. All rights reserved. + * + * Author: williamcliu@tencent.com + * + */ + +#ifdef ENUM_BEGIN +#ifdef ENUM_MEMBER +#ifdef ENUM_END + +ENUM_BEGIN(Intensity) +ENUM_MEMBER(Intensity, NONE_HAVE) +ENUM_MEMBER(Intensity, ALL_HAVE) +ENUM_MEMBER(Intensity, NO_LIMIT) +ENUM_END(Intensity) + +ENUM_BEGIN(TclPosition) +ENUM_MEMBER(TclPosition, TAIL) +ENUM_MEMBER(TclPosition, ANYWHERE) +ENUM_MEMBER(TclPosition, NOWHERE) +ENUM_END(TclPosition) + +#endif +#endif +#endif + +#ifndef UTIL_GENERIC_H +#define UTIL_GENERIC_H +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ttts { + +enum Anomally { + // I, II + DIRTY_WRITE, + EDGE_CROESS, + LOST_UPDATE, + READ_SKEW, + READ_WRITE_SKEW, + THREE_TRANS_WRITE_SKEW, + MULTI_TRANS_ANOMALY, + // WSI + WRITE_SKEW, + WW_CONFLICT, + // SSI + RW_CONFLICT, + // BOCC FOCC UNKNOWN + UNKNOWN +}; + +std::unordered_map Anomally2Name = { + // I, II + {DIRTY_WRITE, "DIRTY_WRITE"}, + {EDGE_CROESS, "EDGE_CROESS"}, + {LOST_UPDATE, "LOST_UPDATE"}, + {READ_SKEW, "READ_SKEW"}, + {READ_WRITE_SKEW, "READ_WRITE_SKEW"}, + {THREE_TRANS_WRITE_SKEW, "THREE_TRANS_WRITE_SKEW"}, + {MULTI_TRANS_ANOMALY, "MULTI_TRANS_ANOMALY"}, + // WSI + {WRITE_SKEW, "WRITE_SKEW"}, + {WW_CONFLICT, "WW_CONFLICT"}, + // SSI + {RW_CONFLICT, "RW_CONFLICT"}, + // BOCC FOCC UNKNOWN + {UNKNOWN, "UNKNOWN"}}; + +std::ostream& operator<<(std::ostream& os, const Anomally e) { + switch (e) { + case DIRTY_WRITE: + return os << "DIRTY_WRITE"; + case EDGE_CROESS: + return os << "EDGE_CROESS"; + case LOST_UPDATE: + return os << "LOST_UPDATE"; + case READ_SKEW: + return os << "READ_SKEW"; + case READ_WRITE_SKEW: + return os << "READ_WRITE_SKEW"; + case THREE_TRANS_WRITE_SKEW: + return os << "THREE_TRANS_WRITE_SKEW"; + case MULTI_TRANS_ANOMALY: + return os << "MULTI_TRANS_ANOMALY"; + case WRITE_SKEW: + return os << "WRITE_SKEW"; + case WW_CONFLICT: + return os << "WW_CONFLICT"; + case RW_CONFLICT: + return os << "RW_CONFLICT"; + default: + return os << "UNKNOWN"; + } +} + +enum class SerializeReadPolicy { UNCOMMITTED_READ, COMMITTED_READ, REPEATABLE_READ, SI_READ }; + +class Operation { + public: + enum class Type : char { + UNKNOWN = '?', + READ = 'R', + WRITE = 'W', + COMMIT = 'C', + ABORT = 'A', + SCAN_ODD = 'S' + }; + using ReadTypeConstant = std::integral_constant; + using WriteTypeConstant = std::integral_constant; + using CommitTypeConstant = std::integral_constant; + using AbortTypeConstant = std::integral_constant; + using ScanOddTypeConstant = std::integral_constant; + Operation() : type_(Type::UNKNOWN), trans_id_(0) {} + Operation(const std::integral_constant dtl_type, const uint64_t trans_id) + : type_(dtl_type.value), trans_id_(trans_id) {} + Operation(const std::integral_constant dtl_type, const uint64_t trans_id) + : type_(dtl_type.value), trans_id_(trans_id) {} + Operation(const std::integral_constant dtl_type, const uint64_t trans_id) + : type_(dtl_type.value), trans_id_(trans_id) {} + Operation(const std::integral_constant dml_type, const uint64_t trans_id, + const uint64_t item_id, const std::optional version = {}) + : type_(dml_type.value), trans_id_(trans_id), item_id_(item_id), version_(version) {} + Operation(const std::integral_constant dml_type, const uint64_t trans_id, + const uint64_t item_id, const std::optional version = {}) + : type_(dml_type.value), trans_id_(trans_id), item_id_(item_id), version_(version) {} + Operation& operator=(const Operation& operation) = default; + virtual ~Operation() {} + + Type type() const { return type_; } + uint64_t trans_id() const { return trans_id_; } + uint64_t item_id() const { return item_id_.value(); } + uint64_t version() const { return version_.value(); } + void SetTransId(uint64_t trans_id) { trans_id_ = trans_id; } + void SetItemId(const uint64_t item_id) { + if (IsTCL()) { + throw "TCL operations update item id is meaningless"; + } + item_id_ = item_id; + } + void UpdateVersion(const uint64_t version) { + if (IsTCL()) { + throw "DML operations update version is meaningless"; + } + version_ = version; + } + + friend std::ostream& operator<<(std::ostream& os, const Operation& operation) { + os << static_cast(operation.type_) << operation.trans_id_; + if (operation.IsPointDML()) { + if (operation.item_id_.value() >= 26) { + throw "Not support item_id equal or larger than 26 yet"; + } + os << static_cast('a' + operation.item_id_.value()); + } + + return os; + } + + friend std::istream& operator>>(std::istream& is, Type& type) { + char c; + is >> c; + switch (c) { + case 'W': + type = Operation::Type::WRITE; + break; + case 'R': + type = Operation::Type::READ; + break; + case 'C': + type = Operation::Type::COMMIT; + break; + case 'A': + type = Operation::Type::ABORT; + break; + case 'S': + type = Operation::Type::SCAN_ODD; + break; + default: + type = Operation::Type::UNKNOWN; + if (c != '\0') { + std::cerr << "Unknonw operation type character: " << c << ". Supported operations: R W C A" << std::endl; + } + is.setstate(std::ios::failbit); + return is; + } + return is; + } + + friend std::istream& operator>>(std::istream& is, Operation& operation) { + if (!(is >> operation.type_)) { + is.setstate(std::ios::failbit); + return is; + } + if (!(is >> operation.trans_id_)) { + is.setstate(std::ios::failbit); + std::cerr << "Transaction ID character must be a number" << std::endl; + return is; + } + if (char item_c; operation.type_ == Type::WRITE || operation.type_ == Type::READ) { + if (!(is >> item_c) || !std::islower(item_c)) { + is.setstate(std::ios::failbit); + std::cerr << "Data Item must be lowercase letter" << std::endl; + return is; + } + operation.item_id_ = item_c - 'a'; + } + return is; + } + + bool IsPointDML() const { return IsPointDML(type_); } + bool IsTCL() const { return IsTCL(type_); } + static bool IsPointDML(const Type& type) { return type == Type::READ || type == Type::WRITE; } + static bool IsTCL(const Type& type) { return type == Type::COMMIT || type == Type::ABORT; } + // surport std::map + bool operator<(const Operation& r) const { + if (trans_id() == r.trans_id()) { + uint64_t l_item_id = item_id_.has_value() ? item_id() : -1; + uint64_t r_item_id = r.item_id_.has_value() ? r.item_id() : -1; + if (l_item_id == r_item_id) { + uint64_t l_version = version_.has_value() ? version() : -1; + uint64_t r_version = r.version_.has_value() ? r.version() : -1; + if (l_version == r_version) return type() < r.type(); + return l_version < r.version(); + } + return l_item_id < r_item_id; + } + return trans_id() < r.trans_id(); + } + + private: + Type type_; + uint64_t trans_id_; + std::optional item_id_; + std::optional version_; // version_ identify a unique version, but it CANNOT be + // compared to judge new or old +}; + +class History { + public: + History() : History(0, 0, {}) {} + History(const uint64_t trans_num, const uint64_t item_num, + const std::vector& operations) + : trans_num_(trans_num), + item_num_(item_num), + operations_(operations), + abort_trans_num_(0), + anomaly_name_("") {} + History(const uint64_t trans_num, const uint64_t item_num, std::vector&& operations) + : trans_num_(trans_num), + item_num_(item_num), + operations_(operations), + abort_trans_num_(0), + anomaly_name_("") {} + History(const uint64_t trans_num, const uint64_t item_num, + const std::vector& operations, const uint64_t abort_trans_num) + : trans_num_(trans_num), + item_num_(item_num), + operations_(operations), + abort_trans_num_(abort_trans_num), + anomaly_name_("") {} + History(History&& history) = default; + History(const History& history) = default; + ~History() {} + + History& operator=(History&& history) = default; + History operator+(const History& history) const { + std::vector new_operations = operations_; + for (const auto& operation : history.operations_) { + new_operations.push_back(operation); + } + return History(std::max(trans_num_, history.trans_num_), std::max(item_num_, history.item_num_), + std::move(new_operations), abort_trans_num_ + history.abort_trans_num_); + } + std::vector& operations() { return operations_; } + const std::vector& operations() const { return operations_; } + uint64_t trans_num() const { return trans_num_; } + uint64_t abort_trans_num() const { return abort_trans_num_; } + uint64_t item_num() const { return item_num_; } + size_t size() const { return operations_.size(); } + void set_anomaly_name(const std::string& anomaly_name) { anomaly_name_ = anomaly_name; } + std::string anomaly_name() const { return anomaly_name_; } + friend std::ostream& operator<<(std::ostream& os, const History& history) { + for (const Operation& operation : history.operations_) { + os << operation << ' '; + } + return os; + } + + friend std::istream& operator>>(std::istream& is, History& history) { + std::string s; + if (std::getline(is, s)) { + std::stringstream ss(s); + std::vector operations; + uint64_t trans_num = 0; + uint64_t item_num = 0; + //std::unordered_map trans_num_map; + //std::unordered_map item_num_map; + std::set trans_num_set; + std::set item_num_set; + for (std::stringstream ss(s); !ss.eof() && !ss.fail();) { + Operation operation; + if (Operation operation; ss >> operation) { + operations.emplace_back(operation); + trans_num_set.insert(operation.trans_id()); + /* + if (trans_num_map.count(operation.trans_id()) == 0) { + trans_num_map[operation.trans_id()] = trans_num_map.size(); + operation.SetTransId(trans_num_map[operation.trans_id()]); + }*/ + if (operation.IsPointDML()) { + item_num_set.insert(operation.item_id()); + /* + if (item_num_map.count(operation.item_id()) == 0) { + item_num_map[operation.item_id()] = item_num_map.size(); + //operation.SetItemId(item_num_map[operation.item_id()]); + }*/ + } + } + //operations.emplace_back(operation); + } + trans_num = trans_num_set.size(); + item_num = item_num_set.size(); + //trans_num = trans_num_map.size(); + //item_num = item_num_map.size(); + if (ss.fail()) { + std::cout << "Invalid history: \'" << s << "\'" << std::endl; + } else { + history = History(trans_num, item_num, operations); + } + } + return is; + } + + Operation& operator[](const size_t index) { return operations_[index]; } + + void UpdateWriteVersions() { + std::vector item_version(item_num_, 0); + for (Operation& operation : operations_) { + if (operation.type() == Operation::Type::WRITE) { + operation.UpdateVersion(++item_version[operation.item_id()]); + } + } + } + + // update write version, clean up read version + void FillWriteVersions() { + std::vector item_version(item_num_, 0); + for (Operation& operation : operations_) { + if (operation.type() == Operation::Type::WRITE) { + operation.UpdateVersion(++item_version[operation.item_id()]); + } else if (operation.type() == Operation::Type::READ) { + operation.UpdateVersion(-1); // clear read version as -1 + } + } + } + + private: + uint64_t trans_num_; + uint64_t abort_trans_num_; + uint64_t item_num_; + std::vector operations_; + std::string anomaly_name_; +}; + +#define ENUM_FILE "./generic.h" +#include "extend_enum.h" + +struct Options { + uint64_t trans_num; + uint64_t item_num; + + uint64_t subtask_num; + uint64_t subtask_id; + uint64_t max_dml; + + bool with_abort; + TclPosition tcl_position; + bool allow_empty_trans; + bool dynamic_history_len; + + Intensity with_scan; + Intensity with_write; +}; + +} // namespace ttts +#endif diff --git a/src/3ts/backend/util/thread_pool.h b/src/3ts/backend/util/thread_pool.h index e232e910..0529d1bf 100644 --- a/src/3ts/backend/util/thread_pool.h +++ b/src/3ts/backend/util/thread_pool.h @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: williamcliu@tencent.com * diff --git a/src/test/unittest/base_test.cc b/src/test/unittest/base_test.cc index e7a63121..66803e2d 100644 --- a/src/test/unittest/base_test.cc +++ b/src/test/unittest/base_test.cc @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: moggma@tencent.com * diff --git a/src/test/unittest/ssi_test/ssi_checker_test.cc b/src/test/unittest/ssi_test/ssi_checker_test.cc index 87c01352..25da27f2 100644 --- a/src/test/unittest/ssi_test/ssi_checker_test.cc +++ b/src/test/unittest/ssi_test/ssi_checker_test.cc @@ -1,9 +1,7 @@ /* * Tencent is pleased to support the open source community by making 3TS available. * - * Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved. The below software - * in this distribution may have been modified by THL A29 Limited ("Tencent Modifications"). All - * Tencent Modifications are Copyright (C) THL A29 Limited. + * Copyright (C) 2020 Tencent. All rights reserved. * * Author: moggma@tencent.com *