Skip to content

fix some bugs and process more mysql data types #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 84 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
fa91bde
modify src/CMakeLists.txt to add -fPIC when compiling static library …
ideal May 13, 2013
43255c7
comment a cout
ideal May 13, 2013
c3d0824
Merge pull request #2 from ideal/master
ideal May 13, 2013
4df67c9
add install tips to readme and update cmakelists
ideal May 14, 2013
af6f499
Merge pull request #3 from ideal/master
ideal May 14, 2013
f414d4b
ignore asio files when installing
ideal May 14, 2013
2fc0f80
Merge pull request #4 from ideal/master
ideal May 14, 2013
c38e611
fix enum,string bugs
ideal May 19, 2013
6ea807c
remove some print and change m_type of vale when needed
ideal May 19, 2013
c65fd28
Merge pull request #5 from ideal/master
ideal May 19, 2013
9559c76
remove unused variable
ideal May 19, 2013
cd53f6c
add enum and set data type
ideal May 20, 2013
09dcc9a
fix problem
ideal May 20, 2013
ac55b6e
Merge pull request #6 from ideal/master
ideal May 20, 2013
c05e94e
processing new decimal type, not ready
ideal May 22, 2013
ca79782
processing new decimal type, not ready
ideal May 22, 2013
1ff7c8c
check m_waiting_event
ideal May 23, 2013
eaef4c9
temp fix a problem...
ideal May 23, 2013
c277ac3
Merge pull request #7 from ideal/master
ideal May 26, 2013
0ffb410
some change...
ideal May 29, 2013
32d6fc4
Merge pull request #8 from ideal/master
ideal May 29, 2013
6f8a96d
add date type
ideal May 29, 2013
8ae0cfb
fix ashamed problem :(
ideal May 29, 2013
a9a57dc
Merge pull request #9 from ideal/master
ideal May 29, 2013
6b9bd68
add time and year type
ideal May 29, 2013
2b5d609
change master_id in COM_REGISTER_SLAVE
ideal May 31, 2013
2751827
add new decimal data type
ideal Jun 1, 2013
fc5fbdf
Merge branch 'master' of github.com:ideal/mysql-replication-listener
ideal Jun 1, 2013
ad89a9a
Merge pull request #10 from ideal/master
ideal Jun 1, 2013
7fd3800
Change version number.
ideal Jun 2, 2013
164d6be
change version number
ideal Jun 2, 2013
43c7a97
Merge pull request #11 from ideal/master
ideal Jun 2, 2013
68ab25b
add extern C declaration
ideal Jun 14, 2013
66f090f
Build error fix on latest CMAKE on UBUNTU
rami-dabain Jul 1, 2013
600a0b9
Merge pull request #13 from rami-dabain/master
ideal Jul 1, 2013
08eebf1
Merge pull request #14 from ideal/master
ideal Jul 1, 2013
01bf5bc
change cmake minimum to 2.8 and format in src/CMakeLists.txt
ideal Jul 1, 2013
df0c774
revert src/tcp_driver.cpp
ideal Jul 1, 2013
0a1f3c5
update formats and something else
guweigang Aug 30, 2013
b3a2c73
fix bug in computing size of field char
ideal Sep 2, 2013
838a214
fix mysql_type_char
guweigang Sep 3, 2013
193e675
revert mysql_type_set and mysql_type_enum
guweigang Sep 3, 2013
ae5d7f6
fix bug in char column
ideal Sep 3, 2013
d848c4f
merge code
ideal Sep 3, 2013
8b548f7
change format and change declaration of metadata to uint16_t
ideal Sep 4, 2013
590da72
change as_c_str
guweigang Sep 6, 2013
ac43cae
update for as_c_str
guweigang Sep 6, 2013
ea1565f
change CMakeLists to copy asio to bianry/include
guweigang Sep 12, 2013
9357d34
add notice for usage
guweigang Sep 12, 2013
65499f4
add notice for usage
guweigang Sep 12, 2013
057cee0
add notice for usage
guweigang Sep 12, 2013
dada973
add notice for usage
guweigang Sep 12, 2013
a9284e5
add disconnect and server_id
guweigang Sep 22, 2013
46e6d60
fillfull tcp_driver:disconnect
guweigang Sep 22, 2013
4e152fe
fix memory leak in result_set destructor
guweigang Sep 24, 2013
c67a88e
Merge branch 'master' of https://github.com/BullSoft/mysql-replicatio…
ideal Sep 27, 2013
b83e09b
it is better using while()
ideal Sep 27, 2013
48098d2
Update README.md
Sep 29, 2013
e1bda44
Update README.md
Sep 30, 2013
4454996
add get_next_position
guweigang Oct 14, 2013
b50a6c2
it make no sense to compare server_id and 0, if you give 0, you will …
guweigang Oct 18, 2013
59a0890
copy data to fix bug on some systems
ideal Oct 30, 2013
db92154
fix wrong commited file
ideal Oct 30, 2013
28391e2
fix bug in protocol_chunk
ideal Oct 30, 2013
c63a23b
make offset NEED_ALLOC
ideal Oct 30, 2013
287f23d
add more NEED_ALLOC
ideal Oct 30, 2013
07bc60a
add need_alloc for m_server_id
guweigang Oct 30, 2013
1968755
add another NEED_ALLOC
ideal Oct 30, 2013
dc0e297
update .gitignore
ideal Nov 1, 2013
04dcc9e
add notice for setting server-id
ideal Nov 7, 2013
6189ad8
Update README.md
Jan 11, 2014
7146596
add a judgement for set_position
guweigang Jan 23, 2014
368d403
remove use of boost config.hpp
ideal May 15, 2014
ccb4292
add check for boost
ideal May 16, 2014
5d52435
add travis ci
ideal May 20, 2014
66f7da9
add travis ci image
ideal May 20, 2014
af3b18c
currently we need boost header files
ideal May 20, 2014
0f35f7b
add clang compiler
ideal May 22, 2014
6cb40f5
check if boost exists
ideal May 25, 2014
2963c61
check if boost exists
ideal May 25, 2014
3316e4a
add base_from_member
ideal May 25, 2014
f745239
fix php ext build error temporarily
ideal Jul 4, 2014
4ce5647
for cmake policy CMP0042 (see cmake --help-policy CMP0042)
ideal Sep 4, 2014
5cc3d35
change a build status icon
ideal Nov 9, 2014
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ CMakeFiles
cmake_install.cmake
lib
tags
install_manifest.txt
19 changes: 19 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
language: cpp
compiler:
- gcc
- clang

before_install:
- echo $LANG
- echo $LC_ALL
- cmake --version
- gcc -v
- sudo apt-get install -y libboost-dev

script:
- mkdir build && cd build && cmake .. && make

notifications:
email:
- guweigang@bullsoft.org
- shangyuanchun@bullsoft.org
19 changes: 12 additions & 7 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
project (mysql-replication-listener)
cmake_minimum_required (VERSION 2.6)
cmake_minimum_required (VERSION 2.8)

set(MySQL_BINLOG_VERSION_MAJOR "0")
set(MySQL_BINLOG_VERSION_MINOR "0.1")
set(MRL_VERSION "${MySQL_BINLOG_VERSION_MAJOR}.${MySQL_BINLOG_VERSION_MINOR}")
set(MySQL_BINLOG_VERSION_MINOR "2")
set(MySQL_BINLOG_VERSION_PATCH "0")
set(MySQL_BINLOG_VERSION "${MySQL_BINLOG_VERSION_MAJOR}.${MySQL_BINLOG_VERSION_MINOR}.${MySQL_BINLOG_VERSION_PATCH}")

set(CMAKE_VERSION_STRING "${CMAKE_MAJOR_VERSION}.${CMAKE_MINOR_VERSION}")

Expand All @@ -14,9 +15,13 @@ set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY ${PROJECT_BINARY_DIR}/lib)
include_directories(include)
link_directories(${PROJECT_BINARY_DIR}/lib)

# TODO: order: find asio -> allow to define asio directory -> use include/asio
# For ASIO headers
INCLUDE_DIRECTORIES(include/asio)
find_package(Boost)
if(Boost_FOUND)
include_directories(${Boost_INCLUDE_DIRS})
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DHAVE_BOOST")
else(Boost_FOUND)
message(INFO "No Boost found, will not use Boost")
endif(Boost_FOUND)

# --------- Find crypt ------------------------------------------
FIND_LIBRARY(LIB_CRYPTO crypto /opt/local/lib /opt/lib /usr/lib /usr/local/lib)
Expand All @@ -26,5 +31,5 @@ add_subdirectory(src)
include(InstallRequiredSystemLibraries)

# installation configuration
install(DIRECTORY include DESTINATION /usr FILES_MATCHING PATTERN "*.h")
install(DIRECTORY include DESTINATION . FILES_MATCHING PATTERN "*.h" PATTERN "*.hpp" PATTERN "*.ipp")

38 changes: 35 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,40 @@
mysql-replication-listener
MySQL Replication Listener
=========================

[![Build Status](https://img.shields.io/travis/BullSoft/mysql-replication-listener/master.svg?style=flat)](http://travis-ci.org/BullSoft/mysql-replication-listener)

A listener to fetch binlog from mysql server.

Reference:
https://code.launchpad.net/mysql-replication-listener
https://github.com/wangmh/mysql-replication-listener
- Origin repo: https://code.launchpad.net/mysql-replication-listener
- We forked from: https://github.com/wangmh/mysql-replication-listener

Install
--------------------
mysql-replication-listener uses cmake, so you need cmake first.

We use asio for socket communication, and it involes some **boost** libraries, so maybe you need install boost(1.47 and newer).

And then:

$ cmake . -DCMAKE_INSTALL_PREFIX=/path/to/install

$ make

$ make install


Notice
--------------------
We need to make mysql server to dump binlog-stream to us, so we need **REPLICATION SLAVE** privilege.
And We need to get master status, so we need **REPLICATION CLIENT** privilege.

If you want to set_position to a specified binlog-file and the position, we need to ensure the binlog-file and it's position are legal, so we need to execute "SHOW BINARY LOGS" on the server. Unfortunately,
In MySQL 5.1.63 and earlier, the **SUPER** privilege was required to use this statement. Starting with MYSQL 5.1.64, a user with the **REPLICATION CLIENT** privilege may also execute this statement.

So If you use MySQL 5.1.63 and earlier, you need **SUPER** privilege too.

Also, We auth privilege before binlog_dump, so we will access database **mysql** for accout checking... That means you have to give at least **READ** privilege for database mysql ...

Also notice that besides `log-bin` and `binlog-format` set in mysqld section, you should also set a `server-id` for mysql master.

14 changes: 9 additions & 5 deletions include/asio/basic_socket_iostream.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,15 @@

#if !defined(BOOST_NO_IOSTREAM)

#include <boost/preprocessor/arithmetic/inc.hpp>
#include <boost/preprocessor/repetition/enum_binary_params.hpp>
#include <boost/preprocessor/repetition/enum_params.hpp>
#include <boost/preprocessor/repetition/repeat_from_to.hpp>
#include <boost/utility/base_from_member.hpp>
#ifdef HAVE_BOOST
# include <boost/preprocessor/arithmetic/inc.hpp>
# include <boost/preprocessor/repetition/enum_binary_params.hpp>
# include <boost/preprocessor/repetition/enum_params.hpp>
# include <boost/preprocessor/repetition/repeat_from_to.hpp>
# include <boost/utility/base_from_member.hpp>
#else
#endif

#include "asio/basic_socket_streambuf.hpp"
#include "asio/stream_socket_service.hpp"

Expand Down
30 changes: 25 additions & 5 deletions include/asio/basic_socket_streambuf.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,31 @@
#if !defined(BOOST_NO_IOSTREAM)

#include <streambuf>
#include <boost/preprocessor/arithmetic/inc.hpp>
#include <boost/preprocessor/repetition/enum_binary_params.hpp>
#include <boost/preprocessor/repetition/enum_params.hpp>
#include <boost/preprocessor/repetition/repeat_from_to.hpp>
#include <boost/utility/base_from_member.hpp>
#ifdef HAVE_BOOST
# include <boost/preprocessor/arithmetic/inc.hpp>
# include <boost/preprocessor/repetition/enum_binary_params.hpp>
# include <boost/preprocessor/repetition/enum_params.hpp>
# include <boost/preprocessor/repetition/repeat_from_to.hpp>
# include <boost/utility/base_from_member.hpp>
#else
namespace boost {
template <typename MemberType, int UniqueID = 0>
class base_from_member
{
protected:
MemberType member;

base_from_member(): member()
{
}
};
}
# ifndef BOOST_PP_CAT
# define BOOST_PP_CAT(a, b) a ## b
# endif
# define BOOST_PP_INC(x) (x)
#endif

#include "asio/basic_socket.hpp"
#include "asio/deadline_timer_service.hpp"
#include "asio/detail/array.hpp"
Expand Down
14 changes: 13 additions & 1 deletion include/asio/detail/config.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,19 @@
#ifndef ASIO_DETAIL_CONFIG_HPP
#define ASIO_DETAIL_CONFIG_HPP

#include <boost/config.hpp>
#ifdef HAVE_BOOST
# include <boost/config.hpp>
#else
# if defined(_WIN32) || defined(__WIN32__) || defined(WIN32)
# define BOOST_WINDOWS
# if !defined(__GNUC__) && !defined(BOOST_HAS_DECLSPEC)
# define BOOST_HAS_DECLSPEC
# endif
# ifdef _MSC_VER
# define BOOST_MSVC _MSC_VER
# endif
# endif
#endif

// Default to a header-only implementation. The user must specifically request
// separate compilation by defining either ASIO_SEPARATE_COMPILATION or
Expand Down
36 changes: 25 additions & 11 deletions include/binlog_api.h
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
#ifndef _REPEVENT_H
#define _REPEVENT_H

// temp fix
#ifndef HAVE_BOOST
#define HAVE_BOOST
#endif

#include <iosfwd>
#include <list>
#include <cassert>
Expand Down Expand Up @@ -49,12 +54,16 @@ enum Error_code {

class Dummy_driver : public system::Binary_log_driver
{
public:
public:
Dummy_driver() : Binary_log_driver("", 0) {}
virtual ~Dummy_driver() {}

virtual int connect() { return 1; }

virtual int disconnect() { return ERR_OK; }

virtual int set_server_id(int server_id) { return server_id; }

virtual int wait_for_next_event(mysql::Binary_log_event **event) {
return ERR_EOF;
}
Expand All @@ -66,31 +75,33 @@ class Dummy_driver : public system::Binary_log_driver
virtual int get_position(std::string *str, unsigned long *position) {
return ERR_OK;
}

};

class Content_handler;

typedef std::list<Content_handler *> Content_handler_pipeline;

class Binary_log {
private:
private:
system::Binary_log_driver *m_driver;
Dummy_driver m_dummy_driver;
Content_handler_pipeline m_content_handlers;
unsigned long m_binlog_position;
std::string m_binlog_file;
public:
public:
Binary_log(system::Binary_log_driver *drv);
~Binary_log() {}

int connect();

int disconnect();

/**
* Blocking attempt to get the next binlog event from the stream
*/
int wait_for_next_event(Binary_log_event **event);


/**
* Inserts/removes content handlers in and out of the chain
* The Content_handler_pipeline is a derived std::list
Expand All @@ -101,9 +112,9 @@ class Binary_log {
* Set the binlog position (filename, position)
*
* @return Error_code
* @retval ERR_OK The position is updated.
* @retval ERR_EOF The position is out-of-range
* @retval >= ERR_CODE_COUNT An unspecified error occurred
* @retval ERR_OK The position is updated.
* @retval ERR_EOF The position is out-of-range
* @retval >= ERR_CODE_COUNT An unspecified error occurred
*/
int set_position(const std::string &filename, unsigned long position);

Expand All @@ -112,12 +123,15 @@ class Binary_log {
* @param position Requested position
*
* @return Error_code
* @retval ERR_OK The position is updated.
* @retval ERR_EOF The position is out-of-range
* @retval >= ERR_CODE_COUNT An unspecified error occurred
* @retval ERR_OK The position is updated.
* @retval ERR_EOF The position is out-of-range
* @retval >= ERR_CODE_COUNT An unspecified error occurred
*/
int set_position(unsigned long position);


int set_server_id(int server_id);

/**
* Fetch the binlog position for the current file
*/
Expand Down
7 changes: 5 additions & 2 deletions include/binlog_driver.h
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ class Binary_log_driver
{
public:
template <class FilenameT>
Binary_log_driver(const FilenameT& filename = FilenameT(), unsigned int offset = 0)
Binary_log_driver(const FilenameT& filename = FilenameT(), unsigned int offset = 0)
: m_binlog_file_name(filename), m_binlog_offset(offset)
{
}
Expand All @@ -44,9 +44,12 @@ class Binary_log_driver
* @retval 0 Success
* @retval >0 Error code (to be specified)
*/
virtual int connect()= 0;
virtual int connect() = 0;

virtual int disconnect() = 0;

virtual int set_server_id(int server_id) = 0;

/**
* Blocking attempt to get the next binlog event from the stream
* @param event [out] Pointer to a binary log event to be fetched.
Expand Down
Loading