From 226eae4034840da2dccc1ee269e172aca96c5ee7 Mon Sep 17 00:00:00 2001 From: Amihan Date: Thu, 9 Jun 2016 17:33:26 +0800 Subject: [PATCH 1/5] Celery TIL --- .DS_Store | Bin 0 -> 6148 bytes celery/about-celery.md | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) create mode 100644 .DS_Store create mode 100644 celery/about-celery.md diff --git a/.DS_Store b/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..e17e88b8f3c66f8ea9957aba3d0fa6ebcd4cd8a1 GIT binary patch literal 6148 zcmeHK%We}f6g^JT2dxlYAi;8$Y(VNCOw$P{5)Y|?Eh5RJN!m#!O43%@p!o?TzJf1d z#|E+H16Z?xb8V;ccmk|IsPc{M>&H3v`1lnAP|b_<5I6v^%OJk7$7q8|y@ZO@Y{^}E zrQ7Hs!=Rnz{a#i!+3>K_&nGk8zlloqlRQb%i)A19 z^!CdK4e;vI-`3AYmGrcFq?o*pCQsn&2sL_b& zQ!G5z=RdGkY-5gTnN@r3v9{QHkI2RPOk+!@=uM7P{?{hNo| z!Bc8-j23n2pjAfeb@Set(LJuQ0hYL;CUczgOQsll99c0gr%v_8K1zw&F6tgpImf@G z(o6O+VLsysB&}zQ@q&3%ZtthHN=jzIZEc(E6}cm}xk-B$dyA{1g(D<*M!#WY=s9p1 zTl^B)Jyf7Pc0CI&lAraLjd!-#?!NIM4hjSXf&$kH$oUX4h>xu=jMqm8BmD%RZ?IS! z)AFZ5IFa?S^@Z`Qp(#V9H&kVx7|PID9$8#$ePO(zL)piNvOO#NLQ%SRwjXIYRBU{R zg91T;vI2FJJ&^mq_3QV4nG|wCfuO*RQb0A5PSWI%?A=-!oZPh$!y5)M!4<}93L|?R h>x5jz_ZhUYE|LU&Y<*!oOK9PbfR-UdL4kj&z)wGU0-^u_ literal 0 HcmV?d00001 diff --git a/celery/about-celery.md b/celery/about-celery.md new file mode 100644 index 0000000..2077ad8 --- /dev/null +++ b/celery/about-celery.md @@ -0,0 +1,36 @@ +# Getting Started with Celery + +**Date: June 9, 2016** + +Celery is a task queue, which means it is a tool used to distribute tasks across several servers. + +#### Brokers +Celery uses brokers to mediate between clients and workers. There are several choices available for brokers: +- RabbitMQ +- Redis +- Using a database + - Using SQLAlchemy + - Using the DJango database +- Other brokers + - Amazon SQS + - MongoDB + - IronMQ + +#### Installing Celery +``` +pip install celery +``` + +#### Creating a Celery instance +``` +app = Celery('module_name', backend='redis://localhost', broker='redis://localhost') +``` +The first argument to a Celery instance is the module name, next is the backend argument wherein Celery stores or sends the states, and last is the broker keyword that specifies the URL of the message broker. + +#### Running a worker server +``` +celery -A tasks worker --loglevel=info +``` + +#### References: +- [First Steps with Celery](http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html) From 144173165cfac57f8c3b129d02ac0ec261a37c27 Mon Sep 17 00:00:00 2001 From: Amihan Date: Thu, 9 Jun 2016 17:36:40 +0800 Subject: [PATCH 2/5] Added TIL to README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 724e780..967b5f0 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,10 @@ This repository serves as a collection of things learned by people working in AG - [Ambari Managed Ranger Will Auto-Generate Services](apache-ranger/ambari-managed-ranger-auto-generate-services.md) +### Celery + +- [Getting Started with Celery](celery/about-celery.md) + ### Hadoop: HDFS - [What is HDFS](hadoop-hdfs/basic-hdfs.md) From 4c928d23ce51a48e9e7922492a252478427a4a5d Mon Sep 17 00:00:00 2001 From: Amihan Date: Wed, 29 Jun 2016 17:32:25 +0800 Subject: [PATCH 3/5] TIL for Data Warehouse Overview --- data warehouse/about-data-warehouse.md | 32 ++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 data warehouse/about-data-warehouse.md diff --git a/data warehouse/about-data-warehouse.md b/data warehouse/about-data-warehouse.md new file mode 100644 index 0000000..d1df600 --- /dev/null +++ b/data warehouse/about-data-warehouse.md @@ -0,0 +1,32 @@ +# Data Warehouse + +**Date: June 29, 2016** + +#### Basics +Given a String or Unicode object, the % operator (modulo) converts it to the specific format specified to its right. + +``` +print "Let's talk about %s." % my_name +print "He's %d inches tall." % my_height +``` +The above code evaluates to +``` +Let's talk about Zed A. Shaw. +He's 74 inches tall. +``` +The first line of the above code formats my_name into String, while the second line converts my_height to a signed integer decimal. + +#### Important to remember +``` +print "So, you're %r old, %r tall and %r heavy." % ( + age, height, weight) +``` +This code evaluates to +``` +So, you're '38' old, '6\'2"' tall and '180lbs' heavy. +``` + +%r is used for debugging as it shows the raw representation of data while %s is used for displaying of data. + +####References: +- [Learn Python the Hard Way](http://learnpythonthehardway.org/book/) From 8ad4aaaa9fb5ca08d19cb4160e2bc4349ca5172c Mon Sep 17 00:00:00 2001 From: lykamacusi Date: Wed, 29 Jun 2016 17:44:16 +0800 Subject: [PATCH 4/5] TIL for Data Warehouse Overview --- README.md | 4 ++++ data-warehouse/about-data-warehouse.md | 30 ++++++++++++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 data-warehouse/about-data-warehouse.md diff --git a/README.md b/README.md index 967b5f0..cfab594 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,10 @@ This repository serves as a collection of things learned by people working in AG - [Getting Started with Celery](celery/about-celery.md) +### Data Warehouse + +- [Data Warehouse Overview](data-warehouse/about-data-warehouse.md) + ### Hadoop: HDFS - [What is HDFS](hadoop-hdfs/basic-hdfs.md) diff --git a/data-warehouse/about-data-warehouse.md b/data-warehouse/about-data-warehouse.md new file mode 100644 index 0000000..4d1de5d --- /dev/null +++ b/data-warehouse/about-data-warehouse.md @@ -0,0 +1,30 @@ +# Data Warehouse Overview + +**Date: June 29, 2016** + +A data warehouse is a relational database designed for query and analysis. It basically helps in the analysis of data. It makes use of ETL (Extract-Transform-Load) process, OLAP (Online Analytical Processing) engines and other tools that help in the management of data. + +#### Characteristics +- Subject-Oriented + - The ability to define what the data warehouse contains +- Integrated + - Data has a consistent format therefore there are no problems regarding data inconsistency +- Nonvolatile + - Data entered into the warehouse cannot be changed +- Time Variant + - The ability to store historical data + +#### Requirements +- Workload + - Designed to accommodate ad hoc queries +- Data modification + - It is updated on a regular basis by the ETL process +- Schema Design + - Uses denormalized or partially denormalized schema for optimized performance +- Typical Operations + - A typical data warehouse query scans thousands or millions of rows +- Historical Data + - Stores years worth of data for historical analysis + +#### References: +- [Oracle Documentation](https://docs.oracle.com/cd/B10501_01/server.920/a96520/concept.htm#50413) From f96fb7c20a8104fa330815540672b3cb6bf197bb Mon Sep 17 00:00:00 2001 From: Daniellika Macusi Date: Wed, 29 Jun 2016 17:45:37 +0800 Subject: [PATCH 5/5] Delete about-data-warehouse.md --- data warehouse/about-data-warehouse.md | 32 -------------------------- 1 file changed, 32 deletions(-) delete mode 100644 data warehouse/about-data-warehouse.md diff --git a/data warehouse/about-data-warehouse.md b/data warehouse/about-data-warehouse.md deleted file mode 100644 index d1df600..0000000 --- a/data warehouse/about-data-warehouse.md +++ /dev/null @@ -1,32 +0,0 @@ -# Data Warehouse - -**Date: June 29, 2016** - -#### Basics -Given a String or Unicode object, the % operator (modulo) converts it to the specific format specified to its right. - -``` -print "Let's talk about %s." % my_name -print "He's %d inches tall." % my_height -``` -The above code evaluates to -``` -Let's talk about Zed A. Shaw. -He's 74 inches tall. -``` -The first line of the above code formats my_name into String, while the second line converts my_height to a signed integer decimal. - -#### Important to remember -``` -print "So, you're %r old, %r tall and %r heavy." % ( - age, height, weight) -``` -This code evaluates to -``` -So, you're '38' old, '6\'2"' tall and '180lbs' heavy. -``` - -%r is used for debugging as it shows the raw representation of data while %s is used for displaying of data. - -####References: -- [Learn Python the Hard Way](http://learnpythonthehardway.org/book/)