Fix flaky test IntegrationTest_MultiSpoutsMultiTasks

[huijunwu]

Fix
#1561
#1691
#1687
#1700

The cause is:
tuple count includes 'termination tuple', which leads to multiple same acking tuples and stmgr failure to cancel out markers in xor_manager.
https://travis-ci.org/twitter/heron/builds/225447993

Fix:
count only normal tuples
https://travis-ci.org/twitter/heron/builds/225471892

[kramasamy]

@huijunwu - wondering if you ran the tests a few times before the flakiness is gone.

[objmagic]

Also enable #1773 ?

[huijunw]

@kramasamy Test was run hundred times; all passed
@objmagic IntegrationTest_OneSpoutBoltMultiTasks is different. More time needed.

[billonahill]

Great find. I don't understand how this would make the test flaky. Why wouldn't the tests fail consistently without this fix?

[huijunwu]

@billonahill
The time when 'termination tuple' arrives at the IdentityBolt is different everytime. If a 'termination tuple' arrives between two normal tuple, the count is one more than expected and the acking tuple is one more than expected.
Usually, the 'termination tuple' is after all the normal tuples, so there is no issue.
There are 6 spouts in the test. If one of them runs fast and sends out 'termination tuple' before the other spouts sending out all normal tuples, the wrong count appears.

[billonahill]

Thanks @huijunw for clarifying. Great find.

[objmagic]

Great. Thanks!

[nlu90]

Nice finding