Philosophy Of Complex Event Processing

Philosophy Of Complex Event Processing

Question 1:

What are some typzzical uses for complex event processing applications? Give examples and elaborate on the possibilities.

Answer:

Some typical uses for complex event processing are air traffic control, stock market trading, business process management and fraud detection.

In stock market a buyer may place an order depending on many factors like average price over last hour, total turnover for the particular symbol, etc. All these are events which can be processed using a CEP suite like Esper.

It can also be used for making global policies based on events throughout the world.

Question 3:

What is the difference between doing aggregated queries (AVG/SUM/MAX/ETC) in standard databases and stream databases?

Answer:

In standard databases we get a single result when we do aggregated queries but in stream databases we get new result for each event which is the aggregate till that point.

Question 4:

For each non-overlapping 100 tick interval of the BTC stock, find the minimum and maximum weighted price in the interval.

Thoughts:

Esper is very easy for analyzing a set of events and we can filter by time or length.

Question 5:

Extend the previous query so that it now also shows the difference between the minimum and maximum weighted price for each interval.

Thoughts:

Aggregate functions can be used to generate statistics which add value to the business.

Question 6:

Find the total weighted price of all stock tick events in each non-overlapping 100 tick interval of the BTC stock.

Thoughts:

We can also find total over a period of time or number of events.

Question 7:

Find the dates and minimum weighted price for the BTC stock for each, non-overlapping 30 tick interval.

Thoughts:

We can find corresponding parameters for aggregate values.

Question 8:

What are patterns and pattern operators? What does the every and followed-by (->) patterns do?

Explain the difference between the following:

every A -> B

A -> every B

every A -> every B

every (A -> B)

Answer:

Patterns are rules for matching events very similar to the where clause of SQL but they are applied before the events are included in current batch window.

Pattern operators are used to combine patterns (and, or) and order them (->). We can also have custom pattern operators.

Differences:

every A -> B: It means for every event A followed by event B. The pattern restarts whenever event A is encountered.

A -> every B: It means for event A followed by every event B. The pattern restarts whenever event B is encountered.

every A -> every B: It means for every event A followed by every event B. The pattern restarts on encountering both event A and B.

every (A -> B): It means for every event A followed by event B. The pattern restarts after encountering event B.

Question 9:

Find all the start and stop dates with volume, when the weighted price of the BTC stock increases more than 100 times and volume increases by 40%.

Thoughts:

We can track changes in data between events using patterns.

Question 10:

What is technical analysis? What is algorithmic trading?

Answer:

Monitoring the market based on changes in price is called technical analysis. It is the study of supply and demand in the market in order to determine the trend or direction in future.

Algorithmic trading is use of advanced mathematical models for decision making in the market. It is used to optimize the time to place an order so that it makes minimal impact on the stock's price.

Question 11:

Thoughts:

We can combine multiple pattern using pattern operators to get desired results.

Question 12: Thoughts:

We can follow trend easily using Esper patterns.

Question 13:

What are pattern guards? Why cannot the timer:within pattern guard be used in simulated environments? What alternatives exist to control time and dates?

Answer:

Pattern guards limit the scope of the pattern to another quanity like time or condition.

timer:within pattern guard depends on actual latency between events but in simulated environments the events occur in quick succession and hence timer:within cannot be used.

We can use the WHERE clause of EPL to control time and date.

Question 14:

Find the start date, stop date and weighted prices when the BTC stock value has decreased by three times within a 15 day period,

Thoughts:

We can analyze events based on trend in a fixed size time of length window.

Question 15:

What are variables in Esper and how can they be used to hold and change state? Give examples!

Answer:

A variable in Esper is an object, event or scalar value which is available for use after declaration. It can be used in patterns and other statements including output clauses.

Question 16:

Write your own trading algorithm and use it to find buy and sell dates. No hard-coding of dates or numbers are allowed! Use the knowledge you have on Esper to write your query of choice. If you want to change the Java files, you are allowed to, but you do not have to (e.g. needed for multiple queries).

Thoughts:

My trading algorithm is as follows:

We sell when the weighted price has risen for three days (not necessarily adjacent) and the volume has increased by 35% between first and third day.

We buy when the weighted price has fallen for three days (again not necessarily adjacent) and the volume has decreased by 45% between first and third day.

Question 17:

What did you struggle with during this exercise? Was it the provided code's fault, Esper's or your own fault? What was different from working with regular database systems?

Answer:

I struggled with the syntax and reverse flow of data during this excercise. The patterns and batch window concepts were new and after solving the problems seemed like essential to the philosophy of  Complex Event Processing.

It was not anyone's fault but just a learning curve and forgoing many deep rooted practices like getting a single result for a typical query like count(*).

In regular database system we mostly have data at a state and we run the queries whereas in Esper the queries are know beforehand and the data comes as and when.