Event Processing Thinking

I came across a recent presentation given by Gregoris Mentzis (from NTUA, Greece) entitled "towards proactive enterprise intelligence".  In this presentation Gregoris discusses some research challenges.
The capabilities of proactive enterprise intelligence are defined in slide 21 and seem similar to our definition (I also recognized the pictures).  I'll write more about the two patterns expressed in this slide. 
Reading this presentation is recommended. Enjoy! 

Typically I don't recommend commercial Blogs, but I'll make an exception this time, and cite Mark Palmer's post, since I like the metaphor he made, reflected in this picture.  According to Mark, analytics that refers to the past is like driving by only looking at the rear-view mirror,  which of course can show the road you have already passed.  Since typically we drive forward and not backward it should be more useful to look ahead than to look back.   In many cases the road is fixed, in the sense that the road forward looks exactly like the road backward, and then it might make sense to do it,  however, in other cases, like driving in real traffic and not in a bubble, the road ahead may contain surprises that are not evident from the previous parts of the road. The interesting thing is that even law latency event processing system are, in fact, looking at the past,  where the past is almost the present,  looking to the future is not a standard event processing feature.   

I am one of the co-chairs of the "City Data Management 2012" workshop in CIKM'12 The work on smart cities have been emerged in the last few years.  The data management aspect of smart cities is one of the major topics, due to the need to acquire significant amount of data, much of it streaming data, and perform monitoring, search, query and various analytics.  The workshop focuses on the several aspects of city data management.    The call for papers is now out,   the interesting part for me is the data monitoring part, which consists of the following points:

• Complex Event Processing for Smarter Cities
• Anomaly detection and prevention
• Forecasting city events
• Event-based optimization for adaptive city operations
• City process monitoring

If you are in this area -- consider submission. Note that the conference will take place in Maui, Hawaii.

I came across an interesting discussion on the topic what is the right education for entrepreneurs.According to the study cited in this article,  most of the entrepreneurs don't have degrees in computer science or engineering but in various other disciplines.   This echos the debated between Bill Gates who thinks that engineering education is the only one matters, and Steve Jobs who said that in Apple the DNA is mixed between engineering and liberal arts.I have some perspective on this issue since I have enjoyed a diversified education. I did BA degree in Philosophy,  then went for MBA studies and learned the basic of business concepts, and then did PhD in Computer Science.  More than 10 years ago I participated in the "basic blue" management course of IBM, this was a "private course" given for new managers in IBM Haifa Research Lab, since due to organizational changes, and addition of one more level of management, many new managers were appointed within a short period of time to justify bringing the course to us.   As a preparation of the course, the participants fill a substantial questionnaire, and there are also 360 view questionnaires form managers, peers and direct reports.  When showing the results the course's instructor said that he was surprised to see that 16 of the 18 participants had very similar personality profile.  The common denominator:  all of them studied in the Technion computer science or computer engineering, I was one of the two exceptions.    While generalizations never properly work,  it seems that there is something in the engineering education that drive people to concentrate on the technical details and see their challenge in doing strong technical work, there is nothing wrong with it, any company small or big in the technology domain requires strong technical people.   
 Entrepreneurs require something beyond the technical skill,  they need the passion to change the world, to believe that they are doing the "next big thing".  This is another mode of thinking.  I have never been an entrepreneur (perhaps it is still not to late), but I have written about intrapreneuring   (trying to do "start-ups" within a big corporate) before.     If I need to reflect upon my past studies and see what contributed most to me, I view my studies as complementary.  The MBA studies contributed understanding of the basics of the business concepts,  the computer science studies contributed to the understanding of technology and the content of what I've been doing,  but the passion to change the universe, and the contribution to thinking out of the box was definitely been of the philosophy studies.  In retrospect they were most influential on shaping my personality.  
I am not surprised that many entrepreneurs are not engineers, there is something inherent in engineering education that makes people think in boxes.

I have been asked by several people to write in more detail about what I meant  by the layered approach, so I thought it should be better illustrated in an architecture diagram (converting it to picture did not result in high quality, somehow).  The rationale behind it that event processing became pervasive in use, with both event processing products, and "build your own solution".    In the same way that application servers based on standards like J2EE contributed to building of certain type of web-based enterprise applications,  an application server for event-based applications is required to provide services from various types: from context service, adapters to sensor and mobile platforms, dashboard, management services, meta-data service and more.   The second layer is the agent layer which provide both directory of agents and tools to build your own agent.  The application is the third layer.   This architecture can provide independence, and ability to get best of breed in both services and functionality components from different sources, and combinations of "build" and "buy".Standards are of course the key to make it happen.  

I have always been fascinated by the potential of combination of event processing and robotics, and have written about it before, as well as followed some work in this area.   Recently I have read some material about EU project called ROBOT-ERA,  which is aimed at providing robot's support for the elderly.  Their plan is to create three type of robots for the elderly, which they call - domestic (for the room/apartment), condominium (for the bigger house/ home for the aged), and outdoor, with communication between the three types.  This system is obviously event-driven, but as many of the applications of the world it does not use COTS of event processing, but home-built solutions.    I think that while the event processing community concentrated in the COTS, it should also pay attention to help those who role their own --- I'll write more about it.I also think that there is a huge  potential of smart systems and robots to help the elderly which sometimes the world looks to them very challenging,   Besides the EU projects there are other activities in this area in Japan, and Australia (watch this video). I assume there are others as well.  When such technology will become pervasive and affordable, it will have a significant impact on quality of life of elderly people. 

Sensors and social media are becoming the major sources of data, while the traditional organizational data does not grow in the same rate.    Many information system both in the enterprise level and in the individual level are being based on streaming data - these are all old news.   I have recently written about event server as the 21st application server, following TIBCO's  statements.   Progress Software in its recent statement also talked about cloud-based middleware that is geared towards real-time analytics, within its new re-structuring.     
Currently, products and solutions in this area provide a mix of different things.   I envision that we'll settle in three tier architecture, with different products over the food chain that might be more specialized: 
 The event application server layer:
  This will provide services like:  

• sensor and actuator framework, including adapters, data normalization, de-duplication and more.
• routing services -  pub/sub, channel management .
• state services -- access to external stores (databases, files), global state, local state, state machine services 
• meta-data repository services -- representation of events, relationships among events, and other meta-data entities.
• event flow services -- ability to devise event flows ("event processing networks") from the "programming in the large" point of view (not the implementation of specific agents), this will provide API for event producing, event routing and event consuming. 
• Tuning and scalability services -- ability to tune applications by using various ways of parallelism and distribution, fast routing,run-time scheduling and load balancing,
•  tracking and provenance services
• context services 
•  management services
• visualization --- dashboard and visual analytics for feedback services. 
• recoverability services
• high availability services
• security and privacy services

and maybe more...

The application service layer will be based on event-driven  agent-oriented architecture, but will also support request-driven access to various components.  This will support large quantity of lightweight agents, and the flow among them.  While this is quite different from the way most people think about programming, it will become one of the pervasive programming models -- so it is a good investment for the future career of professionals to understand it.
The agent building environment:   
The second layer is the layer that assists in constructing event processing agents.    One way of using the event application server is to use the API  and implement the logic within traditional programming languages, this will require standards like JDBC to access events and event operators in the same way that we currently issue SQL queries inside programming languages.    The other way of doing it is to use agent building tools that generate code for agents,  these might have benefits of resusability, optimized implementation, and reduction in cost.  This setting will allow hybrid application -- an application may consist of filter agent that is written in Java, using the assertion operator API,  aggregation agent taken from a vendor that specializes in optimized statistical functions, and pattern matching agent taken from another vendor who specializes in some type of patterns (e.g. spatio-temporal patterns).
The application layer:
Applications will be constructed on top of the first and second layer.   It may be all developed using the second layer supplied by a single vendor, multiple vendors, home-made, or any combinations.  In some cases the applications will be developed by a specific user, in other cases it will be developed by independent software vendors specializing in a certain domain. 
In this food chain we'll be able to see different vendors -- those who specialize in the application server layer  and compete on the quality of services provided there; vendors who specialize in the agent building environment and provide filtering, transformation, aggregation, pattern matching, some of them maybe specialized for specific application type or domain; the third type of vendors will be application providers, which typically require subject matter expertise.
Some of the big vendors will attempt to be active in all three layers or create partnership.  

I typically don't publish the same post twice, but today, due to the holocaust day, I am returning to something I published in this blog 3 years ago (and got many reactions upon it), the reason is that according to the statistics, during these 3 years the readers population for this Blog has grown significantly - so while old readers might remember it, I also want to share it with the newer ones,  This day is very much associated  in  my mind with my later father who was the only survivor of a big family.   Here is a picture of my father late in his life: 
I don't have any picture of him until after the  war, but here is a picture of the kitchen the  Lodz Ghetto, where he worked at the beginning of the war (he said that he might be the person in the middle, but was not sure about it)

My father survived the war, while his parents and 7 brothers and sisters did not.He never talked much about this period in life,  saying that this was on another planet, and cannot be described.  Once he told me that immediately after the war a relative found him in the survivors list and send him a letter asking him to describe whatever occurred to him during the war, my father answer was:
I will not tell you about what happened to me in the war, instead I will tell you the story of King Alfred  





King Alfred has escaped due to some revolt and was hiding in a farm, when the rebelling soldiers looking for him got there, the farmer hid the King below a big pile of straw; the soldiers started to look at the straw, they nearly removed all of it, and then decided to move on. At later time the King succeeded to overcome the mutiny and returned to his throne. At some point the farmer came to visit him and he said to him -- you save my life, I can give you whatever you wish, the farmer said: I am a modest person, don't need anything, have one question to you, what did you feel when the soldiers almost got all the straw removed. The king has shown an angry face and said: this is a very rude question, hang this man immediately. The farmer was about to be hanged, a rope was already tightened to his throat, and then the king said: stop, now you see what I felt.

I have written before about the recent return to the bi-temporal databases, in conjunction with DB2.  In the 1990-ies was the first attempt to create bi-temporal extensions to SQL, at that time there was a language war, some of it is reflected in the book that I have co-edited, published in 1998.  Now after some attempts, SQL:2011 does include support in bi-temporal databases.  The terminology was changed from the original terms.  What was called in the original version - "valid time" is called in the SQL version "application time',  an what was called in the original version - "transaction time" is called in the SQL version "system time".   
More details about the SQL extension can be found in the overview presentation that Craig Baumunk uploaded to slideshare.      As I have written before, temporal database is vital for maintaining historical events, and thus the importance of this standard, and the supporting databases to event processing application is noticable

Recent financial news indicate that the Financial Industry Regulatory Authority has fined one of the analysts firms for failing to supervise equity research analyst communications with traders and clients and for failing to adequately monitor trading in advance of published research changes to detect and prevent possible information breaches by its research analysts.   This is interesting, since it seems that the regulator now expects firms to monitor the consequences of their actions.   This calls for real-time monitoring, monitor causalities between various events, and eliminate increased trading based on unpublished information.     Recently, I have participated in a meeting with stock exchange people in a certain country,  and heard about their efforts to detect undesired phenomena in trade.  They also were looking at real-time monitoring, and even proactive behavior, trying to detect undesired phenomena before they happen.   I guess that we'll see more of these applications from different sides - regulators, traders, and in this case analysts.   

Paul Vincent has posted in the TIBCO Blog a post entitled:  "event server as the 21st app. server".   
Paul cites TIBCO CEO Vivek Ranadivé in TIBCO's quarterly earning report, and concludes that an event server will is a requirements in many applications that process events in various ways.  
Getting to the notion of application server (see illustration below taken from an article on Websphere Application Server)
Application servers are intended to support services to applications such as:  transaction, storage, database approach, security, high availability, administration and more.     


In the event-driven world there are flowing events, and with the Internet of Things, most data in the universe will be in form of events.   In the event processing manifesto (that Paul has been one of the contributing members to its creation) we talked about "event fabric" which will enable Internet scale sharing of events and will support many applications. Some of the fabric properties mentioned were providing services of  privacy, security, interoperability among fabric instances, provenance, energy efficiency, autonomic computing support (self-tuning etc...), availability, scalability, anonymity, non-repudiation, QoS with multiple criteria.     These are some of the services, and there are of course functional services like context service, adapter and transformation service, filtering service, aggregation service, pattern matching service and more that should be built into the server and can be used by various applications from various application areas and types (BPM, CRM, Social computing, track and trace and many more).    


Paul rightly notes that standards have key role in establishing such event server,  Paul indeed wrote the standards chapter in the manifesto. 


I think that the equivalent of app server based on events is inevitable since events will be at the heart of all applications that take sensor data an input.     Work on standards in this area is an old dream, and hope that we'll be able to advance towards it.   

Will Cappel from Gartner has written about two-tier analytics and went back to Immanuel Kant (in the picture above) as support to his thesis.   Kant argued that the human cognition work in two levels:  the first level that grasps objects and raw facts about them, the second level which captures causality between these objects over space and time, applying some levels of simplification to what Kant said, he is right.  Cappel makes the analogy to the analytics world, and says that the first level is satisfied by event processing that process events by filtering, transformation and pattern detection to identify higher level situations.  The second level is satisfied by pattern discovery engines that work on top of the first level.    This is an interesting observation, I think that the picture is somewhat more complicated as there are more tiers.    Event processing detect patterns in real-time, and indeed one of the ways to obtain these patterns are the pattern discovery mechanism over historical data, which may include the results of event processing systems, but should also include many other data items that describe the impact on the environment,  since situation detection triggers actions, and actions impact the environment,  the pattern discovery needs feedback from the outcome along with feedback from the process itself.    The interesting part comes when we add real-time adaptation to the picture, here, in a similar thing to how the cognition works,  the causality relations may change on the fly.   Consider traffic management systems, studies show that these systems are chaotic in nature and one cannot forecast patterns of behavior based on past experience with sufficient accuracy, forecast is limited to about 15 minutes to the future in some cases and the control policies for highway should constantly adapt.    Here we need four tier analytics:
The first tier is the off-line tier which change the setting of the system based on historical learning.  The second tier is the event processing tier which observes and monitorsThe third tier is the real-time forecasting tier which adapts the causalities and make the short term forecastingThe fourth tier is the real-time decision making tier which makes the best decision possible within the time frame allocated for the decision (which may not be the global optimized solution).
Bottom line:  I agree with Cappel about the multi-tier approach,  and pointing out that reality is somewhat more complicated...   

I used a spoiler of the movie "Source Code" as the opening slide to my recent talk in ETH.   The presentation now is uploaded on slideshare.   Enjoy! 

 I realized that I have not posted in this Blog for two weeks,  I have not really disappeared;  I am involved in a proposal for EU project, and last Tuesday we had the "hearings", which is the second phase of decision about  who will win --- the competition is very tough, and we invested a lot in preparation - in fact, never in my life I spent so much time and energy in preparing to a one hour meetings,  as for the details -- if we'll win this project I'll write more about it.  One detail:  the meeting took place in Luxembourg, where many houses look like palaces (see picture above). 


For now -  catching up with some stuff that was published in the last couple of weeks,  I came across a  Forresters' article about smart computing.    Chris Mines from Forrester writes that smart computing is the next big things and viewing the planned investments of organizations, as seen in the figure below, many of the planned investments are in this area:

The claim is that smart is the next big thing - and mentions three points (copied from the posting)

• Improving transactional processes is yesterday's story. The back-office challenges of preparing financial statements, fulfilling customer orders, or tracking inventory are well addressed by enterprise and personal productivity software. These traditional workloads are migrating to cloud computing resources in some cases, but are not creating incremental technology investments nor opportunities to transform how a business operates.
• Optimizing assets is the next important challenge. Especially in service-based industries, favorable economics result from building an asset base and then selling access to those assets in the form of services. Managing and optimizing the use of those assets -- physical, human, financial, and intangible -- is the source of revenue growth and improved margins. Smart computing systems provide the awareness, analytics, and actions required to improve the utilization of a firm's assets. And in most cases, these systems represent incremental investments in computing resource, with the payback coming in the form of more efficient, more highly utilized, and therefore more profitable assets.
• Smart flexes to meet industry-specific challenges. Each industry has a different mix of assets with different optimization opportunities. In high-tech, development and channel resources must be marshaled in the face of accelerating development cycles, long supply chains, and unpredictable consumer tastes. In finance, mountains of data must be sifted to find the best investment opportunities, while managing market and compliance risk and keeping employees productive. Finding the right mix of online and brick-and-mortar, the right product portfolio, and lean inventories are paramount for retail execs. The right portfolio of smart computing solutions will vary widely across these and other industries

This is in line with my previous posting about the new world vs. the old world, we are in the new world, and the IT industry is gradually shifting towards this new world.     Optimizing assets (traffic, power, human) is certainly part of the game,  eliminating problems before they happen (proactive computing) is another aspect.

A relatively recent posting from MSFT.   In this posting there is a question about the relationship between the batch oriented Map Reduce,  and the on-line oriented event processing.   The answer, according to MSFT is - event processing can be used as a reducer in the Map Reduce, where there are multiple copies of an event processing engine perform the reduce function.


I have written before about offline event processing,  with the insight that event processing is useful not only in online, but in offline, since it provides both efficient implementation and high-level abstraction in certain functions (pattern matching, aggregation and more) that makes it also attractive to use in batch.  


Of course, another synergy may be using event processing within real-time hadoop, like Darkstar.  as is frequently articulated by Colin Clark.    

In Wednesday we held an open conference call of EPTS members.  The call discussed EPTS activities for 2012.  While in the past EPTS concentrated in working groups that have created glossary, reference architecture, investigated use cases and languages, and some other stuff, the idea is that in 2012 EPTS will attempt to address larger audience by creating online magazine which will consist of:  news (including connecting to social media),  general material about event processing (starting point: FAQ taken from analysts in this area),  and industry specific content.   We are now looking for volunteers to set up all aspects: orchestrating the content ("editorial board"),  soliciting readers and topics they are interested ("reach out board"), and a coordinator for sponsorship and advertisements.    Anybody interested to participate - please let me know.   I'll post updates in this Blog

I am browsing through the book "The Live Web" by  Phil Windley, whose sub-title is "Building Event-Based Connections in the Cloud".   Phil sent me the book (with handwritten dedication), and starting from the back cover I liked his motivating examples: "Imagine a world in which your phone automatically mutes the ringer when you start watching a movie" (actually today my mobile phone rang when I was giving a talk, forgot to mute it) or another one  "imagine a workd in which your alarm clock sets itself based on your schedule and other information like weather, traffic, and your past behavior" (I don't need much sleep so getting up early without alarm clock, and some members of my family have the habit to turn of the alarm clock and get back to sleep, but I am sure it is helpful for some people). 
This reminds me of many examples that I used over the years to explain either event processing or autonomic computing (e.g. the refrigerator invited the technician and more).   Phil's answer to realize these scenarios is what he calls "the live web", which according to him is the evolution beyond Web 2.0 which is still static web. I think terms like "the active Internet" or "The event web" was used before in a similar context.  Much of the book deal with projection of event processing to the web interfaces,and working in web environment. Some of it is dedicated to events, their semantics, and operations around them.  He also dedicates chapter to the current hot topics - cloud and mobile and connects them to the story of the live web.   Phil also founded a start-up called Kynetx to implement these ideas, the book also describes their system and language in detail.
Overall - interesting book, both for those who want to understand the principles and those who wish to drill down on the details of how to build such applications (or learn the Kynetx stuff)...   It is also another book that explain the principle of programming with events -- although we somewhat different perspectives of the other books. 

One of my IBM colleagues, Jim Spohrer, the IBM University Relations Director,  has written about the need of students to receive transdisciplinary education.   Indeed we find more and more that doing things require not just multidisciplinary skills, of people from several disciplines working together, but also transdisciplinary skills, where people have good skills in several disciplines and a single person knows how to integrate these disciplines.  This is, of course, contrary to the academic tradition, where promotion processes for faculty members advocate the association with a single well-defined discipline.    We can see much more  transdisciplinary programs today then 20 years ago.  I believe that a program in which a student learns combination of computer science with another discipline (biology, business, mechanical engineering, cognitive psychology and more...),  not only at the intersection between the disciplines, but rather at the union, of having deep understanding in both disciplines as the type of skills required in the marketplace.  
  

While surfing the web to find something,  through the wonders of Google search, I reached the site of the Langley flying school and came across the illustration above under the title "the pilot decision making process". 
The cycle starts with a situation, which is defined as a critical situation which requires action.   The detection and identification of such a situation is what starts the process, thus the decision process is inherently event driven, then the pilot needs to assess the options,  sometimes the options are very obvious, and sometimes there is a creativity in that process,  then there is a need to chose among the options, act according to the selected action and assess whether the situation has been resolved.   Note that this may be either reactive (situation already happened) or proactive (situation is expected to happen).  Of course, when possible it is better to be proactive, it is much better to eliminate crash of the aircraft then try to rescue the crew and passengers when the crash already happened, both from economic point of view and from safety to human life point of view.     Note that the pilot needs to react to critical situations, for the process-driven business as usual case, an autonomic pilot is sufficient.    Can we extend the autonomic pilot also to deal with critical situations?  

A recent article in the IBM developerWorks discusses two ways to enrich data used for rules from external databases,  one of them is doing the enrichment in the request level, before calling the "decision server" (which is the current name for using BRMS system using the request-response protocol), the other one is doing enrichment during the rule processing itself. The article describes how each of these options is done and also discusses pros and cons,  the benefits of enrichment by the request level are - less complexity, and better performance of the rule component; the benefits off enrichment at the rule level are - handling dynamic data and more specialization for the exact data that is being used by the rule.


Thinking about event processing -- there is similarity to the BRMS case,  event can be enriched both by the event producer and as part of the event processing itself, the arguments are not far from those in the BRMS case,  there is one fundamental difference in event processing --  the work is not done using the "request-response" protocol,  moreover, the different part of the system are decoupled, thus the event producer does not necessarily know what purposes the event is going to be used, thus there may be different types of enrichment needed for different uses.   The dynamic aspect is applicable here, and there may be some race conditions in highly dynamic systems between updates in the database that was enriched and its use in enrichment, unless the event processing enrichment system locks the data in the database until it is being used in the event processing system, which requires the event processing system to exhibit a transactional behavior for part of it, but I'll not get now into this issue,


Bottom line:   The considerations in event driven architecture are somewhat different than the request-response systems that are the most common one in computing.