Process and Task Mining

Authors: Alan Pelz-Sharpe

Vendor Landscape 2021-22

The Essentials

Every time a new business application is installed, a business process is re-engineered, or an organization starts a “digital” project, the first step is to undertake business analysis. The goal of business analysis is to understand your current situation in detail so that you know what you are working with. Only then can you determine how to change, improve, or transform it.

The business analysis may run for days or months, and it may involve an army of outside consultants or a single internal employee. Likewise, the analysis may be done well or poorly: every project is different. Though there are readily available templates, structures, standards, and methodologies, for example from long-established bodies like the International Institute of Business Analysis (IIBA), few projects make use of them.

The bottom line is that the business analysis phase of any IT project is the most critical phase by far, yet it is also the one that is typically given the fewest resources and the least time and effort. As a result, most IT projects fail or fall short of expectations. Indeed, the failure rate for IT projects is high, and in our analysis the number one reason for their failure is the poor business analysis associated with them. Over the last couple of years, technologies in the form of process and task mining tools that claim to automate much of a business analyst’s work have gained prominence in the market as a potential solution to this situation. This report provides a critical analysis of these tools’ sudden increased popularity, examines where and how they can bring measurable benefits to business analysis, and assesses the size of this niche market and how it might evolve.

The Traditional Approach to Business Analysis

At a basic level, business analysis involves observing, measuring, talking, and mapping.

  • Observing how people work, where they work, and in what order they undertake tasks and processes.
  • Measuring how long work takes and how much it costs to do.
  • Talking to workers to understand the vagaries, challenges, and workarounds they encounter in doing their assigned work.
  • Mapping this information into an understandable format, whether a flow chart or a document.

Such work lets the organization understand the “current state,” or “as is” situation, and in almost every case, reveals much that was previously unknown. In fact, in many – if not most – situations, business analysis provides the first picture of how work is truly undertaken in the organization, as opposed to how managers, executives, supervisors, and business owners think it is undertaken.

Doing this work successfully requires skilled business analysts who are adept at gathering information and parsing it into a unified and actionable form. Gathering information from workers who may be reluctant to share or who find it hard to explain exactly what they do and how they do it is tricky; understanding what happens at a data and system level can be equally tricky.

Organizations tend to assume that their business applications, data silos, and networks do precisely what they were configured and programmed to do. Hence, other than potentially pulling some reports or log data from an IT system or recording end users’ frustrations with the “system,” little true analysis of the underlying data processes typically occurs. We know for sure that such assumptions are incorrect – keep in mind the adage about what happens when we “ass-u-me.”

At one extreme, an organization’s complex and expensive decades-old ERP system may appear to work well. But closer observation at the data level may reveal human workarounds, bottlenecks, and unused or unusable functionality. It is not uncommon to find that workers use only a small amount of the functionality associated with a software license. At the other extreme, it may be assumed that staff members store and file critical business documents in their assigned folder locations, whereas they are really storing them in their email or in non-approved cloud file-sharing applications like Google Drive. As businesses have become more digitized, and in many cases now distributed and remote, the need to understand what truly happens at a granular level has become critical. Without that, most attempts to improve efficiencies, reduce costs, or transform will fail.

Good business analysts are in very short supply, and their skills are often underappreciated and underused. That is a business culture and project management situation that can be improved quickly, simply by hiring quality analysts and properly budgeting and accounting for their work. Understanding what is happening within IT systems themselves is a tougher challenge, and until recently the tools to do this either didn’t exist or had limited capabilities. This explains the wave of excitement and interest in process and task mining tools that are now available and that claim to fix this problem.

Process and Task Mining Technology

The term “process mining” has become popular in the technology sector, while “task mining” is much less known, though equally important. Process and task mining tools are complementary but work quite differently and serve different purposes. This section looks at each in turn: what they are, what they do, and how they work.

Process mining

Essentially, process mining tools read the log data within a process application. Log data is the granular data generated by the system each time a transaction occurs (see Figure 1). By reading and extracting this data, process mining tools can parse the data and map it into an easily understood format such as a dashboard or set of visualizations.

Diving a bit deeper, process mining tools normally follow these steps: data ingestion, process discovery, process analysis, and process benchmarking (see Figure 2).

Data ingestion
Each time something happens within a process or business application, such as creating a new case or making a financial transaction, the system leaves a trail in the form of timestamps, case ID’s, or other data. To do anything useful, process mining first needs to capture this data trail, whether through reading and copying live system data or simply uploading a spreadsheet of log data from the system, for example. Likewise, the data trail may come from a single system or multiple interconnected systems. In raw form, such data is pretty much unintelligible.

Process discovery & analysis
In these stages, the process mining tool pulls this event data together to visualize chronologically all the steps and activities involved in the process, creating what some term a “digital twin.” This visualization allows you to explore each process path, see where there are variations, identify bottlenecks, and find other issues. You can use the data in many ways to see what is working, what is not, what steps are regularly skipped, and to some degree why this is happening. From a usability perspective, how well the tool visualizes the data and how it allows you to explore the data is critical. Every tool on the market has its own spin (see Figure 3 for an example).

Process benchmarking
Process benchmarking does what it says on the packet: you can benchmark how your business process works versus how it should be working. Or, you can benchmark a process running in one location against the same process running in another, and so on. For example, it may take longer to onboard a new employee in one department than another, and by comparing and contrasting the two you have a chance to find out why.

Process mining tools can bring a lot of value, but without expert business analysts to use them and interpret the data, they bring no value. Process mining tools do not replace business analysts; rather, they augment and accelerate their work. We state this because some buyers of process mining tools seem to think these tools can automate the work of business analysts. That is incorrect: process mining tools are used by business analysts and do not replace them.

Figure 1
Example of Log Data

Figure 2
Process Mining Steps

Figure 3
Data Visualization Example from ABBYY Timeline

Task mining

Task mining tools are often thought of as much the same thing as process mining tools and tend to fall under their shadow, but they are quite different and equally important in the business analysis mix. Where process mining tools examine the underlying process data embedded in business applications, task mining tools examine the activities of workers at the desktop screen (UI) level. To put it another way, process mining examines the back end, while task mining examines the front end.

Fundamentally, task mining systems follow a similar operational path to process mining tools in that they capture and ingest data, parse and visualize it into a usable form, and allow you to analyze it and hopefully figure out how to improve current work activities.
From a technical standpoint, though, task mining tools read and capture keystrokes, mouse swipes, clicks, and screen jumps that a worker undertakes on their device, be that a cellphone, laptop, or desktop. They often use technologies like optical character recognition (OCR). In some cases, natural language processing (NLP) technologies are also used to provide context and/or categorization to the capture data: for example, “this is a customer inquiry,” or “this is an internal record update.”

Context is key here, as task-related data is by definition messier than system-level data. For instance, you need to be able to understand when somebody is randomly checking the news headlines on their screen and eliminate that set of activities from core work tasks. Parsing, cleaning, and making sense of such things as screen scrolls in context is no easy undertaking. In the process mining world, the data captured, though difficult to comprehend, is highly structured. In the task mining world, there is an ocean of unstructured variations as every worker doing the same task may well do it a little differently.
Therefore, task mining tools need to correlate different data points together, and in turn match that correlated data to business activities, in a format that business analysts can use to first understand and then optimize further (see Figure 4).

But wait, there’s more!

Task and process mining separately make up the bulk of the tools on the market today. But new tools are coming that take a different approach and to some degree build on top of, or at least alongside, the work of task and process mining. These tools aim to add context to work situations. Process and task mining tools excel at understanding repetitive activities, such as updating a client account or processing an application. Where they are less valuable is in understanding unstructured work activities such as knowledge work, or tasks related to work such as coordinating meetings, searching for relevant information, etc. Though not yet a defined market, we can call these knowledge-mapping or work-mapping tools. Examples of such emerging technologies include NolijWork and Aithin. As workers continue to move from corporate offices to home and hybrid working environments, we expect to see more tools enter this emerging market.
Another set of technologies loosely labeled “process discovery” aim to bridge the gap between what is happening at the process/data level and the human level. These tools map the human interactions with the system, similar in concept to task mining, but in practice they are more a meld of process and task mining tools than a defined and separate technolog

Figure 4
Process Map Example from Minit

The Vendor Landscape

Deep Analysis does not rank or score vendors. The technology vendors mentioned in this report all have good technology; some firms are larger and some smaller, but all of them work and will be a good fit for the right buyer. That said, this is a particularly difficult vendor landscape for buyers to navigate as there has been an explosion in innovation and new approaches recently, even though the concept of process and task mining is not new.
At the top end of the market in terms of scale and revenue sits Celonis alongside Signavio (SAP) and ABBYY Timeline. Smaller but still significant vendors follow in the shape of FortressIQ, Minit, myInvenio (IBM), and QPR. Then there is a growing list of smaller vendors such as PuzzleData, Everflow, Logpickr, Lana Labs, Alana, and UltimateSuite. More start-ups are coming to the market, and we expect the number of visible players to increase significantly over the next two years.

Current Market Dynamics

The surge of interest in process and task mining is driven largely by two key market dynamics:

  1. The surge of growth in the robotic process automation (RPA) world.
  2. The upheaval in work patterns forced upon organizations by the pandemic.

Both coincided with pre-pandemic interest and investment in ambitious, and in many cases flawed, digital transformation initiatives. Together, these factors have created a near perfect storm for process and task mining tools. Organizations across the world are in a state of (often unexpected) change, moves to automate and digitize work have accelerated, and to manage any of that you need to understand the current state. Hence the sudden boom of interest in process and task mining products.

Rapid RPA growth in particular is driving the market. The RPA market has grown exponentially over the past five years as these tools have been seen as an easier and lower-cost alternative to traditional business process management (BPM) systems. RPA tools automate tasks that occur the same way every time, not entire processes. But overly ambitious marketing and sales, supported by massive external investment, have posited RPA solutions as capable of doing much more. Organizations that in some cases have bought thousands of licenses to deploy RPA bots (or digital workers, as some prefer to term them) have discovered that though they are easy to deploy technically, using these bots effectively to generate business value is much harder than they expected. Often, incremental automation using bots in turn triggers unexpected changes elsewhere in business processes. Therefore, the need to thoroughly map and understand the full picture of underlying business processes is now understood to be critical, and similarly the screen-level task automation that RPA is often used for requires more thorough analysis before changes are made than many buyers first thought. Process and task mining tools are clearly needed to fill a gap in the business automation market.

Mergers, acquisitions, and investments

Unsurprisingly, process and task mining vendors are in the M&A spotlight with strong and growing market valuations. There have been a few notable and recent acquisitions and investments (see Tables 1 and 2) and at Deep Analysis we are certain that more are in the pipeline. Process and task mining functionality is now seen by many larger business applications to be a critical gap or missing link in their portfolios, and as such there is every expectation that the M&A buzz will be with us for some time. Though not a particularly large market in terms of vendor size and revenue, it is nonetheless a hot one.

The high price SAP paid for Signavio is unlikely to be matched, though Celonis is clearly on the path to a potential IPO this year or next. Having bought myInvenio, IBM is out of play, leaving only one other big hitter, Oracle. We predict there will be a lot of smaller, though still high-multiple, tuck-in deals over the next 18 months, in part driven by cash-rich RPA vendors looking to expand their portfolios and grow their existing businesses. Notably, however, strongly funded Kryon took the decision to build its own process mining capabilities. The major announcement by BluePrism in October 2021 that it will partner with ABBYY Timeline is a sign of more such strategic partnerships and potential acquisitions to come.

Table 1
Process and Task Mining Vendor Acquisitions

Table 2
Process and Task Mining Vendor Investments

Market Forecast

The process and task mining market is relatively small in terms of revenue, and despite some major funding and acquisitions it will remain a niche market. As of 2021, we size the market at approximately $460 million with additional revenue coming from directly associated services. Though the market is set to grow substantially over the next few years, we do not see such fast growth as sustainable in the years that follow, as noted below. Instead, we foresee modest but steady future growth (see Figure 5).

The use of these tools has the potential to expand exponentially, but revenue growth will slow as the functionality becomes bundled and embedded in larger applications and solutions. We predict the market will grow by around 20% each year over the next few years, a prediction that is in stark contrast to other analyst firms that predict growth of around 100% and higher year on year. We believe these estimations are far off the mark, as growth rates for early-stage vendors such as PuzzleData, Dataopolis, and Logpickr will indeed be incredible, but they are growing essentially from zero and scaling their businesses. Similarly, Timeline has had impressive growth since coming into the ABBYY fold, but again this represents individual company performance and is not indicative of the overall global market.

An evolving process and task mining market

For the purposes of this report, we have focused solely on process and task mining tools and referred to the process and task mining “market,” even though we note that these tools can also be seen as part of larger process and analytics markets. That being said, we have identified four key factors that will impact the future of this “market.”

Bundled functionality
As of today, process and task mining are standalone, line-item technologies, but there is a strong industry trend to bundle technology into broader platforms. Over the next few years, we expect to see larger, process-oriented vendors bundle at least basic versions of task and process mining into their suites, systems, and platforms. A market will always exist for independent, best-of-breed process and task mining software, but as this is a requirement for deployment and monitoring it makes sense for it to become part of the overall offering. It would be much like enterprise search, whereby most buyers simply have search functionality embedded into their software systems (e.g., Microsoft and Google) but there remains a small and vibrant best-of-breed market. In our analysis this is the same route that process and task mining will take, with its use broadening and democratizing while its growth as a market sector softens.

An aid to consulting and business analysis
Though we believe process and task mining will be embedded into broader applications and platforms, the nature of business analysis work means that such tools need to span multiple systems, silos, and applications; hence the need for best-of-breed advanced tooling that can be used by internal and external consultants. The challenge to growth here is not a lack of need or opportunity, but rather a dearth of specialist consultants and business analysts who are able to maximize the use of these tools. Indeed, the lack of specialized human skills is the biggest obstacle to growth, in our estimation. There is a perception that process and task mining tools automate the work of business analysis, but as discussed elsewhere in this report, that is only partly true. For process and task mining tools to be of value, skilled analysts need to use them and process the information they present.

An extension to systems monitoring
As of now, most process and task mining tools are used for new automation projects, but the nature of the tools themselves posit them as ongoing monitoring systems. Just as firms like Splunk monitor underlying infrastructure components, these tools monitor live business level activities, thus providing a means to both visualize and reengineer task and process activities. Though some firms today do use these tools for this type of monitoring work, we believe many more will do so in the coming years. This opens a new opportunity for market development for system monitoring firms like Splunk, Software AG, Dynatrace, and Cisco by moving “up the stack.” An early indication of this trend can be seen in the recent launch announcements by both Celonis and Software AG of technologies that will monitor and map streaming IoT data at the process level.

Movement from mapping to predicting
Mapping real world process and task activities is of significant value to organizations that want to improve efficiencies and further automate work. But the potential exists to do more, and to utilize machine learning (ML) and artificial intelligence (AI) to predict future outcomes. We know that many technology vendors are working to develop this kind of functionality. Such a route offers promise as the data these tools gather is accurate and good input material for AI & ML. The challenge is once again less technical in nature, and more human, as few organizations today are able to fully exploit the potential of process and task mining tools’ basic functionality.

Figure 5
Process and Task Mining Growth Forecast

The Industry Analyst Quandary – Which Market?

Industry analysts love to categorize and silo technologies into discrete markets, as can be seen in their creation of Magic Quadrants and Waves. At Deep Analysis we are wary of such defined categorizations, and though this report focuses solely on process and task mining technologies, a fair argument can be made that this is not a true market in and of itself. Instead, we would argue that process and task mining technology vendors play a niche role in much larger markets, primarily broader analytics and system monitoring markets.

Call to Action: Technology Vendors

Process mining is not new, even though the solutions on the market today are a marked improvement over past offerings. Task mining is relatively new, though, and in our estimation has the potential to grow much faster than process mining despite being much smaller in terms of revenue today. Further, it should be noted that the move to hybrid and remote working and the urgent need to re-engineer existing processes and automate new ones plays well for both process and task mining software.

Interest in process and task mining is only going to grow; anything that can shorten the business analysis process and make it more accurate will find a willing buyer. But it would be shortsighted to see process and task mining as simply tools to help accelerate new projects. Used well, these tools can become virtual business dashboards of value to organizations that want to monitor and continually improve the efficiency of business process and work tasks. As these tools gain popularity, few larger automation engagements will go ahead without them playing a role, so if you do not have process and task mining in your portfolio it makes sense to partner or acquire these companies. If you take the acquisition route, expect to pay a high multiple, but affordable deals are possible as there are a number of small start-ups in the space. At the same time, note that after the benchmark set by SAP’s acquisition of Signavio some sellers’ expectations are unrealistically high and you may prefer to wait until the market cools a little before making a move.