thanks for your encouragement. You asked a serious question, that deserves a thoughtful response. I will try my best within the available time constraints.
I think BPM and AWS Step Functions, although sharing partially the same computational graph formalism, are tools suitable for problems at different scales. AWS Step Functions are still a relatively low-level mechanism of orchestrating multiple Lambda functions. It’s far from perfect, but some non-trivial stuff could be done with it. As an illustration, here a AWS Step Functions model (I refuse calling it “state machine”) for the breadth-first traversing of an organization shareholders graph:
In fact, this a solution of stacking multiple Step Functions each one representing one layer of the shareholders graph. It’s ugly in some senses (Step Functions do not have proper asynchronous calls mechanism, activity does not work for me and I was enforced to perform dumb poling loops). But it does the job at scale, has zero servers underneath and allows me to keep narrowly focused Lambdas to do one thing only. Error handling and recovery is a nightmare, but, still, it’s a big step towards the direction. What I would like to see is some way to express the graph structure in a programming language (say Python) letting the infrastructure to generate all YAMLs automatically. Something similar to what happens with TensorFlow ML, for example.
Implementing such task in BPM would be IMHO inappropriate, even if it is transparently serverless and well integrated with Lambda. On the other hand, I’m not sure I would embark on implementing a really complex business process with AWS Step Functions, even though some customers reported particular success with it. Also, BPM is usually associated with some form of WYSIWYG IDE tools letting people without strong software background to define a business process flow.
What I was looking for is a software development solution, expressed in a high-level code and m.b. visualized for convenience, but not development. We tried to develop software with UML picture 15 years ago and it did not work.
What the both mechanisms and software industry as a whole could benefit from is having a strong theoretically sound formalism for describing sequential, parallel, and asynchronous computations. Many applications would benefit from this including Machine Learning, Batch Processing, Lambda Orchestration and Business Process Management. We do not have it today. UML (or Harell) statecharts are good for describing asynchronous processing, but are not so well suited for describing sequential processing. Try to describe in a statechart asynchronous data retrieval from DynamoDB or Athena. Even though the process could run asynchronously to achieve better scallability and avoid polling, expressing it as a sequence of states would be still unnatural. On the other hand, UML Activity Diagrams too often introduce confusion between steps and states.
To sum up, I would probably consider using the tool better tailored for particular scale, namely BPM for business processes and AWS Step Functions for serverless orchestration, not forgetting about AWS Glue, Data Pipeline and Batch for big data processing. Having said that, I think we could analyse successful solutions from all directions and cross pollinate them where appropriate. The current state of affairs with multiple silos constantly reinventing the wheel is hard to call healthy.
Hope this clarifies.