Context and Problem Statement

As it was argued elsewhere in another article an order of magnitude improvement of software development process productivity is required in order to realize full potential of the Serverless Cloud technology. Existing tool chains are conceptually locked in the 50 years old Unix model and are completely inadequate to hyper-fast yet bug-free development process required nowadays.

In particular, we are looking for a way to completely liberate application developers from any DevOpSec concerns by pushing all boilerplate script generation to automatic tools. More specifically, we would like to have a tool that would automatically generate all cloud platform-specific deployment scripts…

Introduction

Python Cloud Importer was developed as a part of the Cloud AI Operating System (CAIOS) project described at a high level in a separate article. Here, we are going to provide a more detailed account of motivation and internal design of this part of the system. This is the first part of a three-part series organized as follows:

1. Part 1: Linux FUSE Cloud Storage Mount — solution for a cloud-based IDE (e.g. AWS Cloud9)
2. Part 2: Python Cloud Importer — solution for a serverless environment (e.g. AWS Lambda and Fargate)
3. Part 3: Deployment Package Optimization — solution for optimal packaging…

Introduction

Python Cloud Importer was developed as a part of the Cloud AI Operating System (CAIOS) project described at a high level in a separate article. Here, we are going to provide a more detailed account of motivation and internal design of this part of the system. This is the second part of a three-part series organized as follows:

1. Part 1: Linux FUSE Cloud Storage Mount — solution for a cloud-based IDE (e.g. AWS Cloud9)
2. Part 2: Python Cloud Importer — solution for a serverless environment (e.g. AWS Lambda and Fargate)
3. Part 3: Deployment Package Optimization — solution for optimal packaging…

Introduction

Python Cloud Importer was developed as a part of the Cloud AI Operating System (CAIOS) project described at a high level in a separate article. Here, we are going to provide a more detailed account of motivation and internal design of this part of the system. This is the third part of a three-part series organized as follows:

1. Part 1: Linux FUSE Cloud Storage Mount — solution for a cloud-based IDE (e.g. AWS Cloud9)
2. Part 2: Python Cloud Importer — solution for a serverless environment (e.g. AWS Lambda and Fargate)
3. Part 3: Deployment Package Optimization — solution for optimal packaging…

Introduction

Unlike containers, which were an incremental change introducing smaller VMs with less isolation, serverless technology has produced truly disruptive change, yet to be fully absorbed by the software industry. We need to stop talking about faster horses and start talking about race cars.

While cloud functions (e.g., AWS Lambda) constitute a crucial ingredient in the serverless revolution, they are just one piece of a bigger puzzle. To comprehend the full meaning of this paradigm shift, we have to look at serverless storage, messaging, APIs, orchestration, access control, and metering services glued together by cloud functions. The cumulative effect of using…

Setting the Stage

My previous post was about Consistent Modeling of Serverless long-running background tasks. There, the following Process Model diagram of a social network crawler was presented:

In such architecture, the whole process of collecting data about a particular MakirOto system user is orchestrated by an AWS Step Function. This type of functionality has many titles such as distributed transaction, workflow, Saga or Process Manager, to name a few.

Yan Cui published a very good article about using the AWS Step Functions and AWS Lambda for implementing the Saga Pattern. …

Warming Up

In the previous post I’ve proposed a new approach to the Serverless Architecture specification. In this post, I argued, that the current modeling practices use too limited vocabulary and produce artifacts without consistent semantics:

As a running sample I chose the MakirOto application, presented by IL AWS Solution Architects Team at the recent AWS Tel Aviv Summit:

As a result of applying new modeling approach to the MakirOto Frontend component the following high-level Process Model was produced:

What’s the Problem?

With Serverless computing platforms we, software developers, can at last put aside all irrelevant technicalities and start delivering what we are payed for — business features. It is, indeed, a potentially tectonic shift in the software industry (S. Wardley suggests some important strategic insights on the subject).

But look at the picture above. This is how we typically document our Serverless architectures. It is impossible to assign precise and unambiguous meaning to neither the individual elements, nor the diagram as a whole. This diagram is too ambiguous.