Serverless Cloud Import System

Part One: Linux FUSE Cloud Storage Mount

Introduction

  1. Part 1: Linux FUSE Cloud Storage Mount — solution for a cloud-based IDE (e.g. AWS Cloud9)
  2. Part 2: Python Cloud Importer — solution for a serverless environment (e.g. AWS Lambda and Fargate)
  3. Part 3: Deployment Package Optimization — solution for optimal packaging between cloud function local storage, shared EFS and Cloud Storage

Acknowledgments

Yes, we follow the news

Industry Landscape

Operating System (Linux)

Installation Packages

  • Package dependencies are checked and all missing dependent packages are installed first
  • All distribution files are copied to some predefined location, such as /usr/lib
  • Symbolic links to the package executable from another predefined location, such /usr/bin, are created
  • Automatic start of daemon processes, where appropriate, is configured
  • Symbolic links, environment variables or other forms of configuration are established making package source and binary files accessible during the compilation, link and run-time processes (this is an extreme case for languages such as C/C++)
  • Automatic compilation of binary extension packages (in the case of Python)

Run-time and Programming Language-Specific Packages

C/C++

Python

  • Installation target
  • Python environments
  • Package specification
  • Package format
  • Package repository
  • Installation tool

Docker

Cloud Environment

How did it all start?

CAIOS Cloud Shelf

Linux FUSE Cloud Storage Mount

  • It is fast enough and does not incur any significant latency penalty
  • It allows having a fine-grained access control to different libraries
  • It is not limited to Python, but rather is equally applicable to any language, run-time environment, data, and executable
Fig 1: Cloud 9 with goofys Mount Solution
  • We connect AWS Cloud 9 environment to an AWS CodeCommit bootstrap repository, which is thus automatically cloned into the Cloud 9 environment file system
  • After the Cloud 9 environment is up and running, we invoke the bootstrap/run.sh script to download the goofys daemon executable, to copy the local.sh script file to the /etc/profile.d folder and to invoke sudo reboot
  • Now, at every boot of Cloud 9 environment, the /etc/profile.d/local.sh will be automatically executed. It will establish the goofys mount connection between some local folder to some AWS S3 bucket and will adjust PATH, PKG_CONFIG_PATH, and PYTHONPATH environment variables accordingly
  • After that, not only Python modules could be imported from the S3 bucket, but also Linux executables (aka tools) could be invoked directly and Linux shared libraries could be built up correctly

--

--

Software technologist/architect; connecting dots across multiple disciplines; C-level mentoring

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Asher Sterkin

Software technologist/architect; connecting dots across multiple disciplines; C-level mentoring