For this experiment I wanted to assess how difficult it would be to create a small Apache Spark cluster using an Ubuntu Desktop machine as a master, and a single Raspberry Pi as one of the worker nodes. The desktop instance also hosts a worker node as well. The plan is to eventually expand the number of Raspberry Pi nodes over time, since they are quite inexpensive — less than USD $100.00 for a Raspberry Pi 4.

The Hardware

  • Ubuntu Desktop running Ubuntu 20.04.2 LTS. This is my main work machine, and will be the Master node.
  • A cheap network switch from…

Image for post
Image for post
Gilbert Strang, a great professor of Mathematics at M.I.T. His excellent book “Linear Algebra and Its Applications (third edition)” is a great book, and he has a number of excellent YouTube lectures.

Computing the Inverse in NumPy

Suppose we are given a matrix M:


Image for post
Image for post

Motivation

My goal was to add a user-commenting capabilities to a blog, without having to pay monthly fees like Disqus. I also wanted to keep things as inexpensive as possible, so where possible no dedicated servers or traditional databases. Lastly, I wanted to do it all on Google Cloud Platform — I have had a good experience with deploying infrequently-used containerized applications using Cloud Run. Like Cloud Run, both BigQuery and Cloud Functions have a free-tier:

  • BigQuery has 10 GB of free storage, plus 1 TB of queries free per month
  • Cloud Functions has 2 million invocations per month, 400,000 GB-seconds…

Scott Ralph

Cloud Architect, AI/ML Scientist, Big Data Practitioner

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store