For this experiment I wanted to assess how difficult it would be to create a small Apache Spark cluster using an Ubuntu Desktop machine as a master, and a single Raspberry Pi as one of the worker nodes. The desktop instance also hosts a worker node as well. The plan is to eventually expand the number of Raspberry Pi nodes over time, since they are quite inexpensive — less than USD $100.00 for a Raspberry Pi 4.
The Hardware
Computing the Inverse in NumPy
Suppose we are given a matrix M:
Motivation
My goal was to add a user-commenting capabilities to a blog, without having to pay monthly fees like Disqus. I also wanted to keep things as inexpensive as possible, so where possible no dedicated servers or traditional databases. Lastly, I wanted to do it all on Google Cloud Platform — I have had a good experience with deploying infrequently-used containerized applications using Cloud Run. Like Cloud Run, both BigQuery and Cloud Functions have a free-tier:
- BigQuery has 10 GB of free storage, plus 1 TB of queries free per month
- Cloud Functions has 2 million invocations per month, 400,000 GB-seconds…
Scott Ralph
Cloud Architect, AI/ML Scientist, Big Data Practitioner
