Posted on

A Moment To Rave About Server-less Computing

Knowledge Leaps now uses AWS Lambda. A Server-less compute technology to parallelize some of the more time-costly functions.

In layman's terms, servers are great but they have finite capacity for calculations, much like your own computer can get stuck when you have too many applications open at once, or that spreadsheet is just too large.

Server-less computing gives you the benefit of computing power without the capacity issues that a single server brings to the party. On AWS you can use up to 1024 server-less compute functions to speed up calculations. There are some limitations, which I won't go in to, but needless-to-say this technology has reduced Knowledge Leaps  compute times down by a factor of 50. Thank you Jeff!

Posted on

Building An Asset, Being Strategic, Learning Important Lessons

Since shifting out of a pure-play service company to building a product-led  company, I am now seeing what it is to be strategic.

In building a product, you are investing in an asset. Investing in an asset forces you to make strategic decisions since the product features define the course and goals for a company. When resources are limited, decision-making needs to be better since the direction these decisions impose on your company's direction are costly to undo.

Bootstrapping the development Knowledge Leaps for the past three years has been eye-opening and a great learning opportunity. The top lessons learnt so far are:

  1. Don't invest money in features that don't make it easier to use the product, today.
  2. Use the product, experience the pain points, then write the scope for the next build.
  3. Get the basics done right before moving on to build more advanced features.
  4. Work with the right team.

Fundamentally, I have learnt that if I am  allocating finite resources that have a compounding effect on my company then I am making the right  strategic.

 

Posted on

Parallelization Begins

Having built a bullet-proof k-fold analytics engine, we have begun the process of migrating it to a parallel computing framework. As the size of the datasets that Knowledge Leaps is processing has increased in terms of volume and quantity, switching to a parallel framework will add scalable improvements in speed and performance. While we had limited the number of cross validations (the k value) to a maximum of 10, we will be able to increase it further with a minimal increase in compute time and much improved accuracy calculations.

Adding parellel-ization to the batch data engineering functionality will also increase the data throughput of the application. Our aim is to deliver a 10X - 20X improvements data throughput on larger datasets.