My last living grandparent (my mother’s mother) was 98 years old when we once visited her in Negombo. We exchanged cheerful chats over tea, walked, and talked about the kids. For my sons, she was their great-grandmother!

After merely 3 months, we saw her again. But this time, she was stuck to a bed, visibly just the bones and skin remaining in her body, barely being able to move a hand.

For the majority of us (based on un-statistical evidence backed by non-scientific observations), we will see our grandparents die in our 30’s.

We will see their last days, the struggle at the mouth of death, knowing that there’s not much time left in this world.

We will see the way they fall to disease and the way they yearn to unite their families. To see their kids and grandkids one more time.

The helpless days, the final days.

I witnessed both my grandmothers (mother’s mother and father’s mother) passing away around the time Sahaswara (my second son) was born. And I remember noticing the resemblance between a newborn baby and a grandmother in her late nineties fighting death.

Seeing that really puts the Shakespearian writing on the ‘second childhood’ into perspective (from The Seven Ages of Man).

The helplessness in the eyes, fully dependent on the ones that surround you. Trusting they would look after you and keep you safe and fed.

I would sometimes just close my eyes and imagine myself in that state, it’s good to be familiar with it, I thought.

Life (survival) is a miracle…

The thing you also realize is - the stories you had heard as a child and thought ‘that could never happen..’, start happening mostly in your 30’s.

These are stories of despair, from death to divorce and losing health or wealth.

But these could be the same stories that motivate you and remind you to fill your life with un-regrettable experiences now that you are still in your 30’s.

The reminder that you shouldn't or (actually) cannot wait around..

As another famous saying goes - “Every day I look in the mirror and ask, 'If today were the last day of my life, would I want to do what I am about to do today? ' And whenever the answer has been 'No' for too many days in a row, I know I need to change something.”

I’ll leave you with that thought…

I am noting this down in October 2024 just after checking out the new AI podcast generation feature of Google’s revamped NotebookLM product.

And I just had an “Oh Shit, Wow!” - moment.

A quick rush of endorphins or whatever that makes you smile unnoticeably and think, I should share this everywhere, write a LinkedIn post about it, post a tweet, share a clip of me using it on Instagram stories!, because this is just brilliant!

A follow-up thought rushes in — “I want others to know that I have unraveled this new magical experience and I discovered it before them (in a way). Maybe it’ll elevate my persona as a well-informed person, up-to-date in cool tech!” haha.

In the world of software and AI that I am in, if you want a killer business, this is the emotion you are looking to erupt in users. Well, it is safe to say that this might be common to many other forms of business as well.

Remember ChatGPT? How we found out about it instantly wanted to show off that we are using it to the rest of the world? Ah yes, showing off.. Great products or experiences make you wanna show them off to others.

Anyway, I can further argue that this phenomenon contributes to social content going viral as well. Whenever you are sharing a reel or a tiktok with someone else, the indirect thought can be - “Hey look — I found this very funny thing… yes, I am cool like that.” Just notice, we are sort of trying to project ourselves through that content as well.

What about music? Us sharing songs we like, making covers of them? We sort of do the same thing, while relating those to ourselves in a way that says - “hey look, I listened to cool songs like this because I’m cool.”

But is this a framework? something we could reverse engineer from when coming up with a product idea? I’ll leave that for further thought…

[Present day]

It’s been a few months, and I just came back to this post in my drafts..

Honestly, I didn’t post this because it sort of felt like a stupid point of view after writing it on a whim. But hey, thought of just letting it go.

“Let’s take him out this Saturday.” The doctor’s words came out suddenly looking at the oral glucose tolerance test results. Although we were expecting the arrival in a couple of weeks, this unanticipated reduction of two weeks from the plan came out with a shock. I believe that in life there are two kinds of shocks according to the very next emotion that follows them, the extremely happy ones or otherwise. This time it was somewhat in the middle, hence I found it hard to explain. One thought said that I’m going to see my son in a few days’ time. The next thought quickly followed reminding me of the change. A life-long change.

We got in the car to go back home. I could see the same expression of excitement mixed with a hint of fear in Tharani’s eyes. But she chose to talk about the plans rather than the emotions. She wished for a normal delivery until the last moment. Although a procedure of induction could grant her that wish even now, we decided that the cesarean surgery was the safer and the less worrying option. This was our first child. The decision had given us just two days until the trip to the hospital. The following day we decided to spend shopping for more things that we thought we should have at the baby’s arrival. As with all significant events in life, we made sure that we spent more money than what’s needed for this one. But for a change, this time I didn’t complain.

Appropriately, along with the macintosh sheets, swaddling blankets and feeding pillows, the shopping list also contained a Christmas tree. This was the epilogue of November 2021, hence Tharani insisted that we should have all the Christmas decorations done when he arrives home. So we spent that evening setting up a Christmas tree while the cot was still in a box. The next morning we woke up for a call from Tharani’s father who reminded us that we were late for the daane (offerings) they had organised for the nearby temple. It was a monthly practice for them, but this time they were dedicating it to the birth of their first grandchild. That felt special.

The temple rush followed by some more last-minute shopping led us to the realisation of the uniqueness of that evening.

The other day my academic supervisor, Uli, was telling me about a casual tradition that Germans tend to follow, where the night before a child’s planned birth, the mother and the father to be would go to an Italian restaurant for dinner which incorporates a glass of red wine. Although we had no wine, we decided to make some Italian at home that evening.

A baked pasta carbonara with bacon wrapped in every cheese that the local Cargills food city offered, cheddar, mozzarella and parmesan. A fresh salad chaser incorporating cucumber, lettuce and red apples. Tharani made this Italian recipe-with-a-twist more often than one would guess. I loved it. Also on the side, I grilled some chicken breasts seasoned with almost every spice on our rack, just to make sure we have more food than we could finish that night.

We forgot the candles because we were too hungry by the time we finished serving all the food. So we sat down, ate and reflected on life. These nights would never be the same again. And we were looking forward to the change. The IG story said “homemade dinner just for two today”.

The anticipated day arrived. The usual morning coffee tasted the same. The utterance of good morning sounded the same. But I guess there was a slight hush of nervousness in my breath. After all, it was going to be a major surgery.

But Tharani didn’t show any nerves at all, although she is usually the one with most of the complaints. She told me that all she was worried about was the spinal anaesthesia injection before the procedure.

She had assisted many cesarean surgeries before and remembered how painful it was usually for the patient. Being a medic herself, she knew every detail about what they were going to do on her that day. In situations like these, knowledge can sometimes be the bearer of fear.

But she looked calmly at her tummy sitting down on the sofa and told that she might miss the baby bump after today.

The rest of the day went pretty fast, at least for me, with everything from loading the suitcase into the car and unpacking it in the hospital room. Nurses in and out of the room until one came in suddenly and asked Tharani to get into the surgery gown. It was around 7 o’clock in the evening. The events that soon followed would be, by far, the most emotionally striking in my entire life. And I didn’t have a single idea about them while accompanying the rolling bed carrying Tharani into the lift, down to the second floor and through a two-part door that was named “operation theatre”. “Please wait here.” The person taking Tharani through the doors told me. “We’ll call you in.” I stopped and turned to see some others standing outside that door as well. Anxious. Pacing.

The area right outside the theatre door didn’t have any sitting arrangements. It’s as if they knew that none of the people waiting there would be sitting down. Nevertheless, I joined with the others, waiting.

I wanted to be with Tharani when they were giving that spinal anaesthesia injection. This was the thought that revolved in my head at that time. But it seemed like I would miss it. I realised. In a few minutes, a person rushed out of the door and uttered “Mirantha”. As I went in he showed me a changing room with piles of theatre scrubs and face shields. I changed quickly to go out and notice several other guys waiting in the same attire. Sitting. Anxious. With just phones in their hands ready to capture the first moments of their babies coming out to this world. And there I was, also fitting in.

It was surreal to think how everyone sitting there next to me was going through one of the most special moments in each of their individual lives. The many theatre rooms behind us delivering those emotions in bulk, in a queue.

The breathing under the face mask had formed a slight mist on the face shield making my view a bit blurry. I noticed that the guy seated next to me was also trying to wipe his shield. So similar I thought, yet two entirely different stories.

A few moments later, time started to move a bit faster, I felt. A person came and escorted me along a corridor to a room on the right. As I reached the door, I saw Tharani lying on the theatre bed under waves of green colour sheets. A tidal wave of blue colour cloths was flowing above her. I figured that was the gynaecologists, anesthesiologists and nurses wearing blue theatre scrubs. They were already on it. I had missed the spinal injection.

I was directed up to the top of the bed where a chair was placed to sit down. “Hey..” I said. Tharani took a second to recognise that it was me near her. It was all happening so quickly. Her hands were tied to either side of the bed, on one, a big cannula had been pierced in. I held her other hand and leaned in behind the screen that a green colour cloth had formed in front of us.

While I stayed looking at her face, she told me that the anesthesiologist was a lecturer she knew from her medical school and that the spinal injection didn’t hurt as much as she thought it would. The excitement in her voice overshadowed the fear in her eyes. It did that for me too. Soon after, she told me that she had some difficulty in breathing. I quickly called a person in blue nearby who removed her face mask and administered some nasal drops. “It’s just the coldness in the room.” The person smiled and uttered.

Honestly, writing this I figured that I cannot exactly recall what Tharani and I mumbled in the next few minutes. I think I kept asking her about the breathing while still trying to keep calm.

Suddenly the doc peeped from above the screen and said that they were going to apply some pressure. The bed wobbled along with my thoughts, while I kept telling myself, it was okay. It was normal. A few moments later, we heard. We heard the cry. The long-awaited cry. Just twice, not much… The next moment we saw, we saw him above our heads. The doc held the child and brought him to our side of the screen.

He didn’t cry. I guessed he was cold. But he didn’t complain.

An attendant had already taken my phone to take the photos. He came from the other side and asked us to smile for a group picture. And there it was. Our first picture with all our hopes bundled in a life form.

That done, they took the baby away for further checks. I saw the needle and the thread from above the screen. I wished to stay with Tharani until the rest of the surgery was completed, but a person called me and pointed at the door. I objected for a moment to stay a little longer. “I’ll meet you outside,” I remember telling her before letting go of her hand, which I was continuously holding.

Back through the corridor to the changing room, I was out of the theatre area. I called my mother immediately to share the news. “How’s Tharani?”, she asked. “OK, I’m waiting till she comes out.” I hung up.

Back to waiting near the same two-part door from before, I didn’t expect the next one and a half hours to be the hardest part of the evening. Harder than the time inside the theatre.

To the left of the theatre door, was the elevator that routinely brought more pregnant mothers rolling in beds to go in. Same drill. It was like a flashback scene from a movie. Mother taken in, the partner asked to wait near the door for ten thoughtful minutes until called in. And towards the end of the scene, the partner comes out of the door and waits till the mother returns with the baby. Same drill. I looked around and realised that some of them who were waiting there when I was going in were still there. The anxiety slowly started creeping in, I was pacing. I knew everything was ok, but what happened after I came out?

The next moment, the two-part door slightly opened and there was a bed on the other side. A small iron cot with wheels was taken in. It was for the baby to come rolling alongside the mother. Everyone waiting on this side peeped in with the hope of seeing their partner with their baby. Two minutes passed and the doors fully opened. One guy next to me walked up to the bed and helped roll the baby’s cot into the elevator to go start their new life. Back to waiting for others. The caution-biased mind kicks in at times like these and that moment was no exception. I kept on thinking how Tharani was finding it hard to breathe during the surgery. “Was the baby fine? Did the other babies cry more? I hoped they didn’t mix up the babies. Well, no. I have the pictures.” The thoughts kept cycling and cycling and cycling.

The doors opened again. This time the father looked worried. Mother rolled out with the iron cot, it was empty. Maybe their baby needed some special care after birth. Just taking a bit more time.

None of the people standing there talked with each other. They were engulfed in their own worlds, their own thinking. Some of them were constantly on their phones, providing updates to the families, “No. didn’t come out yet. It’s been more than an hour.” My legs were starting to hurt. I kept pacing until I saw another bed waiting against the door. A slight opening showed the hands of the mother. I knew it was Tharani.

As I sighed with relief they came out. Mother and son, both triumphant and rising from the depths of the operation theatre. I smiled.

I held Tharani’s hand as the elevator doors opened. “Ting”.

I didn’t sleep that night. As I waited near his small wooden cot in our smaller than average deluxe hospital room, I had my eyes locked on him. Just two expressions or three. Sleeping peacefully till he takes on the challenges of surviving in this new world.

All the slightest moves, the slightest noises from his mouth, I heard. I listened. He reminded me of how simple life can be. A matter of just being until the next feeding session. That night I wrote —

I want him to start life with the lessons I’ve learned so far in my life.

I want him to know that he should look openly at life and that the world rewards authenticity, not fitting into norms.

I want him to know that he doesn’t have to impress his parents but impress only himself with the way he has lived.

I want him to know that even though he’s the most special person in our lives, it is not like that out there. He’s just one in 7.5 billion, and he should face the world with that attitude. Be kind, everyone is equal and just trying to survive.

I want him to know that it is not his duty but his choice to look after his parents when he’s older. It was our duty to bring out the best in him. And then it is his right to spend his life according to what he wants.

So here we go…

I felt extremely sad when I heard about the passing of musician Sunil Perera last week. The amount of sadness was somewhat surprising because I did not know him personally, nor have I spoken with him ever.

It made me wonder what makes us so emotionally attached to personalities like musicians or, on a wider scale, artists (actors/writers/creators) that we adore. I'm trying to gather some of my thoughts here.

1/ Art relates to our identity

The memories went back to my childhood when my father bought me the ‘Signore’ cassette tape by the Gypsies in 2005. We listened to it on Sundays, laughed, and sang along.

Growing up, being the ‘guitarist’ among my friends, I was a guy who knew these songs and sang them at parties. Experiences with art become parts of our identities. We talk about them all the time.

2/ Artist’s success is his followers’ success

Whenever I saw the Gypsies perform, I saw the audience having a cracking time. It made me proud to be a fan of them and tell others around me how good they were.

Artist’s success somehow translates to their follower’s success. I know it’s a bit weird when you put it like that, but think about the last time talked highly and recommended someone a good song or a movie….

3/ We know all about our favourite artists

The good and the bad. Fame makes artists vulnerable. The ones who turn this vulnerability into impact effortlessly communicate their authentic self to their followers. We love authenticity. This makes us stronger with 1/ and 2/ above.

4/ We cry for the emotions they created in our lives.

In hindsight, I figured I wasn’t sad about the passing of the ‘person’ Sunil Perera. He probably lived a happy life with what he did and his loved ones. But I was sad about the passing of the feelings that the ‘artist’ Sunil Perera had created in my life.

I felt proud when I knew all the words of his songs while singing them with my friends. I felt overjoyed when we laughed at the meaning of those songs and enjoyed the tunes. We simply had so much fun for ourselves with that art.

So ultimately, maybe it’s not what the artist accomplished, but what we accomplished with the art. It’s not what he felt, but what we felt with him. The stronger we felt, the stronger we thought we were connected to him.

At this point, you might wonder whether this is a conclusion about selfishness. But I would argue otherwise. These are natural traits of being human. The point is — it’s only art and artists who can trigger these feelings in us at scale.

RIP Sunil Perera. You have inspired generations…

You can now utilize the power of large language models (LLMs) like ChatGPT to answer questions about your data. In this blog post, I will guide you through the process of implementing a simple web application that integrates ChatGPT to answer queries about your own data.

We will be using the Langchain prompt chaining library to build an interactive chat interface that communicates with OpenAI’s GPT endpoints, and we will perform a semantic search to obtain relevant context for user queries. This approach doesn’t involve any fine-tuning of language models. Instead, we’ll leverage a vector database to power our search and provide the context to the LLM for generating accurate answers.

Demo:

Understanding the Basics:

Before diving into the implementation, it’s crucial to understand some key concepts related to large language models, semantic search, and vector databases.

1. The context for Large Language Models (LLMs):

Context is the background information or relevant details that guide an LLM like ChatGPT in generating meaningful responses. In our case, context is extracted from the user’s data to help the model understand the specific domain and provide accurate answers. By providing an appropriate context, we ensure that the model’s responses are tailored to the user’s dataset and query.

1. Semantic Search:

Semantic search is an advanced search technique that aims to understand the meaning and intent behind a query, rather than just matching keywords. It uses natural language processing (NLP) algorithms to identify relevant context and relationships between words, phrases, and concepts in a dataset. This results in more accurate and relevant search results compared to traditional keyword-based search methods.

1. Vector Databases:

A vector database is a specialized database designed to store and search high-dimensional vectors efficiently. In our implementation, we use Pinecone, a powerful and scalable vector database service. Pinecone enables us to perform fast and accurate semantic search by converting our data into numerical representations (vectors) and indexing them for efficient retrieval.

Implementation Steps:

You can find the full code here — https://github.com/miranthajayatilake/nanoQA2

Steps to quickstart locally:

• Clone the repo git clone https://github.com/miranthajayatilake/nanoQA2.git

• Move into directory cd nanoQA2

• Set up your python>3.8 environment (virtual environment preferred)

• Install the dependencies pip install -r requirements.txt

• Assign the environment variables by running bash env-local.sh. Make sure you have the following API keys and variables replaced

• OpenAI API key (OPENAI_API_KEY). You can obtain this by creating an account at OpenAI

• Pinecone API key and environment name (PINECONE_API_KEY, PINECONE_ENV). Obtain these by making an account at Pinecone

• Next, we have to create an index in the Pinecone account. You can use the create_index.py to do this. Make sure to provide the parameters below.

• create_index.py --pinecone_api_key <asdf> --pinecone_environment <asdf> --index_name <asdf>

• Copy the index name used above (INDEX_NAME)

• Provide a namespace as well (NAMESPACE) just to organize data in the database

• Provide a name that you want your chatbot to have (EGPTNAME)

• Run the web app with streamlit run Chat.py

The repo also gives you instructions on how to deploy the app to the cloud easily.

Code walkthrough:

First, let’s look at the Chat.py script that has chat-related components.

We will import the necessary libraries and initialize Pinecone.

# Import necessary libraries
import os
import streamlit as st
import pinecone
from PIL import Image
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Pinecone
from langchain.chat_models import ChatOpenAI
from langchain.prompts.chat import (
    ChatPromptTemplate,
    SystemMessagePromptTemplate,
    HumanMessagePromptTemplate,
)
from langchain.schema import (
    AIMessage,
    HumanMessage,
    SystemMessage
)
from langchain.chains import ChatVectorDBChain

# Initialize Pinecone
pinecone.init(api_key=os.environ["PINECONE_API_KEY"], environment=os.environ["PINECONE_ENV"])

# Load logo and display it
image = Image.open('utils/logo.jpg')
st.image(image, width=200)

# Set the title
st.title(f'{os.environ["EGPTNAME"]}')

Next, we’ll initialize the embeddings and vector store using OpenAI Embeddings and Pinecone. Also, we’ll create the chat prompt templates and initialize the ChatVectorDBChain. This will allow us to use the ChatOpenAI model with a temperature parameter to control the randomness of the generated responses.

# Initialize embeddings and vector store
embeddings = OpenAIEmbeddings()
index = pinecone.Index(os.environ["INDEX_NAME"])
vectorstore = Pinecone(index, embeddings.embed_query, text_key='text', namespace=os.environ["NAMESPACE"])

# Define the system message template
system_template = """Use the following pieces of context to answer the users question. 
If you cannot find the answer from the pieces of context, just say that you don't know, don't try to make up an answer.
----------------
{context}"""

# Create the chat prompt templates
messages = [
    SystemMessagePromptTemplate.from_template(system_template),
    HumanMessagePromptTemplate.from_template("{question}")
]
prompt = ChatPromptTemplate.from_messages(messages)

# Initialize the ChatVectorDBChain
qa = ChatVectorDBChain.from_llm(ChatOpenAI(temperature=0), vectorstore, qa_prompt=prompt, return_source_documents=True)

Next, we will initialize the chat history and define a function to execute a query. The function will display the generated response from GPT-3.5 and the sources used for the latest answer.

# Initialize the chat history
chat_history = []

if 'chat_history' not in st.session_state:
    st.session_state['chat_history'] = []

# Define the function to execute a query
def execute_query(query):
    with st.spinner('Thinking...'):
        for i in range(len(st.session_state['chat_history'])-1, -1, -1):
            chat_history.append(st.session_state['chat_history'][i])

        result = qa({"question": query, "chat_history": chat_history})

        st.session_state.chat_history.append((query, result["answer"]))
        chat_history.append((query, result["answer"]))

    st.info(query)
    st.success(result['answer'])

    # Display the sources for the latest answer
    with st.expander("Sources for the latest answer"):
        sources = result['source_documents']
        for idx, i in enumerate(sources):
            st.markdown(f"**Source number {idx + 1}** \n")
            st.markdown(i)
            st.write('-'*10)

    # Display the previous chat history
    if len(chat_history) > 1:
        for query, answer in chat_history[:-1]:
            st.info(query)
            st.success(answer)

# Create input field and sample question buttons
query = st.text_input("Ask a question or tell what to do:", key="input")

# Execute the selected sample query
if query:
    execute_query(query)

That completes the main portion of the code.

I created a separate page to handle uploading data to Pinecone. I believe the most useful features would be the ability to upload PDFs and provide URLs. Given a URL the app will automatically scrape that page and index the text into the database.

The code for this resides in the pages/Contribute_data.py script. I encourage you to clone the repo and go through it. But here’s a snippet of how URLs are handled.

with st.expander("URL"):
    url_input = st.text_input('URL', '')

    if st.button("Read from URL"):

        with st.spinner('Wait for it...'):
            urls = [
                url_input
            ]
            loader = UnstructuredURLLoader(urls=urls)

            documents = loader.load()

            text_splitter = CharacterTextSplitter(chunk_size=2000, chunk_overlap=0)
            documents = text_splitter.split_documents(documents)

            embeddings = OpenAIEmbeddings()

            isNotDone = True
            while(isNotDone):
                try:
                    reinitate_connetion()

                    vectorstore = Pinecone.from_documents(documents, embeddings, text_key='text', index_name=INDEX_NAME, namespace=NAMESPACE)
                    isNotDone = False
                except:
                    pass

        st.info('Done')

That’s pretty much all of the main components you need. Again, the full code can be found at https://github.com/miranthajayatilake/nanoQA2

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

🌟 Please feel free to fork the repository, star it, and contribute to the project. I am looking forward to hearing your thoughts, suggestions, and any success stories that result from using this code. 🚀

I’ve enjoyed writing and sharing this. Thanks for reading!

First things first

Addressing the elephant in the room: no, we will not build a version of ChatGPT here, nor will we follow anything related to how it was trained.

But we will use Large Language Models (LLMs), which is the family of models that ChatGPT is based on, still not in a generative way but to learn representations of text. Ultimately this guide aims to build a simple question-answering interface, that answers only using the prior information you have provided.

This can also be viewed as a basic semantic search engine with a chat interface.

Why do this? What are the use cases?

While an all-embracing, generalized, know-it-all answering machine is impressive, in reality, there is more value in an expert agent that allows more control over the generated outputs and a lesser probability to make mistakes.

Such an agent can help us retrieve meaningful information from large corpora with the ease of a natural conversation, which is otherwise quite cumbersome to handle. The use cases from the top of my head are;

• get answers from a software documentation

• get answers on legal matters from a large collection of legal documents

• get healthcare information but only from a collection of trusted sources

• …

You get the idea. Let’s dive in.

Demo

This is the goal.

Core Concept

The way we are utilizing LLMs in this app is to generate embeddings of text. An embedding is a numerical representation of a text segment that can be either several words, a single word, or even a single character. This is something to evaluate and decide depending on your application.

In this basic version of the demo app, we perform sentence embeddings where a sentence is a question that is asked by the user.

Also in advance, we generate embeddings for the questions that we have in our data store. So upon a user query, how an answer is chosen is by looking at the similarity between the ‘user question embedding’ and a ‘stored question embedding’.

A bit about this approach

One of the most discussed drawbacks of ML in general is the difficulty to control the outputs from a model. But in the above approach, an LLM is only used to understand language and to select which answer is the most relevant. Since the answers are predetermined, we have more control over the ultimate output to the user in this setup.

Of course, this approach will not match all use cases. But in applications where a ‘generative’ output is too risky, I argue that this is a good workaround.

System Overview

The main components of the system and the utilized tools are as follows;

• LLM (Retriever) — The LLM used in this version is a sentence-transformer from HuggingFace. This is the most crucial component of this app and I use a tool called haystack to build it.
  Haystack makes it easier to work with transformer models and it has pre-built functionality for question-answering capabilities, so we don’t have to build everything from scratch.

• Data store — I use Elasticsearch
  Elasticsearch is a database that stores documents in an unstructured manner and is optimized for search functionalities. I chose this with scaling in mind.

• User interface — Streamlit
  Streamlit is a python tool that lets you build quick web applications. Highly recommended.

The steps

Download the nanoQA GitHub repo

git clone https://github.com/miranthajayatilake/nanoQA.git
cd nanoQA

Next, we have to set up the python environment. The best thing is to create a new virtual environment for this project with python3. I personally use Anaconda to manage python environments. Once you have it, install the required dependencies.

pip install -r requirements.txt

The data store setup

Let’s set up elasticsearch locally to be used as the data store in this project. I prefer using Docker for this because it is easy to use. Make sure you have Docker running on your machine.

Run the bash script that will pull the elasticsearch image and run it on the relevant ports.

bash datastore.sh

Sample data and indexing

The next step is to obtain your data source, restructure it and store it in a meaningful way so that it fits a question-and-answer format. Let’s dive into what this means.

In the demo app, we use a sample dataset (a CSV file) that has some FAQ data on COVID-19. Since this already has a structured set of questions and answers along with some more meta-data, there is no need of restructuring it. The script sample_data.py takes care of downloading and indexing data. The arguments are: sample_data.py path_to_save_data download_location elasticsearch_index_name

python sample_data.py data/faq_covid https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/small_faq_covid.csv.zip index_qa

The index name can be replaced with anything you desire. Also, please go through the script to see how the question embedding is performed and stored along with the data. I hope to make this process more customizable in future releases of the repo.

The UI

Now that we have all the backend components set up, what’s left is to spin up the sample user interface. You can easily do this with the below command.

streamlit run app.py

The script responsible for the UI isapp.py . The main components are all built on top of haystack . And the LLM we are using in this demo is calledsentence-transformers/all-MiniLM-L6-v2 . This is pulled from HuggingFace — model page.

More control over outputs

I perform a custom filtration of the results to make it more fun. One of the outputs from the similarity calculation step is a similarity score between 0 and 1 (1 meaning identical).

I set a minimum threshold of 0.7 for an answer to be eligible. If the maximum similarity score among the outputs is less than 0.7, I overwrite the output to the user saying that nanoQA could not find an answer. This prevents the app from giving irrelevant answers.

if prediction['answers'][0].score >= 0.7:
   ...print answer...
else:
   st.write('nanoQA failed to find an answer with confidence...sorry!')

Providing evidence for the answer

Another addition to the demo app is printing the source of the answer and the link. This is possible because, in our sample data, we had metadata on these sources and links.

I believe this is a powerful feature for an application such as this in terms of user experience.

That’s it for this one.

I hope you gained useful knowledge from this project. I plan to keep improving this application and write successive posts on more exciting features. So make sure to follow my blog and the nanoQA GitHub repo 🔥.

I also started a newsletter where I interview founders and AI engineers who are building impactful AI applications. Check it out 🫶

Paradigm | Mirantha Jayathilaka, PhD | Substack
Stories of companies and individuals building impactful AI applications and sometimes even complete businesses with AI…paradigmai.substack.com

We also have interesting chats over at our Discord. Join in 🤝🏽

Join the ParadigmAI Discord Server!
Check out the ParadigmAI community on Discord discord.gg

Thanks for reading!

Undoubtedly Natural Language Processing (NLP) has come a long way over the recent years with the advancements in the area of language modelling and ever-increasing computational efforts put in. This has enabled many capabilities and tasks related to text processing, leading to several high-impact applications. This is a comprehensive list of different tasks and applications possible with current NLP techniques. Here we go…

1. Information retrieval

Finds documents of text that satisfies an information need from within large collections

2. Named entity recognition

Seeks to locate and classify entities into predefined categories such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.

3. Relation extraction

Extracts semantic relationships from the text, which usually occur between two or more entities

4. Text classification/Document Classification

Assigns a text/document to one or more classes or categories.

5. Document Ranking

6. Annotation

7. Topic modelling

Discovers the abstract “topics” that occur in a collection of documents

8. Keyword Extraction

9. Machine translation

10. Parts of speech tagging

Process of marking up a word in a text as corresponding to a particular part of speech

11. Semantic Role Labeling

Indicates the semantic role in the sentence, such as that of an agent, goal, or result

12. Word Sense Disambiguation

Identifies which sense of a word is used in a sentence

13. Grammatical Error Correction

14. Semantic textual similarity

determines how similar two pieces of texts are. This can take the form of assigning a score from 1 to 5. Related tasks are paraphrase or duplicate identification.

15. Text summarization/Meeting Summarization

16. Reading comprehension

17. Question and answering

18. Question Generation

19. Image captioning

20. Fake News Detection/Hate Speech Detection

21. Text generation

22. Sentiment/emotion analysis

Interprets and classifies of emotions (positive, negative and neutral) with text data

23. Speech-to-text

Translation of spoken language into text

24. Text-to-speech

Converts text into spoken voice output

25. Dialogue Understanding

I hope this list is useful to whoever wanting to dive deeper into the NLP realm, either for research or industry application ideas. Good, that was a quick one.

Thanks 🙏.

In this post, we consider the importance of user research in AI projects and describe our approach at Mindwave.

AI is the new electricity

This is a phrase often heard from the pioneers of the technology. It is meant to transform industries from healthcare to manufacturing, from logistics to retail. But there is a clear lesson that we can draw from our experience with electricity over the past years — it is an enabler. Not all products or solutions that use electricity changed the way we live today. But some of them really did.

The impact of AI is fairly evident from the success stories that have emerged so far. Take smart assistants for example, devices such as Amazon Echo and Google Home. Manually coding a software system that could interpret the many variations of human conversation effectively seemed like a far fetched goal a few years ago. But recent AI techniques in natural language processing (NLP) have enabled this capability for software and the applications have soon followed into our lives. Smart assistants today have transformed the way we search on the web, listen to music and even how we buy.

But it is important to see what drove the adoption of this new technology. Is it the intrinsic affordance of the technology alone or something else? We argue that the main reason for the huge popularity of smart assistants lies in how they changed the fundamental way we communicated with the machines. Verbal dialogue is the most natural method with which we communicate with each other, hence it becomes the best method in the case of human-machine interface as well. Thus we make a clear distinction — product success was due to solving a major challenge in human-machine interaction, and AI was just the enabler.

Problem Identification — First step in AI development workflow

Problem identification should become the initial step in any AI development workflow. We like to highlight two separate avenues where problems can occur with respect to a given end-product.

I. Problems between user and the product

Product adoption is often driven by the user experience and the many aspects that revolve around it. Hence it is an area that requires proper user research and feedback generation.

Take an example use case where the idea is to develop a virtual psychiatrist to treat patients with mental and emotional challenges. Here the objective is to provide the most suitable advice on the basis of the input from the patient regarding his/her mental and demographic status. Looking at a scenario of psychiatrist-patient interaction, the effectiveness of diagnosis and treatment depends heavily on the expressivity of the patient in describing what he/she feels. Here a challenge is identified — how does one facilitate this nuanced interaction via a software interface? — this should become an initial concern for the development workflow of this product.

II. Problems with inefficiencies and improvements in an existing product

Often opportunities for automation addressing inefficiencies and challenges with regard to innovation could surface in existing products.

Consider the use case of a movie recommendation system. One means of implementation is to carry out a traditional execution which produces a new set of recommendations given the user’s past choices. In this case, the instructions are hardcoded by the designers of the system. But a more advanced and effective strategy would be to understand how the users across the system have reacted to the recommendations in the past, learn from the historical data and improve the recommendations with respect to user clusters automatically. This is the challenge identification which leads us to the next point.

Assess if AI is the enabler for the solution

Following the identification of the problem and potential solutions, it is vital to assess the possible methods of developing a solution and work out if AI would act as the enabler or not.

Connecting with the virtual psychiatrist use case from above — a method of improving user expressivity in the virtual psychiatrist system might be to allow the user to input free text into the system, explaining his/her issue. Now in order to process and retrieve elements of the text which are important to the diagnosis, NLP techniques become a requirement. Here we choose AI to be the enabler.

Let us consider another use case of a virtual pre-diagnosis for a condition concerning physical health. In this instance a specific condition has to be determined by discrete inputs from the patient, such as age, symptoms, severity level etc. Here, a more traditional programming approach of producing the results would fit better. With respect to the accuracy and transparency of the process, software that is programmed explicitly to make decisions comes above the existing AI approaches in this case.

Final thoughts

Jeff Catlin makes a firm point in his Forbes article about identifying the need and the desired outcome before thinking of the technology aspect of an AI project. He argues that building a business case for AI isn’t so different from building one for any other business problem. Accurate study on the feasibility and return on investment will drive AI adoption and make the best use of the technology. Robert Pearl goes on to mention how AI could live up to its hype in the healthcare industry, where he emphasises how tech companies often focus squarely on the technology while routinely overlooking the human fears and frustrations that AI can cause. Addressing these needs such as ethicality and security can surface new opportunities and areas of growth for AI in the much-anticipated field of healthcare

At Mindwave, we follow the above approach towards AI developments in the field of healthcare, carefully identifying the problems through deep user research and feedback. We assess the feasibility of using AI in solving the identified problems and focus on strong evaluation strategies to measure their impact.

We argue that the use of AI for the technology’s sake, without a proper understanding of how it can solve real problems and impact our lives, can lead to failed investments of time and money. It can damage the trust towards the technology and eliminate potential long-term outcomes that can be favourable to everyone.

If you have any feedback or would like to chat about any of the above points, please drop an email to mirantha@mindwaveventures.com

Healthcare is often spoken of as a field that is on the verge of an AI revolution. Big names in AI such as Google DeepMind, publicise their efforts in healthcare, claiming that “AI is poised to transform medicine.”

But how impactful has AI been so far? Have we really identified areas in healthcare that will benefit from new technology?

At the ACM CHI Conference on ‘Human Factors in Computing Systems’ held in May this year, Carrie J. Cai of Google presented her award-winning work on ‘Human-centered tool for coping with Imperfect Algorithms During Medical Decision-Making, discussing the increasing usage of machine learning algorithms in medical decision making. Her work proposes a novel system enabling doctors to refine and modify the search of pathological images on-the-fly, continuously enhancing the usability of the system.

Retrieving visually similar medical images from past patients (e.g. tissue from biopsies) for reference when making medical decisions with new patients is a promising avenue where the state-of-the-art deep learning visual models can be highly applicable. However, capturing the exact notion of similarity required by the user during a specific diagnostic procedure offers big challenges to existing systems because of a phenomenon known as the intention gap, which refers to the difficulty in capturing the exact intention of the user. We will discuss this in more detail later.

Cai’s research showcases how the refinement tools they developed on their medical image retrieval system increases the diagnostic utility of images and most importantly, increases a user’s trust in the machine learning algorithm for medical decision making. Furthermore, the findings show how the users are able to understand the strengths and weaknesses of the underlying algorithm and disambiguate its errors from their own. Overall the work presented an optimistic view of the future of human-AI collaborative systems in expert decision-making in healthcare.

In this post, we want to look into three main areas namely — (1) the state of content-based image retrieval systems, (2) the role of deep learning in these systems and finally, (3) a discussion on their application and impact in healthcare.

The state of content-based image retrieval systems

Over the last two decades or so, content-based image retrieval (CBIR) has been a vivid research area in computer vision, mainly due to the ever-growing accessibility of visual data on the web. Text-based search techniques for images suffer many inconsistencies due to mismatches with the visual content, hence considering the visual content as a ranking clue for similarity is seen to be important in many cases.

Wengang Zhou et al. point out two crucial challenges in CBIR systems that they call as the intention gap and the semantic gap.

The intention gap, as implied by the meaning, refers to the difficulty in capturing the exact intention of the user by a query at hand, such as an example image or a keyword. This is the challenge addressed by Carrie J. Cai et al. with their refinement tools in the user interface. Looking at past research, query formation by example image seems to be the most widely explored area, intuitively due to the convenience of obtaining rich query information through images. This demands accurate feature extraction from images which brings us to the next point, the semantic gap.

The semantic gap deals with the difficulty in describing high-level semantic concepts with low-level visual features. Now, this topic has attracted a considerable amount of research over the years with several notable breakthroughs such as the introduction of invariant local visual feature SIFT and introduction of Bag-of-Visual-Words (BoW) model.

Figure 1 shows the two main functionalities of a CBIR system. Matching the similarities between the query understanding and image features can also be an important step, but it fully depends on how well the system expresses the query and the image.

The recent explosion of learning-based feature extractors such as deep convolutional neural networks (CNN) opened up many avenues for research that can be directly applied to deal with the semantic gap we discussed in CBIR systems. These techniques have shown significant improvements over the hand-crafted feature extractors and have already demonstrated potential in semantic-aware retrieval applications.

The role of deep learning

The underlying details of the CBIR system analysed by Carrie J. Cai et al. are presented in detail by Narayan Hedge et al. in their study “Similar Image Search for Histopathology: SMILY”. The overview of the system is shown in Figure 2.

A convolutional neural network (CNN) algorithm is used for the embedding computation module shown in Figure 2, which act as the feature extractor in the system. The network condenses image information into a numerical feature vector, also known as an embedding vector. A database of images (in this case patches of pathology image slides) along with their numerical vectors were computed and stored using the pre-trained CNN algorithm. When a query image was selected for searching, the embedding of the query image is computed using the same CNN algorithm and compared with the vectors in the database to retrieve the most similar images.

Further, Narayan Hedge et al. explains that the CNN architecture is based on a deep ranking network presented by Jiang Wang et al., that consists of convolutional and pooling layers together with concatenation operations. During the training stage of the network, sets of 3 images were fed: a reference image of a certain class, a second image of the same class and the third image of a totally different class. The loss function was modelled such that the network assigns a lower distance between the embedding of the images coming from the same class than the embedding of the image of the different class. Thus the image from the different class helps strengthen the similarity between the embeddings of images from the same class.

The network was trained using a large dataset of natural images (e.g. dogs, cats, trees etc) rather than pathology images. Having learnt to distinguish similar natural images from dissimilar ones, the same trained architecture was directly applied for feature extraction of pathology images. This can be seen as a strength of neural networks in applications with limited data, commonly termed as transfer learning.

The CNN feature extractor computed 128-sized vectors for each image and L2 distance was chosen to be the comparison function between vectors. Narayan Hedge et al. visualised all the embeddings produced from the dataset of pathology image slides using t-SNE as shown in Figure 3. (a) shows embeddings coloured by organ site and (b) shows embeddings coloured by histologic features.

In fact, similar architectures and training techniques to deep ranking networks can be widely seen in deep learning literature such as Siamese Neural Networks and have been even applied for face detection applications. 
Now, coming back to CBIR systems, we see that deep learning could help in reducing the semantic gap (discussed above) as these learning-based approaches are proven to be impressive in identifying important features even in noisy natural images.

Application and impact in healthcare

So far we looked at what goes into CBIR systems and the potential of deep learning in overcoming the crucial challenge of the semantic gap. But how applicable is CBIR in healthcare? And can we clearly quantify the impact?

Henning Müller et al. states that the radiology department of the University Hospital of Geneva alone produced more than 12,000 images a day in 2002 with cardiology being the second-largest producer of digital images. The study further argues that the goal of medical information systems should be to “deliver the needed information at the right time, the right place to the right persons in order to improve the quality and efficiency of care processes.” Thus in clinical decision making, support techniques such as case-based reasoning or evidence-based medicine are desired to benefit from CBIR systems.

No matter how sound the technology is, integration of these systems in real clinical practices demands significantly more work, especially in building trust between the system and its users. This is where the study of Carrie J. Cai et al. stands strong by being very flexible with user’s relevance feedback, which provides the ability to the user to rate the returned results of the system. Henning Müller et al. also talks about the importance of relevance feedback in an interactive setting for improving the system results and also to increase the adaptability of the CBIR systems.

Another major point is quantifying the impact of these systems which is crucial for the adaptation and advancement of this field of research. After a user study with 12 pathologists, Carrie J. Cai et al. claims that with their CBIR system users were able to increase the diagnostic utility of the system with less effort. Also, results show increased trust, enhanced mental support for the users and improved likelihood to use the system in real clinical practices in the future. But diagnostic accuracy (although empirically stated to remain the same) was not evaluated in this study as it was out of scope.

Looking forward, it is evident that continuous collaboration of medical experts and AI system developers is required in both identifying the use cases and evaluating the impact of AI applications in healthcare. Additionally, the research community should focus on the development of open test datasets and query standards in order to set benchmarks for CBIR applications. This would be immensely helpful to drive the research forward with clear ideas of the contributions.

This article was originally published in The Gradient. Hope you enjoyed reading.

Special thanks to Hugh Zhang, Max Smith and Nancy Xu for their insight and comments.

Life

People don’t want to start over. You have to willing to be a fool and kind of have that beginner’s mind and go back to the beginning to start over. If you’re not doing that you’re just getting older.

Social Media

(In social media), rather than you looking at yourself, you’re looking at how other people look at you.

It’s kind of a disease. Because social media is making celebrities of all of us. And celebrities are the most miserable people in the world. They have this strong self-image that gets built up by compliments, and it doesn’t take many insults to cancel out a lot of compliments. And then you’re carrying around this big weighty image. It is very easy to be attacked.

Success

You want to be rich and anonymous, not poor and famous.

You don’t want to be the guy who succeeds in life while being highly stressed, unhappy and leaving a trail of emotional wreckage with you and your loved ones. You want to get there calmly, quietly and without struggle.

Desire

Desire is a contract you make with yourself to be unhappy until you get what you want. Don’t have too many.

Pick your one overwhelming desire. It’s okay to suffer over that one. But on all the others, you want to let them go, so you can be calm, peaceful and relaxed.

Business

If you want to be effective in business, you need a clear, calm, cool and collective mind.

We live in an age of infinite leverage. Your actions can be multiplied by a thousand fold. Because of that, the impacts of good decision making is much higher than it used to be.

A clear mind leads to better judgement, leads to better outcomes. So if you want to operate at peak performance, you have to learn how to tame your mind, just like you have to learn how to tame your body.

Work

We like to view the world as linear. Which is saying, I’m gonna put in 8 hours of work, I’m gonna get back 8 hours of output. It doesn't work that way.

Outputs are non-linear based on the quality of work that you put in. The right way to work is like a lion. You and I are not like cows, we are not meant to graze all day. We are meant to hunt like lions. Machines are meant to work 9–5, not humans.

You are not gonna get rich renting out your time. You need to have equity to gain your financial freedom.

The idea that we are all factory like cogs in a machine, who are specialised and have to do things by memorisation or instruction, is gonna go away and we are gonna go back to being small groups of creative bands of individuals setting out to do missions. And when those missions are done, we are gonna collect our money, we get rated, and then we rest and reassess until we are ready for the next sprint.

Automation

Automation has been happening since the dawn of time. When electricity came along, that put a lot of people out of work. What it does is, it frees people up for new creative work. So the question is not ‘ is automation is gonna eliminate jobs?’ There is no finite number of jobs. Obviously new jobs are being created. They’re usually more creative jobs. So the question is ‘ how quickly is this transition going to happen? And what kind of jobs will be eliminated and what kind of jobs will be created? It’s impossible looking forward to predicting what kinds of jobs will be created.

Capitalise

The correct criticism of capitalism is when it does not provide equal opportunity. But people confuse this with equal outcome. When you have equal outcome, that can only be enforced by violence. Because free people make different choices, and when they make different choices, they have different outcomes.

Socialism comes from the heart. Capitalism comes from the head.

Technology

Technology empowers the individual, the individual means that you have the break down of family structure and religion and it does mean that there is a leftward shift to it. Technology leads the world left.

Politics

If all your believes line up into one political party, if all your believes are the same as your neighbours and friends, you are not a clear thinker. Your beliefs are socialised, they are taken from other people. So if you want to be a clear thinker you cannot pay attention to politics, it will destroy your ability to think.

Modern society

For most of modern life, all of our diseases are diseases of abundance. Not diseases of scarcity.

We are over-exposed to everything. So the way to survive in modern society is to be an ascetic. It is to retreat from society, there is too much society everywhere you go. Everyone is trying to program everybody. The only solution is to turn it off.

All of man’s problems arise because he cannot sit by himself in a room for 30 minutes alone.

Peace and Happiness

Peace is happiness at rest. Happiness is peace in motion. You can convert peace to happiness anytime you want. But peace is what you want most of the time.

The way we think you get peace is by resolving all your external problems. But there are unlimited external problems. So the only way to actually to get peace is on the inside by giving up this idea of problems.

It’s easier to change yourself than to change the world.

Meaning of life

The question (of the meaning of life) is more interesting than the answer.

If there was a single answer we will not be free. We would be trapped. Because then we would all have to live to that answer.

The fundamental delusion is thinking that there is something out there that will make you happy and feel free forever — there is. Its called death.

Retirement

Retirement is when you stop sacrificing today for some imaginary tomorrow.

Way to get out of the competition trap is by being authentic. Way to retire is to actually find the thing that you know how to do better than anybody else. And you know how to do better than anybody else because you love to do it. So nobody can compete with you.

Then figure out how to map that to what society actually wants. Apply some leverage, put your name on it so you take the risks but you gain the rewards, have ownership and equity in what you do and just crank it up.

Reality and life

Your real resume is a catalogue of all your suffering. You have to do hard things anyway to create meaning in life. Making money is a fine one.

Life is really a single player game. It’s all going on your head. Whatever you think you believe would very much shape your reality.

The world just reflects your own feelings back at you. Reality is neutral. Reality has no judgements. It’s your chiose how you see your senses.

Every man has two lives. And the second starts when he realises he has just one. — Confucius

If you have read this post so far, I hope that you find these ideas inspiring as much as I did. And as always, it was a pleasure listening to Naval Ravikant.

Thanks for reading!

Recall the last data science talk that inspired you? The one you thought that had an impressive impact. Was it only the mathematical technicalities or the state-of-the-art accuracies that caught your attention?

Surely not. Effective presentation techniques play a major role in taking an idea to a wider audience. And having the skills to deliver a memorable presentation is vital to any researcher.

Great presentations help you to build a brand for your research and yourself, which will guide you immensely in your academic or professional career prospects.

This post guides you through some of the key points that would make a data science research presentation more effective. I start by discussing five generic ideas and dive a bit deeper into one vital point, which is the storyline of your presentation.

1. Good structure. A good presentation takes the audience on a journey step by step. And at each step as you progress, your ideas should reveal an ultimate goal that is clear to the listener. A common structure always starts with an introduction about yourself and your research problem. It is important to set a clear scene initially so you can build your story from there. The storyline, which is in fact the most important feature of a good presentation will be discussed in detail later in the post.

2. Professional slides. Slides reinforce your story. You cannot underestimate the influence of quality slides on an audience. Here, it is important to keep in mind that your audience would tend to read everything on your slides while listening to you. So choose wisely what you want on your slides. Otherwise, it can distract your listeners and confuse them in the end. A lot of text on the slides is usually a bad idea. Effective graphics are proven to grab attention and help clear understanding. Keep your slides uncluttered as much as possible.

3. Technical contributions. The ultimate goal of any research is the contribution you make to an industry or knowledge in general. So you want to emphasise on the technical contributions you were able to deliver with your work. Again this is an important part of your storyline which is discussed further in this post.

4. Clear and entertaining delivery. No matter how good your material is, the way you deliver it would decide whether your presentation is going to be memorable or not. Clear and correct use of language is important together with good pronunciation and phase. Make sure that you are audible to everyone and do not forget to smile.

5. Confident speaker. Some might argue that confidence depends on each person’s character, but there are factors that you can use to improve your confidence levels walking into a presentation. Good preparation before the presentation is utmost important here. Make sure you practice delivering the talk at least several times before going on stage. Trust me, it won’t be a waste of time and it would enhance your confidence more than you think!

Storyline in key

Now let’s focus on one of the most crucial points when preparing and delivering your research presentation, the storyline. This decides the number of ears you will have throughout your presentation.

Below are the key areas you want to build around in order. Of course, some topics are interchangeable according to your preference, but this an effective outline if you are looking for one to guide your presentation preparation.

1. Setting the scene. This should be the first and foremost target. What is the problem you are addressing in your research? And why is it timely, relevant and interesting? Why does it matter? Make this setting very clear, and you have the attention of your audience grabbed. It should form the basis of what you are trying to convey. And the purpose of your efforts. When listeners can relate to your setting, it is hard to ignore you.

2. Focus on your hypothesis/solution for your setting. Soon following the setting, you can get your audience to focus on your hypothesis/solution for the problem you are addressing. This is vital because now they see instantly what you are proposing, and they are keen to know more. So then you can guide them to the next steps of revealing your methods and procedures.

3. Methodology/approach/work done. Now you explain your work towards achieving what you claimed in the previous step. Make sure to be concise. Nobody likes math formulas or deep technical details. Also avoid the urge the show-off your math or coding skills during this step. Simply present the steps you have taken on a higher level. If anyone is interesting to dig deep into the technicalities, they will follow up later. Flow charts and graphics are highly effective in this step. Paragraphs of text is a mistake.

4. Context/related work. Now some might argue that this point should come before the methodology. In some cases, it can be true. But my argument is that the attention span of the audience will be high at the beginning of the talk. And that is where you have to get your work in the spotlight. If you do a good job at it, you anyway have the audience when you go into this step. Since they are eager to know what you have done, presenting related work, in the beginning, might bore them up without cutting to the chase. Nevertheless, context is important and you have to make your approach stand out from the existing techniques and ideas.

5. Contributions made and results. Highlight your impact. Show your outcomes and results of your applications. Point out the contributions you have made to the knowledge and understanding of the field. This is where you strengthen your stand in the solution you have proposed.

6. Outlook/open questions/future work. Eventually, keep a note on the broader arena that your efforts showcase. Now one point to ponder upon here is that, do you mention any limitations of your research if any? My opinion is yes. This is the place to convey your weaknesses and this will come out as a positive impression on your work. Also, open up questions for future work with your particular approach and emphasise the potential impact.

I hope this post provides guidance to anyone looking to improve their presentation skills, especially in the highly technical field of data science. Personally, this is something I have spent considerable time on improving myself because I strongly believe in the impact of effective presentations for your research. Executed properly, these skills would help you boost your career as a researcher or data scientist.

I would like to hear your thoughts on this as well. Reach out to me on Twitter or comment below.

Thanks for reading.

In the path of building machines to understand the real world we live in, it becomes evident that the ability to just identify individual items or recognise a pattern out of the given set of samples is merely a small part of the puzzle.

For instance, take the common application of machine learning to train a model to identify classes of objects in a set of given images. We know that state-of-the-art models are able to identify the different objects to almost the level of humans in those images.

But do they really understand what it sees? Or can they reason about the objects they detect? Do they really understand? No.

Towards the goal of building intelligence, it is clear that the systems should have the ability to formulate far more complex models of the world from the individual items they detect/sense.

How do we know that this is what should happen? Well, because that’s what we do. Humans. The best evidence of intelligence that we know of in the world.

A model can be interpreted as a representation of a thing or a set of things. Narrowing down to our area of focus, a model can be seen as a graph.

Taking the same example of detecting objects in an image, imagine we detect an instance of a dog and a person in a given image. Now a trained model will easily detect the dog and the person individually. But if it was us, after seeing the dog and the person in the image we naturally go further to see that, oh, that must be the pet of that person. And normally people and dogs have a good relationship.

Now what we did there was, we made sense. We reasoned. This phenomenon of ‘making sense’ can be interpreted as an act of knowledge. Knowledge about the interactions between the detected object and past experience that we have seen these objects together before.

This infers that we need to integrate knowledge into our systems if we want them to build models of the world.

As we know, recent advancements in technology have gathered a huge magnitude of data about us and the world we live in. And the more data we collect, people have thought of ways to organise and manage this data, specifically in methods of graphs. It turns out that many points of data can be represented symbolically in ways defined by logic.

With the surge in the success of machine learning, integration of knowledge is clearly identified as a path to improvement. Hence more research interest has been gathered around the areas of using graphs in the pipelines of machine learning, that turns out to be quite promising.

But one challenge in this quest is that the machine learning approaches and the graph building approaches, don’t specifically speak the same language. In other terms, machine learning prevails in the numeric space, whereas graphs prevail in the symbolic space.

In the field of machine learning, several recent research attempts have taken strides towards trying to encode the graph structures in machine-readable forms.

Even though traditional methods have taken more statistical approaches towards this goal, it is seen that more and more approaches of automatically learning these embeddings have emerged more triumphant.

The following few posts aim to synthesise these concepts and present an overview of the different approaches of representation learning on graphs. The ideas will be discussed as a walk-through of the paper by Hamilton et al. called ‘Representation Learning on Graphs: Methods and Applications’.

The discussion is divided into two main sections, namely, Embedding of Nodes and Embedding of Subgraphs.

The notions throughout the explanations are as follows;

Input is an undirected graph G = (V, E), associated binary adjacency matrix A

Node attribute matrix X ∈ R m×|V|

Output vector z ∈ Rd , where d << |V|

The goal is to use A and X to map a node/subgraph to a vector z.

You can avoid the paywall and read the rest that follows this post using the links below :)

See the second post that follows this — Embedding of Nodes
And the third post in this series — Embedding of Subgraphs

Thanks for reading.

The algorithm is quite the workhorse in the majority of widely used, human-level surpassing learning systems based on neural networks. It was popularised by the 1986 paper published in Nature authored by David Rumelhart, Geoffrey Hinton, and Ronald Williams.

The original paper concludes, “applying the procedure to various tasks shows that interesting internal representations can be constructed by gradient descent in weight-space, and this suggests that it is worth looking for more biologically plausible ways of doing gradient descent in neural networks”. Well, backpropagation might not exactly be what’s happening in our natural neuron networks, but it surely presented great results in mathematical learning systems. And this would give birth to an exciting new era of Artificial Intelligence.

Backpropagation algorithm deals with a systematic method of continuously adjusting the internal parameters (weights and biases) of a neural network so that the error in the predictions made by the network will be minimum. Understanding of the inside workings of it seems vital if you are looking to work with complex applications of machine learning and deep learning.

The most clever thing about backpropagation seems to be the method used to calculate the partial derivatives of the cost function with respect to each weight and bias in the network. This paves the way to ponder even how this elegant algorithm was found for the first time.

But if you carefully look at the behaviour of the neural network, there might be a systematic way of deducing the mystery of the derivatives.

Imagine the case of changing a single weight w value by a small factor Δw as shown below;

Now this change will affect the immediate activation involving that weight, changing it by Δa.

The change of Δa will in-turn affect all the other nodes in the following layer.

And ultimately, passing through all the layers in a similar manner, Δw that started this entire change affects the final cost function.

Now, this we can think of it as a forward propagation of change. In other terms, we can represent the final change in the cost of the network in terms of the first change in weights we manipulated.

Now calculating this change in cost looks like a task of figuring out this term 𝜕C/𝜕w. And looking at how it has been propagated over the network, we can formulate it with respect to all the changes that took place after the first Δw.

The above expression shows the change in the most immediate activation due to Δw. And this will, in turn, affect the next activations following Δa.

Now here Δa can be replaced by the previous expression and you will get the expression below;

We tend to see a pattern arising here. Starting from the first change in Δw, the following activations are affected by the change of activations of the previous layer. ∂a/∂a begins to be a common term to every node we pass through. Now it's clear that for every node in every layer the pattern will prevail as below;

The m, n, p… terms here denote the different layers and the ∑ denotes that we sum over all the node in each layer.

As as we recall the backpropagation algorithm is tasked with finding the partial derivatives of all the parameters, i.e. all the weights and biases with respect to the change in cost at the end of the network. So if we simplify the above equation, it suggests that we can formulate expressions for all the weights and biases in terms of the values of activations and weights. And this procedure of calculating the magnitude of change in every parameter, giving us a direction to follow if we want to reduce the entire magnitude of the cost in the network. The derivatives calculated gives the direction for the gradient descent. Pretty intuitive and elegant.

Now, is this how David Rumelhart et. al. came up with the solution for backpropagation? Well, maybe. But the most fascinating thing about this algorithm is that how these procedures of consistently adapting to the change in cost eventually result in a parameter space that emulates the representation of input data so that it could predict/identify new instances of the same distribution.

This fact is often seen as the mystery of backpropagation, that had made critiques to call deep neural networks a black box. But are they really?

My argument is that if we systematically assess and take time to actually think about what happens during an optimisation period of a neural network, maybe we might be able to wrap our heads around it. But the many levels and levels of abstraction seems to be too much sometimes for our short term memory to withhold and formulate an overall picture of the entire process. And this is what machines are helping us to do.

My closing thoughts are, is this the best that we could come up with though? Because if you think about it, the whole process of generalisation through backpropagation is a very data-hungry and time-consuming process. Will there be more optimum ways for machines to obtain a representation of the objects of the real world? Well, these questions are right at the frontier of research. And we have never been more excited to work on them.

The ideas presented in the post are directly influenced by the book “Neural Networks and Deep Learning” by Michael A. Nielsen. Highly recommended for anyone who wants to learn deep learning from scratch.

Thanks for reading.

I woke up this morning and thought “hey, it’s been a while since I wrote a blog post, hasn’t it?”. For the past few months I have been getting great feedback on the articles I had written on this blog, especially on the topics of Machine Learning/Deep learning and I highly value all your thoughts even though I couldn’t get back to some of you personally. I thoroughly enjoyed the process though!

As mentioned in my bio, I started writing this blog purely because I wanted to share what I was learning in the world of Tech and I was quite inspired when I saw others who did the same. And believe me when I say, you cannot underestimate the value of ‘SHARING’, especially in this exciting movement we call Tech. I encourage you to try it!

All in all I want to share a brand new step I’m taking along with my good friend and AI startup founder, CD Athuraliya, towards scaling this ‘sharing-what-we-learn’ 10 times bigger. Yes, I used the word ‘share’ too many times so far but I don’t care. I’m quite excited about this.

So Podcasts. I listen to great podcasts everyday. They have enabled me to gain valuable knowledge and ideas while on my daily commutes and leisure hours. Naming a few podcasts I love at the moment — Master of Scale with Reid Hoffman (my first introduction to podcasts), Techmeme Ride Home with Brian McCullough (best everyday Tech updates), Pivot with Kara Swisher and Scott Galloway and Converge with Casey Newton.

There’s clearly so much space to deliver great value via the stream of audio, so we decided to give it a try and launch a podcast ourselves! Neither CD nor I have any prior experience in audio recording, but we thought that would make it more interesting to learn it on the go. Wish us luck!

So the podcast is called TECH SAVVY. It is available now across 10 different platforms with the help of Anchor. — Anchor is a great APP for anyone who wants to get into podcasts. Highly recommended!

What’s the goal?

We don’t want to just report on Tech. In fact we want to take a deeper and a different look at what’s happening inside Tech. Each episode we would take one or more facts, either prominent or timely, and form it into a discussion, where will be talking about our own opinions and ideas related to it.

One of the most important distinctions we are trying to make is that (as mentioned in our very first episode as well), we will be focusing more on the creation of Tech, not just the consumption of Tech.

So considering this I would suggest that the podcast is more targeted towards listeners who want to dig deeper, even upto a conceptual level of how Tech works. But we promise not to geek it out too much also at the same time. We’ll try…

Also you could expect a bit of a ‘startup’ approach in our talks as well given that both CD and I have been working with tech startups for several years now.

All things considered, TECH SAVVY will be a place where we share almost all of our interests and motives related to Tech. From time to time we will try to invite some interesting personalities as well to talk in the podcast, so it won’t just be a our squeaky voices rambling every time.

And of course this doesn’t mean that I will stop writing on this blog. In fact I would be having more things to share along with this experience, which I’m totally psyched on!

So make sure you subscribe to TECH SAVVY with CD and Mirantha, available wherever you listen to your podcasts on!

And also we still have a lot to learn and improve, so make sure to drop in your constructive criticism and feedback if you listen to us. You can use the hashtag #techsavvypodcast anywhere on social media to reach out to us and also when sharing the content.

Thanks for reading and hope you enjoy the pod!

Identification of human body poses with mere 2D imagery had been a grand challenge in the field machine vision and with the explosion of Deep Learning algorithms, the area has seen quite promising advancements recently. As shown in Figure 1, the outputs from these models are able to accurately identify human body keypoints and form skeleton structures depicting the poses. Advancements of these techniques imply great potential with applications in areas such as sports, security surveillance, patient monitoring etc.

The process of human pose estimation can be divided into several parts:

1. Identifying anatomical keypoints of the human body

The first task is to identify the different parts of a body, preferably joints, so that these points can be tracked in the imagery. The model has to be trained initially using annotated data to be good at this task. Techniques such as heat maps are being widely used at this step and the accuracy directly impacts the nest step

2. Joining the keypoints to form the skeleton structure

Next the identified keypoints have to be joined with each other to form a skeleton structure. This task can be quite tricky as to decide which keypoints should be connected to which. Techniques such as assignment of a confidence score in-between keypoints are used here.

3. Forming a 3D representation of the skeleton structure

So far the process has being able to extract a 2D skeleton structure from the imagery which depicts the pose accurately, but can we derive a 3D model out of this? This can be a very powerful tool to have and as shown in Figure 1 (right), the 3D estimations are somewhat accurate, but there’s surely room for improvements.

4. Executing the above three point for multiple humans in a single image

Identifying the pose of a single human is impressive. Then what about doing that for multiple people in the same image? This question offers a brand new set of challenges in the steps of detecting humans and joining body keypoints.

On top of that, how about executing all these steps in a Real-time implementation? This is where the real deal lies. The research referred in this post tackles this frontier and you can find the paper here-> (https://arxiv.org/abs/1611.08050)

The code found here on my GitHub is based on the tensorflow implementation of the algorithm of the paper open sourced by the authors. (Find the original repo here)

This is quite an interesting frontier in Deep Learning towork in and there are many possibilities to explore!

The hype around Generative Adversarial Networks (GANs) had been growing ever since the architecture was first introduced by Ian Goodfellow and the numerous advancements and applications become more and more fascinating by the day. But for anyone who wants to get started with GANs, it can be quite tricky to figure out where to start. ‘Don’t Panic.’ This post shall guide you.

As with many things, the best way to fully understand a concept is to hit the roots. Grab hold of the first principles. For GANs, this is the original paper here ->(https://arxiv.org/abs/1406.2661). Now to understand a paper of this sort there can be two approaches, theoretical and practical. I often prefer the latter, but in case you love to dig deep into the math, here is a great post by my buddy Sameera breaking down the entire algorithm theoretically. In the meantime this post will present a simple implementation of the algorithm in its purest form using Keras. Let’s get started.

In the underlying setting of a GAN are two models namely, the generator and the discriminator, where the generator is constantly competing with the discriminator, which is an adversary that is learning to distinguish between the model distribution (e.g. generated fake images) and the data distribution (e.g. real images). This concept is famously visualized by a counterfeiter vs policeman scenario, where the generative model is thought of as a counterfeiter generating fake cash and the discriminator model as a policeman trying to detect the fake cash. The idea is that with constant competition between each other, both the counterfeiter and the policeman improve in each other’s role but ultimately the counterfeiter achieves a stage of producing fake cash that is indistinguishable from the real ones. Simple. Now let’s put this into code.

The example script provided with this post is used to generate fake images of faces. The ultimate result we are trying to achieve with the algorithm is represented in Figure 1.

Building the Generator model

So the generator model is supposed to take in some noise and output a desirable looking image. Here we use Keras Sequential model along with Dense and Batch Normalization layers. The activation function used is Leaky Relu. Refer the code snippet below. The generator model can be divided into several blocks. One block consisting of Dense Layer -> Activation ->Batch Normalization. Three such blocks are added and the last block transforms the pixels into the desired shape of the image we’re expecting as the output. The input to the model will be a noise vector of shape (100,) and the model is returned at the end. Note how the nodes in each dense layer increases as the model progresses.

Building the Discriminator model

The discriminator takes in an input of an image, flatten it and pass it through two blocks of Dense -> Activation, to finally output a scalar between 1 and 0. Output 1 should represent that the input image is real and 0 otherwise. Simple as that. Refer the code below.

Note :-
You can modify these models later to have more blocks, more batch norm layers, different activations etc. As per this example, these models are enough to understand the concepts behind GANs.

Finding loss and training

We calculate three losses, all using Binary Cross Entropy in this example in order to train the two models.

First the discriminator. It is trained to go two ways as shown in code below. First to output 1 for real images (array ‘img’) and then to output 0 for generated images (array ‘gen_img’). As training progresses the discriminator improves at this task. But our end goal is attained at a theoretical point where the discriminator outputs 0.5 for both types of inputs (i.e. indecisive if fake or real).

Next is training the generator, which is the tricky bit. To do this we first formulate a combined model of Discriminator given the output of Generator. Remember! Ideally we want this to be 1, which means the Discriminator identifying a fake generated image as real. So we train the output of the combined model against 1. See the code below.

Now let’s play!
That is pretty much the gist of the code to simply understand the workings of GANs.

*** The full code can be found here on my GitHub. You can refer to all the additional code for importing RGB images, initializing the models and logging the results in the code. Also note that during training the mini-batches are set to Hi32 images in order to be able to run on CPU.
Also the real images used in the example are 5000 images from the CelebA dataset. This is an open source dataset and I have uploaded it to my Floydhub for easy downloading which you can find here. **

There are many ways you can optimize the code to obtain better results and also to get an idea how different components of the algorithm affect the efficiency of the results. Observing the results while tweaking different components such as the optimizers, activators, normalizations, loss calculators, hyperparameters etc, is the best way to enhance your understanding of the algorithm. I chose to vary the optimizers.

So training for 5000 epochs with batches of 32, I tested with three types of optimization algorithms. With Keras this process is as easy as importing and replacing the name of the optimizer function. All optimizers in-built to Keras can be found here. 
Also the losses where plotted in each instance to understand the behavior of the models.

1. Using SGD ( Stochastic gradient descent optimizer). The output and the loss variations are shown in Figure 2 and 3 respectively.

Comment — Though the convergence is noisy we can see here that the generator loss is decreasing over epochs, which implies that the discriminator tends to detect fake images as real.

2. Using RMSProp optimizer. The output and the loss variations are shown in Figure 4 and 5 respectively.

Losses:

Comment — Here also the we see that the generator loss is decreasing which is a good thing. Surprisingly the discriminator loss on real images increases which is quite interesting.

3. Using Adam optimizer. The output and the loss variations are shown in Figure 6 and 7 respectively.

Comment — The adam optimizer yields the best looking results so far. Notice how the discriminator loss on fake images retains a larger value, meaning the discriminator tends to lean towards detecting fake images as real.

Remarks

I hope this post conveys a basic look into the inner workings of GANs from a practical perspective to understand and see how you could improve on the basic models. There are numerous implementations of GANs within the open source community on different applications and having a sound understanding on the first principles will help you immensely to understand the advancements. Also GANs being relatively new to Deep learning, there are many research avenues open for anyone interested.

So the possibilities are immense for you to explore!

Real time multiple object localization remains a grand debate in the field of digital image processing since many years. With the invent of Deep Learning and convolutional neural networks, the efforts have yielded quite promising results and the ability of well trained models detecting many classes of objects very accurately is in our hands now.

In this post I intend to present a model famously known as Yolo which stands for ‘You Only Look Once’, proposed by Joseph Redmo et al, that has shown great strides towards very fast multiple localizations of objects and its implementation using Keras, which is a high level Deep Learning library.

Let us first look differentiating among the terms classification, localization and detection. We hear these terms often in the image processing world and these are distinctive to each other in their applications. 
Classification — Refers to identifying if a given object is present inside an image or not. Common example: Cat or no-cat. 
Localization — Refers to not only identifying is a given object is present inside an image, but also distinguishing the object’s location using a bounding box.
Detection — Simply refers to multiple localizations in a single image.

Yolo is addressing the detection of objects in images and with the publication of Yolo V2 paper, this technique was quickly popularized it the field. Let’s look at the main steps in the Yolo V2 algorithm. These can be pointed out as below;

• Divide the image using a grid (eg: 19x19)
  Dividing the image into a grid of smaller images increases the possibilities of object detection inside each cell easier.

• Perform image classification and Localization on each grid cell 
  A vector for each cell representing the probability of an object detected, the dimensions of the bounding box and class of the detected image are given as the output at this step.

• Perform thresholding to remove multiple detected instances
  Thresholding picks the cells with the highest probabilities so that the more correcet bounding boxes are selected

• Perform Non-max suppression to refine the boxes more
  The technique of non-max suppression offers a convenient way to refine the results more using a calculation know as Intersection of Union

Additional point — Anchor boxes are used to detect several objects in one grid cell. This is a specialty in the Yolo V2 algorithm compared to the others.

The first implementation of Yolo was presented using a model in C known as Darknet by Joseph Redmon et al and over the evolution of the method, implementation with currently more popular ML libraries such as Tensorflow and Keras were also built.

My Github repository here presents a quick implementation of this algorithm using Keras. Using the code anyone can test with their own images and dig down into its workings. All the details regarding its installation and execution can be found in the repo.

Feel free use the code in your applications and share to spread the knowledge!

I’m not trying to sound racist but you’ll see why i said that just towards the end of this post. But first here’s the story.

I have been familiarizing myself with data science, the widely popularized field of study with the rise of AI, over he last couple of months and have been constantly fascinated by its potential applications. This inquisitiveness led me into the world of Kaggle one day. When I discovered Kaggle the little fan girl inside me rejoiced seeing how cool the Kaggle community is.

The amount of knowledge you can gain from following the work of other professional and aspiring data scientists plus the amount of shared community datasets are such incredible resources for anyone looking to step into data sceince. Highly recommended.

So going through some datasets I came across the striking US Mass Shootings — Last 50 Years (1966–2017) dataset generously shared by Zeeshan-ul-hassan Usmani. I have met Zesshan once during a speech he delivered in Colombo where he presented few of his inspiring case studies. Incredible guy. Go follow him on Twitter ->

Coming back to the dataset, it contained detailed information of 398 mass shootings in the United States. Not only due to terrorism, but also due to other factors such as domestic violence, mental health conditions, social problems etc. The overview I quote:

The US has witnessed 398 mass shootings in last 50 years that resulted in 1,996 deaths and 2,488 injured. The average number of mass shootings per year is 7 for the last 50 years that would claim 39 lives and 48 injured per year.

These facts resemble a darker side of the world we live in. The fact that we can use our knowledge to create meaning out of the past and help the future ways of peace even by a tiny amount should be the inspiration behind analysis on such data.

With that in mind here are some of the visualizations generated using the dataset.

Side Note: I’m no professional in data science. I took great help from others who have shared their kernels for these results. I used Python with numpy, pandas, matplotlib, seabron and geopy libraries. If there are any wrongdoings with the analytics, let me know in the comments.

First, the US map showing the locations of all reported incidents. The size of the bubbles identifies the amount of fatalities in each incident.

Let’s have a look at the yearly number of incidents.

A significant increase in attacks is very much clear towards the latter. We might think of connections between the rise of religious extremists and social instabilities with these results, but I would also argue that this can be a outcome of the improvement of reporting such incidents especially with the rise of social media.

Next, what can we see from the total number of attacks categorized by the month?

Seem to increase towards the beginning and the end of an year. Also see the total number of victims against each month.

We can go on breaking down the number of incidents reported by each day of the month.

Looks like the attacks are somewhat spread all over a month. But then if we group the attacks by each day of the week, we get this.

Looking at this it’s clear that the cumulative number of attacks during weekdays would be greater than that of weekends. To just make it clearer, see the below graph.

Significantly the number of victims would be larger with attacks on weekdays for obvious reasons.

Next we look at how the gender of the shooter affects the number of victims.

Male shooters have caused more damage considering with the outliers of the plot. Should be the case since the number of incidents involving a male shooter is significantly higher.

Does the age of the shooter tell us something about the incidents?

Well, the ages seems to spread across the range of 15–45 years when considered more than 3 attacks.

Lastly the below plot visualizes the cumulative number of victims by their race.

Looking at the plot it’s obvious that white Americans including European Americans have faced significantly more damage during mass shooting in the US over the last 50 years. The values report almost 2000 victims of these races, which is around 60% out of all reported victims in this dataset.

Does that depict a major target of these mass shootings over the years? Does it mean that white people in the US are more prone to terrorism in the future? Well, not exactly. Not all mass shooting incidents in this dataset were caused by terrorism at the first place. Although the data shows such a tendency with regard to all mass shooting causes, I think there has to be more elements to this in order to draw such a conclusion. Preferably more data points.

Anyway, food for thought.

Remarks:

This article takes a general path along some common EDA techniques for beginner level data analytics. This can be a good starting point for anyone interested in data science to grab hold of the general norms of data handling and interpretation of visuals.

The evolving power of data science will pave way to reveal insights that were unseen before. The applications will be soon popularized over many fields and industries that will reshape the way we think and act.

I’m looking forward to share more learnings on data science in time to come.

Give a clap👏 if it was worth reading so others could also find this article. Thanks.

A few days back I was tackling the task of developing a custom skill for Amazon’s voice assistant, a.k.a. ‘Alexa’. I thought of a very basic execution to learn from, which was to build a setup to output an on/off signal to a device using voice commands. Sounded simple enough for the first step.

Background check: I had my hardware device already integrated with AWS IoT MQTT broker, subscribed to a topic and ready to accept messages. You can do this easily following Amazon’s developer guide. Having this, the straight-forward method to achieve my objective was to make Alexa publish a message to the same MQTT topic so that the device receives the message.

Although I could find many resources on Amazon’s documentation and other forums on building Lambda functions with Alexa, I couldn’t find a direct implementation guide for my objective. Hence here it is.

Note: I’m not going to make this a step-by-step guide for setting up an Alexa skill and an AWS Lambda function. Amazon resources like this does a better job at it. Also you can search for other forums that explains the entire process. I’m just going to share the two major codes that you’ll need in the process, which are the Alexa skill interaction model settings and the Lambda function index.js code.

Interaction model of the Alexa Skill

I wrote a simple intent schema as below. I didn’t go into slots in order to keep complications out. Here the two intents — SwitchOnIntent and SwitchOffIntent defines the two voice commands expected from the user.

A simple set of sample utterances were defined as below.

Lambda Function

The lambda function facilitates communication between the Alexa skill and AWS IoT broker. The Zip file that includes the code plus all dependencies needed for the function can be found in my github repository below. The main code resides in the index.js javascript file.

miranthajayatilake/AWS_LAMBDA_LIGHTCONTROL
AWS_LAMBDA_LIGHTCONTROL - Projects with AWS servicesgithub.com

Upload the Zip file to the lambda function. For the execution role setting I used a custom role with the policy AWSIoTFullAccess for now.

You have to fill in the Alexa skill ID, the thing endpoint and the topic you want Alexa to publish to in the index.js file.

The two main handlers are as below. For instance if the SwitchOnIntent is triggered, the message ‘on’ is published to the given endpoint topic, and if the publishing is successful, Alexa provides the voice feedback saying ‘Switched on’.

Everything done right your device will receive the message from the topic and with a simple logic statement you can make the device control anything using a digital output.

That’s pretty much it. With all other configurations in place now you can test the service. I used the test option in the Alexa skill terminal first and then since I don’t have an Echo myself, used the web simulator at Echosim. It was quite cool to see the result.

Remarks:

This shows a very basic execution of Amazon’s voice assistant to perform a simple task to be used as an initial step to learn VUI. Of course there is so much more you can build on top of this, combining many features and services.

It is surely fascinating to see the way platforms like Amazon services have come to make it really easy for developers to develop sophisticated applications on a high level.

The great potential of VUI applications is pretty evident and it is surely on track to become the most used man-machine interface in the near future.

Ok. The title might seem to have two contrasting statements, but let me explain.

The big hype around AI has led to many debates on whether people are going to lose their jobs when machines take over. And the obvious answer? YES! It is going to happen.

Although I believe that the existence of a so called “Artificial General Intelligence”;

Artificial general intelligence (AGI) is the intelligence of a machine that could successfully perform any intellectual task that a human being can. (wikipedia)

is decades away even if it’s ever going to come up, the development of “Narrow Artificial Intelligence”;

Weak artificial intelligence (weak AI), also known as narrow AI, is non-sentient artificial intelligence that is focused on one narrow task.(wikipedia)

is already done and famously proven to be better than humans by developments such as AlphaGo (by Google’s DeepMind).

So what can be seen here?

Machines are thousand times better than humans in “TASK EXECUTION”.

I argue that around 4 out of 5 jobs we do today involves repetitive execution of a task at some point, that involves only a minute amount of thinking. Machines are surely going to be so good at this than us. Thousand times faster with almost no margin for error. That is good!

Think about it. We are going to get so much work done the right way and with so little time. Put into good use, see the opportunities of achieving that infinitely high efficiency and productivity! Yes please, take the jobs.

But what is left for us?

Creativity!

Put those brains into use. Soon gone will be the days where we could sit at a desk, execute the same tasks for years and years and expect a promotion. Mark my words.

If you’re not constantly in that innovative mindset, you WILL be left behind.

Creativity is going to be the most valuable skill to have and master in the future, in almost all areas of work. The question won’t be how well you can do something, but rather how creative can you get with what you do. So that you can pivot the process and let a machine take care of the execution towards meeting the goals. With execution out of the way you’ll have so much more time to be creative about the next move.

So this is good for humanity, isn’t it?

I argue that Artificial Intelligence will create a world of greater productivity and lesser mistakes with machines taking care of the execution. Along with that will be a generation with a more creative and an innovative workforce who will enjoy more time in their hands to think out-of-the-box that will pave the way for more great inventions going forward.

Food for thought!