What is Microsoft Fabric: A beginners guide

Microsoft Fabric is a new SaaS data platform announced in MS Build Conference on May 23, 2023.

As per the official doc

Microsoft Fabric is a unified data platform in the era of AI.  Fabric integrates technologies like Azure Data Factory, Azure Synapse Analytics, and Power BI into a single unified product, empowering data and business professionals alike to unlock the potential of their data and lay the foundation for the era of AI.

To put it simply, Microsoft Fabric simplifies and unifies data operations. From data ingestion to data engineering, data management, machine learning, real time insights and reporting, you can do all in one unified platform.

The Fabric platform is a SaaS service for data and analytical workloads. The platform provides core capabilities of Azure Data Factory, Azure Synapse, and Power BI.

The platform can be used for:

-> Data Ingestion using Data Factory

-> Data Engineering using Synapse

-> Data Science operations using Synapse

-> Data Warehousing using Synapse

-> Real Time Analytics using Synapse

-> Reporting using Power BI

-> Actions using Data Activator (coming soon)

-> Governance using Purview

Why should I care about?

One obvious questions that comes to our mind is why should I care about it? I can anyways go a create a data pipeline today using Azure Data Factory, I can create Spark notebooks on top of Synapse, and create my reports in Power BI. Why should I go and do the same in Fabric?

The answer is: simplicity and unification.

The answer is: It simplifies data collaboration across multiple data professional disciplines

The answer is: It simplifies licensing and purchasing. You don’t have to spin up multiple capacity and infrastructure for your multiple data workloads

The answer is: It simplifies setup and configuration

The answer is: It simplifies data management. No data redundancy or data silos

The answer is: It simplifies data governance

With Microsoft Fabric – a data engineer can create data pipelines to ingest data, using the ingested data, a data warehouse engineer can create data warehouses, an ML engineer using the same data can create models, and an analyst can create data models and Power BI reports – all within the same platform.

Where is my data stored?

All workloads store the data in something new called as “OneLake”. The way we have OneDrive for documents, Microsoft has introduced “OneLake” for data.

OneLake provides a single, unified storage system for data workloads in Fabric. Each tenant has one OneLake provisioned.

There is also a concept of ShortCuts in OneLake which blows away my mind. You can create “shortcuts” of your data in AWS or other clouds right inside OneLake. This means the data stays in the original source and can be used for analytical workloads right through OneLake.

What happens to current Azure analytics solutions?

As per Microsoft they will continue to remain and provide analytical capabilities as a PaaS. Microsoft Fabric simplifies this in the form of SaaS.

Existing Microsoft products such as Azure Synapse Analytics, Azure Data Factory, and Azure Data Explorer will continue to provide a robust, enterprise-grade platform as a service (PaaS) solution for data analytics. Fabric represents an evolution of those offerings in the form of a simplified SaaS solution that can connect to existing PaaS offerings. Customers will be able to upgrade from their current products into Fabric at their own pace.  

How should I get started?

Head to fabric.microsoft.com and try a free trial today. The free trial is for 60 days that will allow you to create warehouses, lakehouses, notebooks and more.

If you are an existing Power BI customer with Premium capacity, you can try it right away.

How to enable Fabric in my tenant?

You need to go to Power BI Admin portal and enable it.

Once on the admin portal, you can “enable” the setting for the entire org or specific security group

After 15 mins you will see an icon at the bottom left corner in your fabric portal. Clicking on it will take you to the Fabric home page.

Is Power BI same as Fabric?

Somehow I feel this is same as Power BI.

Yes, the experience, the UI and core components of Power BI portal are the same in Fabric. This means:

-> Workspaces

-> Navigation

-> Collaboration

-> Content Management

-> Admin portal

-> Capacity

will look familiar to you.

However Fabric is an umbrella platform and Power BI is a critical component of it.

What happens after free trial?

Starting June 1 you can start purchasing Fabric capacities from Azure to supercharge your data and analytical workloads.

Any more questions? Feel free to ask here in the comments.

Fabric learning resources (from Microsoft)

To help you get started with Fabric, there are several resources we recommend:

  • Microsoft Fabric learning paths: Experience a high-level tour of Fabric and how to get started.
  • Microsoft Fabric tutorials: Get detailed tutorials with a step-by-step guide on how to create an end-to-end solution in Fabric. These tutorials focus on a few different common patterns including a lakehouse architecture, data warehouse architecture, real-time analytics, and data science projects.
  • Microsoft Fabric documentation: Read Fabric docs to see detailed documentation for all aspects of Fabric.

My personal favorite is this YouTube video by Justyna Lucznik, Principal Group PM for Microsoft Fabric, on Microsoft Mechanics channel.

Enjoy!

What is LangChain: A Beginners Guide to developing AI-powered applications

Introduction

With the recent advancements in technology, there has been an ever-increasing rise in online platforms that are based on Artificial Intelligence. These platforms, such as OpenAI’s ChatGPT, have started a new wave in the world. Just type in a few words and you’ll get an entire database of factual information in one place. You don’t need to do a lot of research like in earlier days. This technology has been useful for students, teachers, digital marketers, content writers, and many more. No wonder that with its release in 2022, it has gained high popularity among the youth and working professionals. 

Moving back a few years, ChatGPT came into the picture because of GPT-3, which is a Large Language Model or LLM released in 2020. To sum up, based on GPT-3, ChatGPT was created. But what is the technology behind GPT-3? Well, it’s simple, it is a Large Language Model that uses deep learning methods to produce human-like content.  

However, it cannot be ignored that it was through ChatGPT that interest in Large Language Models reached the sky. After all, it opened up new doors for everyone around the world. Indeed, LLMs and generative AI weren’t that popular when they were released but after ChatGPT people started showing their interest in this technology. As a result, there have been significant advances in LLMs. 

Historically, it all started with the release of Google’s “sentient” LaMDA. After that, the first open-source LLM “BLOOM” was released. And then OpenAI released their next-generation text embedding GPT 3.5 models. Finally, with these giant leaps in the LLM world, OpenAI released ChatGPT – which directly put LLM into the spotlight.

Around the same time, the LLM wave ushered to the shore another useful framework, which came to be known by the name LangChain. LangChain was released by its creator Harrison Chase in the fall of October 2022.  

An Overview of LangChain – An Open-Source Python Library 

LangChain is an open-source Python library that helps in the creation of LLM-powered AI applications. LangChain is filled with amazing features and tools which are a boon for anyone wishing to develop software out of it.

At the core, the framework of LangChain is built around LLMs. It can be used for chatbots, summarization, Generative Question-Answering, and a lot more!  The primary concept behind the LangChain library revolves around its ability to interconnect different components, allowing the creation of sophisticated applications for Language and Learning Models. The chains consist of multiple components on several modules such as prompt templates, LLMs, agents, memory, etc. 

Apps need to be super smart when it comes to understanding language, and that’s exactly where LangChain comes into the picture. Through the help of LangChain, it has become super easy to connect AI models with different kinds of data sources so that you can get customized natural language processing (NLP) solutions. LangChain helps to develop applications that are powered by a language model, in particular, a Large Language Model.

The technology used behind LangChain is beyond standard API calls as it is a data-aware agentic framework that enables connections with various data sources for a more enhanced and personalized experience. So, whether you’re a developer, a data scientist, or someone who has always been curious about the latest developments in NLP technology, this article has been made for you. Let’s explore how you can unlock the power of this framework in your organization.  

Key Features of LangChain 

LangChain offers the following 6 modules. 

  1. Large Language Models and APIs 

LangChain’s primary component is a Large Language Model (LLM). At the time of writing this article, LangChain supports 20 LLMs. These include OpenAI, Hugging Face models, DeepInfra, Cohere, to name a few. You may take advantage of LLMs’ full potential for a variety of use cases by using the application programming interfaces (APIs) that LangChain offers to access, interact with, and promote smooth interaction with them. For instance, you could develop a chatbot that generates customized trip plans based on the user’s preferences and prior travels. 

  1. Prompts Templates 

The primary input to an LLM is prompt. It takes effort to come up with the right prompt for the desired output. Once the prompt is built you may use it as a template for other tasks. With LangChain it becomes quite easy to manage prompts with prompt templates. LangChain provides several classes and methods to construct and manage the prompts easily. 

  1. Chains 

To create complex applications, you need to combine multiple components together. LangChain provides interfaces for building chains. For example: you may use a PromptTemplate component with a memory component to create a chatbot. Another example, if you want to ask a followup questions to a bot, you may require a SimpleSequentialChain. Chains can also be used to add context to the conversation.  

  1. Indexes 

LLMs by default do not have contextual information (your business or domain specific information). You can use LangChain Indexes to load your company documents and use specific LangChain classes for a question and answering task. 

  1. Memory 

By default LLMs and chat models are stateless. They don’t remember the previous state. So if you build an application where you require to remember the previous conversation you need memory. LangChain provides interfaces for storing short term and long term memory. 

  1. Agents 

LLMs have limited knowledge. Ex: Open AI ChatGpt is trained with data till Sep 2021. What if you need current knowledge? That’s where LangChain Agents come into play. The following agents can help with our applications: Google Search, Wikipedia lookup, calculators 

Installation and Setup of LangChain 

  1. You need Python on your system to setup and build applications using LangChain. 
  1. Then you using pip, you will install LangChain. Open your terminal and type: 
pip install langchain 

This command will install LangChain and all its dependencies. 

  1. While LangChain can connect to 20+ LLMs, in this example I have used OpenAI. To install OpenAI in your system, type: 
pip install openai 
  1. Once OpenAI SDK is installed you need to get its key. You may put the key in a .env file and load in your Python code 
import os  
from dotenv import load_dotenv  
import openai  

import langchain  

from langchain.llms import OpenAI 

load_dotenv()  

openai.api_key = os.getenv("OPENAI_API_KEY")

  1. Now you can use LangChain to interact with OpenAI with this sample code 
llm = OpenAI(temperature=0.9) 
text = "What would be a good company name for a company that makes colorful socks?" 

print(llm(text))

3 Application Examples of LangChain 

With the help of LangChain, you can create complex applications by leveraging a large language model (LLM). LangChain may be utilized to construct a wide range of applications across various industries thanks to its adaptability, customization opportunities, and powerful components.

Examples of applications that make substantial use of LangChain include the following: 

  1. Text Summarization (Data Augmented Generation) 

Do you remember writing precis in school? It was indeed a tricky task. To summarize a long text into a tiny paragraph is a tiresome and time-taking process because one has to read the entire text for hours just to find a gist. But we don’t have to worry about that anymore, because with LangChain we can easily summarize lengthy texts in a short amount of time.

You can create tools using LangChain that effectively manages text summarizing chores. Your application will be able to produce precise and succinct summaries of lengthy texts by utilizing strong language models like ChatGPT, enabling your customers to rapidly understand the core ideas of complicated publications. 

  1. Question Answering 

Building applications with question-answering features is another application for LangChain. By integrating with a large language model, your application can receive user-inputted text data and extract relevant answers from a variety of sources, such as Wikipedia, Notion, or Apify Actors.  Users looking for fast and dependable information on various subjects can find great value in this feature. 

  1. Chatbots (Language Model) 

Chatbots came into fashion a long time ago, but ever since ChatGPT, they have gained quite a popularity. As a result, the need to build such useful tools has reason among coders as well. Those days are long gone where one had to write huge codes on their own to build applications. With the advent of LangChain, this task has become super easy.

Your chatbot applications may offer a more organic and interesting user experience by utilizing LangChain’s framework and components. The language model will produce context-aware responses when users interact with your chatbots for general discussion, support questions, or other particular objectives. 

These application examples are just a few instances of how LangChain may be used to create robust and adaptable apps. Understanding the advantages of various language models can help you develop creative solutions that are tailored to the requirements of your users. 

Conclusion 

In conclusion, the framework and modules provided by LangChain streamline the development procedure, enabling programmers to take advantage of language models’ full potential and produce sophisticated, data-aware applications.  

LangChain’s modular design and thorough documentation increase its adaptability and customizability possibilities. Applications like text summarization, chatbots, and question-and-answer systems can all be created using LangChain, providing quick and precise language processing solutions.

In general, LangChain enables people, developers, and organizations to unleash the power of language, promoting cross-cultural communication, teamwork, and creativity in the digital age. LangChain has indeed taken over the internet ever since it came into being, because of its robust features and easy-to-use technology.