Building an AI chatbot for your docs using Uiflow, Pinecone, Xano, and OpenAI
March 30, 2023
In this post, we are thrilled to share how you can use Uiflow, Pinecone, Xano, and OpenAI to build an AI chatbot that can answer any question about your documentation!
Overview
First, let's introduce the technologies we'll be using to build our system:
Uiflow: A no-code development platform for building web applications, specializing in catering to engineers and their teams.
Pinecone: A vector database and search platform for storing text embeddings.
OpenAI: A company with a set of APIs that use AI and other cutting-edge technologies to enable new experiences. We will specifically be using their embeddings and GPT endpoints.
Xano: A no-code backend platform for building databases, enabling authentication, and creating new API endpoints.
Step 1: Creating an embeddings database using Pinecone
To begin, we need to set up our database of articles. We will use OpenAI to generate embeddings and Pinecone to store them.
You can find related documentation for each of these here:
Before running any code, you will need to create an index on Pinecone as well. Make sure to set dimensions to 1536, as that is the output size provided by OpenAI.
Once you have created your index and set your API keys for both OpenAI and Pinecone in environment variables, you can start generating your embeddings and storing them. Here is some sample code we used to get our docs into Pinecone:
import requests
import pinecone
import openai
import os
import time
from bs4 import BeautifulSoup
openai.api_key = os.environ.get("OPENAI_API_KEY")
pinecone.api_key = os.environ.get("PINECONE_API_KEY")
# Get this from your pinecone index management screen
pinecone.init(environment="")
index = pinecone.Index("docs-embeddings")
# Base url of the domain you are scrapping
base_url = ""
# A list of urls to scrape
docs_arr = []
# Get embeddings from openAI
def get_embedding(text, model="text-embedding-ada-002"):
text = text.replace("\n", " ")
return openai.Embedding.create(input=[text],
model=model)['data'][0]['embedding']
# Function for crawling a url and storing embeddings in pinecone
def crawl_docs(url):
print("Crawling " + url)
if "http" not in url:
doc_response = requests.get(base_url + url)
else:
doc_response = requests.get(url)
if doc_response.status_code != 200 or "uiflow.gitbook.io" not in doc_response.url:
return
doc_content = doc_response.text
soup = BeautifulSoup(doc_content, 'html.parser')
main_content = soup.find('main').text.strip()
embedding = get_embedding(main_content)
try:
index.upsert([(url, embedding)])
except:
# We may hit a rate limit so rest for 10s and try again
time.sleep(10)
index.upsert([(url, embedding)])
# Crawl all our urls
for url in docs_arr:
crawl_docs(url)
Step 2: Creating a Search Endpoint with Xano
Next, we need to create an endpoint for our Uiflow app to access, which will return the most relevant result for a given query. Since we also want to pass the document content to OpenAI, we will use Xano to scrape the document URLs.