Data breaches at major companies are so now common that they hardly make the news. The Wikipedia page on data breaches lists over 350 between 2004 and 2023. The Equifax breach in 2017 was especially notable because over 160 million records were leaked, and much of the data was acquired by Equifax without individuals’ knowledge or consent. Data breaches are increasingly costly to companies and to affected users who must deal with the ensuing identity theft.
In 2018 the European Union implemented the General Data Protection Regulation, or GDPR. Despite its mild name, the GDPR had major consequences for individuals’ rights to control their data, and for companies that operate in the EU. Among other things, it gives the right to have personal data removed from a company’s records. This is the so-called “right to be forgotten”.
Gal Ringel is the Co-Founder and CEO of Mine, which allows users to identify which companies have their data, and it automates the process of removing data on behalf of its users. In this way, Mine aims to reduce online exposure and minimize risk for anyone using online services. Gal joins the show today to talk about his company, the impact of GDPR, and how his experience in military intelligence, venture capital, and tech led him to co-found the company.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
If you’re a sports fan and like to track sports statistics and results, you’ve probably heard of Sofascore. The website started in 2010 and ran on a modest single server. It now has 25 million monthly active users, covers 20 different sports, 11,000 leagues and tournaments, and is available in over 30 languages.
Josip Stuhli has been with Sofascore for 13 years. He started there as an engineer and is currently CTO. Josip joins the show today to talk about the challenges Sofascore encountered over the years, and how the team solved them. He discusses dealing with traffic spikes from game days, structuring and restructuring the codebase, organizing the frontend and backend, and much more.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
As Large Language Models (LLMs) and generative AI continue to grow more sophisticated and available, many organizations are starting to build, fine-tune, and customize LLMs based on their internal data and documents. This can bring incredible efficiency and reliability to data-driven decision-making processes. However, this practice comes with its share of challenges, primarily around data privacy, protection, and governance.
Let’s consider the construction of the LLM itself, which is trained on a massive amount of data collected from public and private sources. Without careful anonymization and filtering, sensitive data — such as PII or intellectual property — may be inadvertently included in the training set, potentially leading to a privacy breach.
Furthermore, privacy concerns are introduced when interacting with LLMs, as users might input sensitive data, such as names, addresses, or even confidential business information. If these inputs aren’t handled properly, the misuse or exposure of this information is a genuine risk.
In this post, we’ll explore how to work with LLMs in a privacy-preserving way when building an LLM-based chatbot. As we walk through the technology from end-to-end, we’ll highlight the most acute data privacy concerns and we’ll show how using a data privacy vault addresses those concerns.
Let’s start by taking a closer look at the problem we need to solve.
The problem: Protecting sensitive information from exposure by a chatbot
Consider a company that has uses an LLM-based chatbot for its internal operations. The LLM for the chatbot was built by modifying a pre-existing base model with embeddings created from internal company documents. The chatbot provides an easy-to-use interface that lets non-technical users within the company access information from internal data and documents.
The company has a sensitive internal project called “Project Titan.” Project Titan is so important and so sensitive that only people working on Project Titan know about it. In fact, the team often says: the first rule of Project Titan is don’t talk about Project Titan. Naturally, the team wants to take advantage of the internal chatbot and also include Project Titan specific information to speed up creation of design documents, documentation, and press releases. However, they need to control who can see details about this sensitive project.
What we have is a tangible and pressing privacy concern that sits at the intersection of AI and data. These challenges appear extremely difficult to solve in a scalable and production-ready way. Simply having a private version of the LLM doesn’t address the core issue of data access.
The proposed solution: Sensitive data de-identification and fine-grained access control
Ultimately, we need to identify the key points where sensitive data must be de-identified during the process of building (or fine-tuning) the LLM and the end user’s interaction with the LLM-based chatbot. After careful analysis, we’ve identified that there are two key points in the process where we need to de-identify (and later re-identify) sensitive data:
Before ingestion: When documents from Project Titan are used to create embeddings, the project name, any PII, and anything else sensitive to the project must be de-identified. This de-identification should occur as part of the ETL pipeline prior to data ingestion into the LLM.
During use: When a user inputs data to the chatbot, any sensitive data included in that input must also be de-identified.
You can de-identify sensitive data using Skyflow’s polymorphic encryption and tokenization engine that’s included within Skyflow Data Privacy Vault. This includes detection of PII but also terms you define within a sensitive data dictionary, like intellectual property (i.e. Project Titan).
Of course, only Project Titan team members who use the chatbot should be able to access the sensitive project data. Therefore, when the chatbot forms a response, we’ll rely on Skyflow’s governance engine (which provides fine-grained access control) and detokenization API to retrieve the sensitive data from the data privacy vault, making it available only to authorized end users.
Before we dive into the technical implementation, let’s go through a brief overview of foundational LLM concepts. If you’re already familiar with these concepts, you can skip the next section.
A brief primer on LLMs
LLMs are sophisticated artificial intelligence (AI) systems designed to analyze, generate, and work with human language. Built on advanced machine learning architectures, they are trained on vast quantities of text data, enabling them to generate text that is convincingly human-like in its coherence and relevance.
LLMs leverage a technology called transformers — one example is GPT, which stands for Generative Pre-Trained Transformer — to predict or generate a piece of text when given input or context. LLMs learn from patterns in the data they are trained on and then apply these learnings to understand newly given content or to generate new content.
Despite their benefits, LLMs pose potential challenges in terms of privacy, data security, and ethical considerations. This is because LLMs can inadvertently memorize sensitive information from their training data or generate inappropriate content if not properly regulated or supervised. Therefore, the use of LLMs necessitates effective strategies for data handling, governance, and preserving user privacy.
A technical overview of the solution
When embarking on any LLM project, we need to start with a model. Many open-source LLMs have been released in recent months, each with its specific area of focus. Instead of building an entire LLM model from scratch, many developers choose a pre-built model and then adjust the model with vector embeddings generated from domain-specific data.
Vector embeddings encapsulate the semantic relationship between words and help algorithms understand context. The embeddings act as an additional contextual knowledge base to help augment the facts known by the base model.
In our case, we’ll begin with an existing model from Hugging Face, and then customize it with embeddings. Hugging Face provides ML infrastructure services as well as open-source models and datasets.
In addition to the Hugging Face model, we’ll use the following additional tools to build out our privacy-preserving LLM-based ETL pipeline and chatbot:
LangChain an open-source Python library that chains together components typically used for building applications (such as chatbots) powered by LLMs
Snowflake, which we’ll use for internal document and data storage
Snowpipe, which we’ll use with Snowflake for automated data loading
Chroma, an AI-native, open-source database for vector embeddings
Streamlit, an open-source framework for building AI/ML-related applications using Python
RetrievalQA, a question-answering chain in LangChain which gets documents from a Retriever and then uses a QA chain to answer questions from those documents
The following diagram shows the high-level ETL and embeddings data flow:
Example of the ETL and embeddings data flow.
The ETL and embeddings flows from end to end are:
ETL
Start with source data, which may contain sensitive data.
Send data to Skyflow Data Privacy Vault for de-identification.
Use Snowpipe to load clean data into Snowflake.
Create vector embeddings
Load documents from Snowflake into LangChain.
Create vector embeddings with LangChain.
Store embeddings in Chroma.
Once the model has been customized with the Project Titan information, the user interaction and inference flow is as follows:
User interaction and inference information flow
Chat UI input
Accept user input via Streamlit’s chat UI.
Send user input to Skyflow for de-identification.
2. Retrieve embeddings
Get the embeddings from Chroma and attach to RetrievalQA.
3. Inference
Send clean data to RetrievalQA.
Use QA chain in RetrievalQA to answer the user’s question.
4. Chat UI response
Send RetrievalQA’s response to Skyflow for detokenization.
Send re-identified data to Streamlit for display to the end user.
Now that we’re clear on the high-level process, let’s dive in and take a closer look at each step.
ETL: Cleaning the source data
Cleaning the source data with Skyflow Data Privacy Vault is fairly straightforward and I’ve covered some of this in a prior post. In this case, we need to process all the source documents for Project Titan available in an AWS S3 bucket.
Skyflow will store the raw files, de-identify PII and IP, and save the clean files to another S3 bucket.
import boto3
from skyflow.vault import ConnectionConfig, Configuration, RequestMethod
# Authentication to Skyflow API
bearerToken = ''deftokenProvider():
global bearerToken
if is_expired(bearerToken):
return bearerToken
bearerToken, _ = generate_bearer_token('<YOUR_CREDENTIALS_FILE_PATH>')
return bearerToken
defprocessTrainingData(trainingData):
try:
# Vault connection configuration
config = Configuration('<YOUR_VAULT_ID>', '<YOUR_VAULT_URL>', tokenProvider)
# Define the connection API endpoint
connectionConfig = ConnectionConfig('<YOUR_CONNECTION_URL>', RequestMethod.POST,
requestHeader = {
'Content-Type': 'application/json',
'Authorization': '<YOUR_CONNECTION_BASIC_AUTH>'
}
requestBody = {
'trainingData': trainingData
}
# Connect to the vault
client = Client(config)
# Call the Skyflow API to de-identify the training data
response = client.invoke_connection(connectionConfig)
# Define the S3 bucket name and key for the file
bucketName = "clean-data-bucket"
fileKey = "{timestamp}-{generated-uuid}"
# Write the data to a file in memory
fileContents = bytes(response.training_data.encode("UTF-8"))
# Upload the file to S3
s3 = boto3.client("s3")
s3.put_object(Bucket=bucketName, Key=fileKey, Body=fileContents)
except SkyflowError as e:
print('Error Occurred:', e)
Next, we’ll configure Snowpipe to detect new documents in our S3 bucket and load that data into Snowflake. To do this, we’ll need to create the following in Snowflake:
With that, we have raw data that goes through a de-identification process, and then we store the plaintext sensitive data in Snowflake. Any sensitive data related to Project Titan is now obscured in the LLM, but because of Skyflow’s polymorphic encryption and tokenization, the de-identified data has referential integrity, meaning we can return the data to its original form when interacting with the chatbot.
Creating vector embeddings: Customizing our LLM
Now that we have our de-identified text data stored in Snowflake, we’re confident that all information related to Project Titan has been properly concealed. The next step is to create embeddings of these documents.
We’ll use the Instructor model provided by Hugging Face as our embedding model. We store our embeddings in Chroma, a vector database built expressly for this purpose. This will allow for the downstream retrieval and search support of the textual data stored in our vector database.
The code below loads the base model, embedding model, and storage context.
from langchain.chat_models import ChatOpenAI
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.embeddings.openai import OpenAIEmbeddings
model_id = "hkunlp/instructor-large"
embed_model = HuggingFaceEmbeddings(model_name=model_id)
vectorstore = Chroma("langchain_store", embed_model)
Next, we need to load all documents and add them to the vector store. For this, we use the Snowflake document loader in LangChain.
from snowflakeLoader import SnowflakeLoader
import settings as s
QUERY = "select training_text as source from custom_training_data"
snowflake_loader = SnowflakeLoader(
query=QUERY,
user=s.SNOWFLAKE_USER,
password=s.SNOWFLAKE_PASS,
account=s.SNOWFLAKE_ACCOUNT,
warehouse=s.SNOWFLAKE_WAREHOUSE,
role=s.SNOWFLAKE_ROLE,
database=s.SNOWFLAKE_DATABASE,
schema=s.SNOWFLAKE_SCHEMA,
metadata_columns=["source"],
)
training_documents = snowflake_loader.load()
vector_store.add_documents(training_documents)
With the training document and vector store created, we create the question-answering chain.
qa = RetrievalQA.from_chain_type(llm=ChatOpenAI(temperature=0.2,model_name='gpt-3.5-turbo'),
chain_type="stuff",
retriever=vector_store.as_retriever())
result = qa.run("What is Project Titan?")
This question (“What is Project Titan?”) will fail because the model doesn’t actually know about Project Titan, it knows about a de-identified version of the string “Project Titan”.
To issue a query like this, the query needs to be first sent through Skyflow to de-identify the string and then the de-identified version is passed to the model. We’ll tackle this next as we start to put the pieces together for our chat UI.
Chat UI Input: Preserving privacy of user-supplied data
We’re ready to focus on the chatbot UI aspect of our project, dealing with accepting and processing user input as well as returning results with Project Titan data detokenized when needed.
For this portion of the project, we will use Streamlit for our UI. The code below creates a simple chatbot UI with Streamlit.
import openai
import streamlit as st
st.title("Acme Corp Assistant")
# Initialize the chat messages historyif"messages"notin st.session_state.keys():
st.session_state.messages = [
{"role": "assistant", "content": "Hello ! \nHow can I help?"}
]
# Prompt for user input and saveif prompt := st.chat_input():
st.session_state.messages.append({"role": "user", "content": prompt})
# display the prior chat messagesfor message in st.session_state.messages:
with st.chat_message(message["role"]):
st.write(message["content"])
# If last message is not from assistant, we need to generate a new responseif st.session_state.messages[-1]["role"] != "assistant":
# Generate a responsewith st.chat_message("assistant"):
with st.spinner("Thinking..."):
response = "TODO"
message = {"role": "assistant", "content": response}
st.session_state.messages.append(message)
Our simple chat UI looks like this:
As you can see, the UI accepts a user input, but doesn’t currently integrate with our LLM. Next, we need to send the user input to Skyflow for de-identification before we use RetrievalQA to answer the user’s question. Let’s start with accepting and processing our input data.
To detect and de-identify plaintext sensitive data with Skyflow, we can use the detect API endpoint with code similar to the following:
Now that we’ve de-identified the user input data, we can send the question to RetrievalQA, which will then use a QA chain to answer the question from our documents.
We now have our response from RetrievalQA. However, we need to take one additional step before we can send it back to our user: detokenize (re-identify) our response through Skyflow’s detokenization API. This is fairly straightforward, similar to previous API calls to Skyflow.
Everything we need is encapsulated by the function performInference, which calls a function to reIdentifyText after the completion is returned.
Who can see what and in which format is controlled by Skyflow’s governance engine. There’s too much to cover here, but if you want to learn more, see Introducing the Skyflow Data Governance Engine.
These final steps connect our entire application from end-to-end. Now, we need to update our UI code from above so that the response is correctly set.
# If last message is not from assistant, we need to generate a new responseif st.session_state.messages[-1]["role"] != "assistant":
# Generate a responsewith st.chat_message("assistant"):
with st.spinner("Thinking..."):
response = performInference(m["content"])
With these pieces in place, here’s a quick demo of our privacy-preserving LLM-based chatbot in action:
Example of the privacy-preserving bot in action.
Tying it all together
In this article, we walked through the general steps to construct a privacy-preserving LLM-based chatbot. With organizations increasingly using LLM-based applications in their businesses and operations, the need to preserve data privacy has become acute. Concerns about protecting the privacy and security of sensitive data are the biggest adoption blocker that prevents many companies from making full use of AI with their datasets.
Solving this problem requires identifying the key points where sensitive data might enter your system and need to be de-identified. When working with LLMs, those points occur during model training — both when building an LLM or customizing one — and at the user input stage. You can use Skyflow Data Privacy Vault to implement effective de-identification and data governance for LLM-based AI tools like chatbots.
Building an LLM-based chatbot requires the use of several tools to ensure that data is handled in a manner that preserves privacy. Taking privacy-preserving measures is critical to prevent the misuse or exposure of sensitive information. By using the tools and methods we’ve demonstrated here, companies can leverage AI’s benefits and promote efficient data-driven decision-making while prioritizing data privacy and protection.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.
Cloud-based software development platforms such as GitHub Codespaces continue to grow in popularity. These platforms are attractive to enterprise organizations because they can be managed centrally with security controls. However, many, if not most, developers prefer a local IDE.
Daytona is aiming to bridge that gap. It’s a layer between a local IDE and a backend server, so developers can work locally while interfacing invisibly with a remote environment. Ivan Burazin is the CEO and Co-Founder at Daytona, and he joins the show today to talk about how Daytona works, Spotify as an inspiration for his product, and more.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
Please click here to see the transcript of this episode.
Knowledge graphs are an intuitive way to define relationships between objects, events, situations, and concepts. Their ability to encode this information makes them an attractive database paradigm.
Hume is a graph-based analysis solution developed by GraphAware. It represents data as a network of interconnected entities and provides analysis capabilities to extract insights from the data. Luanne Misquitta is VP of Engineering at GraphAware and she joins the show today to talk about graph databases, and the engineering of Hume.
Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.
Shopify is an e-commerce platform focused on enabling small businesses to sell online. The company was founded in 2006 and since then has become a core technology of online business infrastructure.
Mike Shaver is a Distinguished Engineer at Shopify and previously worked at Facebook, Mozilla, Oracle and others. At Shopify he works on the core team, which handles store fronts, merchant experience, and the commerce engine. He joins the show to talk about Shopify’s bet on Rust, their shift to Google Cloud, and the experience of changing from a management role to a high-level individual contributor role.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.
Data residency requirements, which govern where sensitive data can be stored or processed in the cloud (or in an on-prem server) are a common feature of many modern data protection laws. Because of data residency requirements, the location of sensitive data has significant regulatory compliance implications in countries and regions around the world.
In this post, we’ll look at the challenges of managing data residency with Snowflake. We’ll start by examining how Snowflake Cloud Regions address data residency challenges, and consider the compliance implications of this approach — especially when loading data from cloud storage. Then, we’ll look at how to simplify data residency compliance using one or more regional data privacy vaults.
Let’s begin with a deeper dive into data residency, and how it impacts compliance.
The implications of data residency on compliance
When you work with personally identifiable information (PII), where you store and process this information has a direct impact on your legal compliance requirements. Some jurisdictions have regulations that govern the protection and privacy of their residents’ PII, restricting how and where it’s used by businesses and other organizations.
For example, the personal data (i.e., PII) of European Union residents cannot be transferred outside the EU without appropriate safeguards.
The laws of each jurisdiction impact how you transmit, manage, process, and store sensitive data in that jurisdiction. Because data residency dictates where (geographically ) data is stored in the cloud, data residency becomes a critical concern in cloud environments that handle sensitive data.
Choose your cloud region carefully
Cloud service providers have data centers located in multiple regions around the world. When businesses sign up for cloud services and configure storage regions and other tooling, they select specific regions where their data is stored.
For many businesses, the selection of regions and locations for data storage is an afterthought.
But, treating this decision as an afterthought is a costly mistake that can come back to haunt you if you’re handling sensitive data. That’s because choosing storage regions is a weighty decision that can have a long-term impact on compliance, and on your business operations.
Snowflake Cloud Regions: a data residency solution?
Snowflake Cloud Regions let you choose the geographic location where your Snowflake data is stored across the data centers provided by the Snowflake-supported public cloud providers — AWS, GCP, and Azure. Each cloud provider offers a set of regions across the globe, with specific geographic data center locations in each cloud provider region.
If your company uses Snowflake Cloud Regions, you have your choice of providers, as well as regions where your data can be stored. When you create an account to deploy and set up Snowflake, whichever region you select becomes the primary location for data storage and for data processing resources.
At first glance, it might seem like Snowflake Cloud Regions provides a simple, effective solution to your data residency and compliance concerns. But for global companies who need global analytics, it isn’t that simple. That’s because, as noted in the Snowflake Cloud Regions documentation:
Each Snowflake account is hosted in a single region. If you wish to use Snowflake across multiple regions, you must maintain a Snowflake account in each of the desired regions.
This means that for each region where your business operates that has data residency requirements, you’ll need a different Snowflake account hosted in that region. Compliance becomes increasingly complex as you scale globally to more and more regions around the world. With this approach, running global analytics operations across different accounts to get a comprehensive view of your business can be a massive and ongoing challenge.
Instead of managing multiple Snowflake accounts with multiple Snowflake instances distributed in various regions around the world, you’d rather maintain a Snowflake instance in a single region to support global data operations. However, you still need to consider the need to honor data residency requirements for sensitive data so you can uphold your compliance obligations and safeguard customer trust.
For example, if you collect the personal data (PII) of customers located in the EU, but your Snowflake instance is located somewhere else, then you need to think through the privacy and compliance impact of storing and processing that data.
Loading data from cloud storage into Snowflake
Snowflake also lets businesses load data from cloud storage services like AWS S3, Google Cloud Storage, Microsoft Azure — regardless of which cloud platform hosts the businesses’ Snowflake account. This can present additional challenges when working to ensure data residency compliance.
For example, let’s say that your company collects PII from both US and EU customers using its website. And, let’s say that this sensitive data is then stored in a Google Cloud Storage bucket that’s located in the AUSTRALIA-SOUTHEAST1 (Sydney) region.
How does transmitting this PII data to Australia, and then storing it in Australia, affect your compliance with regulations like the EU’s GDPR?
The answer is: doing this likely puts you out of compliance with GDPR. This is just one example of how the location where sensitive data is stored — and where it’s processed and replicated — complicates the compliance requirements faced by businesses that handle sensitive PII.
Businesses that handle PII must ensure regulatory compliance by aligning their choice of cloud storage regions with the data residency requirements of markets where they operate.
And beyond compliance issues, businesses should also consider data transfer costs. Transferring data between cloud storage regions can incur significant additional costs, especially if your company is frequently transferring large volumes of data. So, we not only have compliance concerns with cross-border transfers of PII, we also have a cost concern.
So, to briefly recap our problem:
Countries and regions have their own laws and regulations that govern how to handle their residents’ sensitive data (PII).
The geographic location where your business stores and processes sensitive data impacts whether you’re compliant with the data residency requirements of the markets where you operate.
If you use Snowflake to perform analytics on PII, then the complexity of meeting your compliance obligations will depend on the location of your Snowflake account.
If you load PII data into Snowflake from cloud storage, then your compliance obligations are also impacted by the location of your cloud storage.
So, how can we meet data residency requirements, support global analytics operations, and remove the operational overhead of managing multiple Snowflake accounts and instances?
We can solve our data residency problems and protect sensitive data with one or more data privacy vaults.
How a data privacy vault simplifies data privacy
A data privacy vault isolates, protects, and governs access to sensitive customer data. Sensitive data is stored in the vault, while opaque tokens that serve as references to this data are stored in traditional cloud storage or used in data warehouses. A data privacy vault can store sensitive data in a specific geographic location, and tightly controls access to this data. Other systems only have access to non-sensitive tokenized data.
In the example architecture shown below, a phone number is collected by a front end application. Ideally, we should de-identify (i.e., tokenize) this sensitive information as early in the data lifecycle as possible. A data privacy vault lets us do just that.
This phone number, along with any other PII, is stored securely in the vault, which is isolated outside of your company’s existing infrastructure. Any downstream services — the application database, data warehouse, analytics, any logs, etc. — store only a token representation of the data, and are removed from the scope of compliance:
Example of reducing compliance scope with a data privacy vault
Snowflake handles only de-identified data
Because no sensitive data is stored outside the data privacy vault, your compliance scope is restricted to just the vault. This removes the compliance burden from your Snowflake instance.
Example pipeline where sensitive data is isolated and protected within a data privacy vault
To satisfy data residency requirements, we can extend this approach by using multiple regional data privacy vaults placed near customers whose data is subject to these requirements. With sensitive data stored in these data privacy vaults, Snowflake contains only de-identified, tokenized data. It no longer matters if you operate a single global instance of Snowflake or multiple Snowflake accounts across different regions because data residency concerns no longer apply to your Snowflake instances.
Compliance with data residency requirements now depends solely on where your data privacy vaults are located. You no longer need to worry about data residency for all the different parts of your data tech stack, including cloud storage and Snowflake. All sensitive data goes into your data privacy vaults, and these vaults become the only component of your architecture subject to data residency requirements.
Store PII in a data privacy vault in a specific region
With Skyflow Data Privacy Vault you can host your vaults in a wide variety of regions around the world. You can also route sensitive data to a data privacy vault located in a specific region for storage.
For example, consider how the application architecture shown below supports data residency requirements from multiple regions:
Using vaults to satisfy multiple data residency requirements for one Snowflake instance
Your company’s e-commerce site collects customer PII whenever a customer places an order.
On the client side, the website detects the customers’ location.
Detecting that the customer is in the EU, the client-side code uses Skyflow’s API to send the PII data to your company’s data privacy vault in Frankfurt, Germany. Note: For customers based in the US, the PII data is instead routed to the data privacy vault in the US (in this case, Virginia).
This EU-based customer’s sensitive PII is stored in the EU-based data privacy vault, and Skyflow’s API responds with tokenized data.
The client-side code sends the customer order request, now with tokenized data, to the server.
The server processes the order, storing the data (now de-identified and tokenized) in cloud storage in the “Oregon, US” region.
At the end of the week, your company’s Snowflake instance in Tokyo, Japan, loads the data (already de-identified and tokenized) from cloud storage to perform analytics.
By using multiple vaults located in different regions around the world, you can easily manage all of your sensitive data to meet various data residency compliance obligations across each of your global markets.
The data privacy vault architectural pattern vastly simplifies the challenges of data residency and compliance. Additionally, by de-scoping Snowflake from the compliance burden of data residency, global analytics executes as normal — within a single Snowflake instance.
Final thoughts
Compliance regulations and their data residency requirements require that businesses uphold stringent standards for data localization, protection, privacy, and security to reduce their risk of breaches, penalties, and reputational damage. However, businesses with customers (and data) located in a variety of global regions face the added challenge of managing multiple regulations across jurisdictions.
Using data privacy vaults lets businesses simplify their global compliance obligations around data residency as they relate to Snowflake and cloud storage.
Using a data privacy vault, companies can isolate and secure all sensitive data in one or more data privacy vaults, removing Snowflake and cloud storage from their compliance footprint. At the same time, by leveraging data privacy vaults in different regions, companies can help ensure that sensitive data is stored and transmitted according to the laws and regulations of each specific region where they operate.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.
When Adam Berger was at Uber, his team was responsible for ensuring that Uber Eats merchants correctly receive and fulfill orders. This required them to think hard about engineering workflows and state management systems. Six years of experience at Uber motivated Adam to create State Backed, which is an open-source backend system written in Typescript. The platform is oriented around using state machines to model application logic, and automatically handles the associated persistence, infrastructure, and consistency.
Adam joins the show to talk about state machines, why they’re the right paradigm to manage global application state, and what are the practical advantages of using state machines in a backend platform.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
One of the key challenges that teams encounter is how to smoothly collaborate on converting a design into code. For example, if a designer designs a web component, how can it be most efficiently implemented by a developer? What happens if the designer needs to adjust the design and communicate this change to the developer? These sorts of issues can often lead to inefficiencies and frustrations on a team.
Figma recently announced Dev Mode which aims to smooth design and developer collaboration. The idea is to tie the visual language of designers to the actual component implementation of the developer.
Marcel Weekes is VP of Product Engineering at Figma, and before that he spent 6 years at Slack. Marcel joins the show today to talk about Dev Mode, how it will boost collaboration between designers and devs, and the new Figma VS Code plugin that brings design into the IDE. Marcel also talks about the concept of the “new manager death spiral”, and how individual contributors, or ICs, can smoothly transition to management roles.
Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript.
Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.
Frontend web frameworks are software toolkits that handle many of the low-level and repetitive aspects of building a website. These frameworks have made it easier than ever to build a modern website. The open-source Astro framework was created in 2021 for the purpose of creating simple static sites that load quickly. A key factor to its high performance is that, by default, webpages don’t include any JavaScript. However, Astro gives users the ability to opt into using JavaScript wherever it makes sense in their projects. Astro also has the file based routing and server-side rendering capabilities found in frameworks like NextJS.
James Q. Quick has worked at Microsoft, PlanetScale, and Auth0 and is a popular technical content creator. He’s an advocate of the Astro framework and joins the show to talk about what distinguishes it from NextJS, Sveltekit, and other major frameworks.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
Observability software helps teams to actively monitor and debug their systems, and these tools are increasingly vital in DevOps. However, it’s not uncommon for the volume of observability data to exceed the amount of actual business data. This creates two challenges – how to analyze the large stream of observability data, and how to keep down the compute and storage costs for that data.
Chronosphere is a popular observability platform that works by identifying the data that’s actually being used to power dashboards and metrics. It then shows the cost for each segment of data, and allows users to decide if a metric is worth that cost. In this way, technical teams can manage costs by dynamically adjusting which data is analyzed and stored. Martin Mao is the Co-founder and CEO of Chronosphere and he joins the podcast today to talk about the growing challenge of managing observability data, and the design of Chronosphere.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
The open source coding philosophy has enormous appeal to many software engineers, and with good reason. Open source libraries, applications, and operating systems are now essential to the overall technology ecosystem. And the number of open source projects is only increasing. But many developers don’t know how to get involved in open source. Or, they may have even faced resistance when trying to make a pull request to their favorite open source codebase.
Open Sauced is a platform to help developers get involved in open source development. While the number of GitHub stars on a project is often seen as a metric of success for a code base, Open Sauced focuses on the number of new contributors on a project. This number serves as a signal to help drive the platform’s recommendation system, which pairs its users with open source projects in need of developers. Brian Douglas is a former Developer Experience Lead at Netlify and he was the Director of Developer Advocacy at GitHub. He is also the founder and CEO of Open Sauced and he is our guest in this episode.
Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.
There are countless real world scenarios where a workflow or process has multiple steps, and some steps must be completed before others can be started. Think of something as simple as cooking dinner. First you look up a recipe, then you write down the ingredients you need, you go shopping, and then you cook. These steps must be run in a certain order, and the state of the workflow must be tracked throughout. Workflow management is everywhere in the software world, and today it’s common for teams to engineer custom solutions. This makes sense, because creating a general-purpose solution for workflow management is a hard conceptual problem, and perhaps an even harder engineering challenge.
Maxim Fateev has a deep background engineering distributed systems and workflow management services at Google, Amazon, and Microsoft. In 2015, he joined Uber and helped create the open-source project, Cadence, which is an orchestration engine to execute asynchronous long-running business logic. The success of Cadence led Max to co-found Temporal, which is an open-source programming package for workflow execution. Max joins the show today to talk about the engineering challenges at Temporal, the concept of “durable execution”, how he organizes his engineering teams, and more.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.
Speech technology has been around for a long time, but in the last 12 months it’s undergone a quantum leap. New speech synthesis models are able to produce speech that’s often indistinguishable from real speech. I’m sure many listeners have heard deep fakes where computer speech perfectly mimics the voice of famous actors or public figures. A major factor in driving the ongoing advances is generative AI.
Speechlab is at the forefront of using new AI techniques for realtime dubbing, which is the process of converting speech from one language into another. For the interested listener, we recommend hearing the examples with President Obama speaking Spanish or Elon Musk speaking Japanese in this YouTube video. Ivan Galea is the Co-founder and President at Speechlab and he joins the show to talk about how we’re on the cusp of reaching the holy grail of speech technology – real time dubbing – and how this will erase barriers to communication and likely transform the world.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
If you’re a developer, there’s a good chance you’ve experimented with coding assistants like GitHub Copilot. Many developers have even fully integrated these tools into their workflows. One way these tools accelerate development is by autocompleting entire blocks of code. The AI achieves this by having awareness of the surrounding code. It understands context. However, in many cases the context available to an AI is limited. This restricts the AI’s ability to suggest more sweeping changes to a codebase, or even to refactor an entire application.
Quinn Slack is the CEO of Sourcegraph. He is now hard at work on the challenge of giving more context to AI – to make it aware of entire codebases, dependencies, error logs, and other data. Quinn joins the show today to talk about what it takes to move beyond code autocomplete, how to develop the next generation of coding AI, and what the future looks like for software engineers and programming languages.
Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript.
Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.
Documentation is something that everyone knows is important but it’s often difficult to get right. On software teams, good documentation can help to onboard new people, improve communication across teams, and troubleshoot technical issues. When an application, API, or library is a commercial product, the quality of its documentation can determine whether it attracts users and succeeds on the market.
GitBook is a popular documentation platform built with TypeScript and Node. It’s found particular use among software teams for creating technical documentation. Addison Schultz is the Developer Relations Lead at GitBook and he joins the show to talk about GitBook’s development, software stack, and how it’s adding new integrations for VS Code, Slack, and other tools.
Full disclosure: GitBook is a sponsor of Software Engineering Daily.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.
Observability is becoming an increasingly competitive space in the software world. Many developers have heard of Datadog and New Relic, but there are a seemingly countless number of observability products out there.
Costa Tsaousis (he/him) is the Founder and CEO of Netdata. His goal was to build an open-source platform that was high-resolution, real-time, and easily scalable. Netdata is the result. It’s relatively new to the crowded observability space, but it’s grown into a major presence. Costa joins the show to talk about the design philosophy of Netdata, and how it inverts a common observability design pattern.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
Ongoing advances in generative AI are already having a huge impact on developer productivity. Tools like GitHub Copilot and ChatGPT are increasing the velocity of code development, and more advances are on the horizon. However, an ever-growing challenge for developers is how to manage their coding resources – things like code snippets, website links, messages, and screenshots. This is hard for individual developers, but even harder for teams.
Tsavo Knott is the Co-Founder and CEO of Pieces. Tsavo thinks deeply about developer productivity and he joins the podcast today to talk about how Pieces is using AI to automate the process of saving, curating, and iterating on coding resources for developers and teams.
Full Disclosure: Pieces is a sponsor of Software Engineering Daily
Mike Bifulco is CTO and co-founder of Craftwork. He’s also a developer advocate, writer, podcaster and serial startup founder. In past lives, Mike worked for Google, Stripe, Microsoft, and Gymnasium. Mike is also co-founder of APIs You Won’t Hate, a community for API Developers on the web. Mike’s publishes a weekly newsletter for product builders called Tiny Improvements at mikebifulco.com. Mike is on Mastodon at https://hachyderm.io/@irreverentmike
The importance of data teams is undeniable. Most companies today use data to drive decision-making on anything from software feature development to product strategy, hiring and marketing. In some companies data is the product, which can make data teams even more vital. But there’s a common problem – analyzing data is hard and time consuming. Lots of people have questions they want to answer with data, but data teams often don’t have the resources to move quickly. This can create a pernicious effect where organizations stop asking questions about their own data.
Amanda Kelly thinks a lot about data and the dynamics of data teams inside organizations. She’s worked at Google X, and on self-driving cars and cybersecurity. Her experiences on data teams inspired her to co-found Streamlit, which is an open source Python library that gives primitives to assemble a data app for rapid data visualization and interaction. Her goal was to accelerate the iteration loop to go from a question to a data-driven answer. Amanda is currently the COO of Streamlit and a Product Director at Snowflake, and she joins us today to talk all about data and how she’s building Streamlit.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.
Platform engineering is difficult to get right, and in the age of DevOps and cloud computing, software developers increasingly serve as platform engineers while they’re building their applications. This can be an engineering challenge because organizations often require their platforms to provide fine-grained control and compliance management.
Cory O’Daniel is the CEO and Co-Founder of Massdriver, which he started in 2021 with the goal of helping engineering and operations teams build internal developer platforms. Cory’s company was in the 2022 Y Combinator class, and he has been hard at work developing his platform. He joins the show today to talk about how he thinks about platform engineering, and the challenge of abstracting away infrastructure.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
Today it’s estimated there are over 1 billion websites on the internet. Much of this content is optimized to be viewed by human eyes, not consumed by machines. However, creating systems to automatically parse and structure the web greatly extends its utility, and paves the way for innovative solutions and applications. The industry of web scraping has emerged to do just that. However, many websites erect obstacles to hinder web scraping. This has created a new kind of arms race between developers and anti-scraping software.
Bright Data has developed some of the most sophisticated consumer tools available to scrape public web data. Erez Naveh is an entrepreneur and former engineer at Meta. He is currently the VP of Product at Bright Data. Erez joins us in this episode to talk about Bright Data’s mission to structure the open web, and the toolkit they’ve developed to make this possible.
Paweł is the founder at flat.social the world’s first ‘flatverse’ start-up and glot.space, an AI-powered language learning app. Pawel’s background is as a full-stack software engineer with a lean and experimental approach towards product development. With a strong grounding in computing science, he spent the last decade getting early-stage products off the ground – both in startup and corporate settings. Follow Paweł on Twitter, LinkedIn and his personal website – pawel.io.
If you’re a developer, you’ve probably worked with an API, or application programming interface. An API is a set of rules for how to communicate with an applications or device. For example, when you build an app and want to use Stripe to handle payments, or use Slack to deliver notifications, it’s APIs that make this possible. Handling communication between different applications was historically challenging, but with the growth of cloud computing and the need for smooth interoperability, APIs have become standard, and are now often considered essential to make a company accessible and visible. The growth of APIs is about to accelerate even more because of generative AI. The reason is that good APIs will be needed so AIs can write code to stitch together multiple systems.
Postman is a popular application and platform for building and using APIs, and they recently released their 2023 State of the API Report. Joyce Lin is the Head of Developer Relations at Postman and she joins the show today to talk about the history of APIs, why APIs have exploded in popularity, and what the future looks like.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
In 1989, British computer scientist Tim Berners-Lee invented the World Wide Web (WWW) while working at CERN. The original motivation behind this invention was to improve information sharing within the institution and with external collaborators. Tim’s creation proved to be a success and it rapidly expanded beyond academia. Fast-forward to today, and the aggregate of all web pages amounts to an immense volume of web data with approximately 1.13 billion websites on the internet.
Much of the web is optimized to be viewed by human eyes, rather than for use by automated services that could reorganize the data, extend its utility, and pave the way for innovative solutions and applications. The industry of web scraping has emerged to meet this technical need, and provide a means to add structure to otherwise unstructured web data. There are numerous companies offering robust APIs, allowing developers easy access to data without having to grapple with undue complexity. Nevertheless, developers frequently find themselves resorting to web scraping techniques to obtain the data they require.
Web scraping in action
Web scraping is nearly as old as the web itself. In essence, it’s the process of automated extraction of data from websites. As previously noted, the internet is filled with unstructured data. Web scraping techniques can transform this yet untapped value into an organized resource, suitable for a variety of new applications.
Let’s consider a practical example. Imagine you operate a large eCommerce website specializing in PC components. With thousands of items in stock, setting competitive prices to maximize profit is crucial. Prices can fluctuate due to broader economic factors (think NVIDIA graphic cards and the crypto boom) or specific events like seasonal holidays. Failing to match competitors’ prices by being too inexpensive or expensive could put your business at a significant disadvantage. Manually checking all product data would be impractical and time-consuming. As a savvy eCommerce owner, instead of doing the work manually, you could employ a web scraper to bring all that data to your doorstep. You might source it from multiple websites or even just one – for example Amazon.
We spoke with Erez Naveh, VP of product at Bright Data. Erez frames web scraping as follows:How do we know what prices are set by the competition? In the physical world, a common way to do it is to send a mystery shopper who can look at the shelves and see how products are priced. Web scraping those prices online is a digital version of the same process.
Another example comes from the travel industry, where numerous websites offer flights, hotels, and other services. Yet again, prices can fluctuate widely, and the information is often dispersed across multiple platforms. While most booking sites, such as Booking.com or Airbnb, primarily address basic user queries, such as availability of properties for specific dates in a given location, the data they hold and present has value beyond answering that single question. Access to this information can enrich the user experience through innovative travel features and also provide valuable insights for business intelligence, such as trend forecasting and alerting.
Practicalities of web scraping
Let’s delve into the technicalities of setting up a web scraping operation. Once a target for web scraping is identified, the developer faces several challenges and decisions. The first step involves understanding the website’s structure and answering key questions including: What type of data is present? How are the page elements organized? Are there discernible patterns that could streamline the scraping process? Does the site utilize pagination? While modern web development typically follows industry standards, some websites may still prove more difficult to scrape than others. Moreover, if the developer has no control over the target website’s architecture, the scraping code may require frequent updates to adapt to any changes in site structure.
Expanding on the technical aspects, once the web scraper is fully configured, it mimics human browsing behavior by sending a series of HTTP requests to the target website’s servers. These requests might include GET or POST methods, depending on what data retrieval or submission is needed. The scraper may also handle cookies, session IDs, and even deal with CAPTCHAs or JavaScript-rendered content if programmed to do so. Typically, the returned data is in HTML format, which then undergoes a parsing process to extract relevant information. Parsing can be done through various methods, for example by traversing the Document Object Model (DOM). Finally, the extracted data is structured into a machine-readable format like JSON or CSV, facilitating easy integration with other applications or data analytics tools.
Although web scraping can be implemented in nearly any modern programming language, Python and JavaScript are nowadays the go-to choices for most developers.
In the JavaScript ecosystem, web scraping is often performed using Node.js with the help of libraries such as axios for HTTP requests and cheerio for HTML parsing. For more dynamic websites that depend on client-side JavaScript rendering, Puppeteer is often the library of choice. It provides a headless browser environment, allowing for the rendering of pages, execution of JavaScript, and interaction with the web page through simulating actions like clicks. This enables the scraping of data that is populated dynamically.
Similarly, in the Python landscape, multiple libraries are available for various aspects of web scraping. The requests library is often used for HTTP requests to fetch web pages. For parsing HTML or XML documents, Beautiful Soup and lxml are popular choices. While Puppeteer can be also used with Python, Playwright emerges to be a popular solution too. Even though it originally has been a framework built for website testing, it does a great job at automating browser tasks which can be used for the extraction of web data.
Not an easy ride – challenges of web scraping
As previously mentioned, developers creating web scrapers usually have no control over the target website but are fully responsible for ensuring their scraping service runs smoothly. Here are some common challenges:
Website structure changes: If the scraper’s functionality is closely tied to the HTML structure of the target website, even a simple change in layout can completely throw it off. There is no guarantee that the structure will stay the way it is nor is there any assurance that the developer will be notified that something is about to change. This unpredictability can lead to both unexpected costs of upgrading the web scraper and down time in its operation.
Rate limiting: Websites may regulate the number of requests you can make in a given timeframe. Some of the common algorithms for rate limiting include Token Bucket and Leaky Bucket, which allow for occasional bursts of traffic but constrain the average rate of incoming requests. Rate limits can be set based on IP addresses, user sessions, or API keys. Running into a rate limit, depending on the nature of the data that is being scraped, might mean that obtaining the data will take too long unless the web scraper is using multiple proxies.
CAPTCHA: Are you a robot? CAPTCHA is a well-known mechanism for distinguishing humans and computers apart by providing challenges that are computationally hard for bots to solve but relatively easy for humans. CAPTCHAs serve as a barrier against web scraping, automated form submission, and brute-force attacks. Nevertheless, they are not foolproof and can be bypassed using techniques like machine learning-based object recognition or sometimes even by employing human-solving services. CAPTCHA is relatively easy to integrate into a website by using a provider like Google’s reCAPTCHA.
Browser Fingerprinting: Websites can store data in cookies and local storage to identify a user. Identifying a user can be as simple as saving one piece of data with a unique identifier. Could a user be still identified and tracked without the ability to use cookies or local storage? Turns out, it can – by using a combination of user-agent string, screen resolution, installed fonts, plugins, and even behavior like mouse movements or keystroke dynamics. In aggregate, these attributes can create a unique “fingerprint” for each user. From the perspective of web scraping, this can pose a challenge as programmatic behavior is usually repetitive in nature and can cause the website to flag it as a potentially automated activity. While hard to circumvent, it’s not impossible using modern methods such as rotating user-agents, modifying street dimensions and even mimicking random mouse movements.
Taking web scraping to the next level
Building a web scraper is a time-consuming process with no guarantee that the final product will be maintenance-free. From adapting to the dynamic and sometimes inventive nature of websites to overcoming obstacles designed to hinder a scraper’s effectiveness, the path to creating a reliable web scraper is often fraught with challenges.
Fortunately, solutions like Bright Data—a comprehensive, award-winning suite of web scraping tools—can significantly improve the web scraper development experience. Bright Data is not just another scraping library but a full powerhouse of functionalities, tailored web scraping templates and proxies. Alongside each other, all aspects and features of Bright Data allow developers to abstract the intricacies of scraping away and focus on what they are actually building.
According to Erez Naveh of Bright Data: “We have customers that range from dedicated web scraping teams, to a huge e-commerce business that needs to keep track of all the prices in the market, to single developers that don’t have many resources. While large customers might already have an entire web scraping department with machine learning talent, small ones usually don’t and cannot efficiently deal with the challenges on their own. We have solutions for both of them.”
What makes Bright Data so valuable? Let’s have a look through some of the most useful features:
Proxies: 72 million strong, ethically sourced, proxy network which includes residential proxies, ISP proxies and even IPs from mobile networks around the world. This extensive network not only allows your web scraper to view websites from various perspectives but also addresses many of the rate-limiting and browser fingerprinting issues we discussed earlier.
Scraping Browser: A specialized automated browser designed to streamline the web scraping process. It offers a 3-in-1 solution that integrates proxy technology, automated website unblocking, and browser functionalities. Compatible with popular scraping frameworks like Puppeteer, Playwright, and Selenium, the Scraping Browser manages challenges like CAPTCHA solving, proxy rotation, and browser fingerprinting automatically. Hosted on Bright Data’s scalable infrastructure, it allows for cost-effective scaling of data scraping projects.
Web Scraper IDE: Web Scraper IDE an all-in-one tool for efficient and scalable web scraping. A developer can jumpstart a project with pre-made templates for popular data sources (like LinkedIn, Amazon and YouTube) and debug the results on the fly with interactive previews. If you’re after scraping data from search engines like Google or Bing, Bright Data also provides – SERP API makes it easy by converting actionable data insights from search results.
Ready datasets: If creating a web scraper is not your thing, maybe taking advantage of data that has been scraped before is a better solution? Bright Data offers fresh datasets from some of the most popular public websites. From LinkedIn to Amazon, there are a lot of ready-made solutions to choose from. It’s also cheaper than scraping the data yourself. Nevertheless, if analyzing the obtained data is also not your thing, you can use Bright Insights to receive actionable eCommerce market intelligence.
Conclusion
In 2023, web scraping remains a pivotal activity for data collection across various industries, from eCommerce to travel. However, the process is often convoluted and laden with challenges like ever-changing website structures and security mechanisms. Bright Data emerges as a comprehensive solution, offering an extensive suite of web scraping tools that streamline the process for developers. It provides a robust proxy network to navigate around rate-limiting issues at scale and a Scraping Browser to facilitate efficient data extraction. Additionally, Bright Data offers pre-scraped datasets, serving as an all-encompassing resource for both novice and experienced web scrapers.
What is coming in the future for web scraping products? While the race in overcoming challenges of accessing websites on a large scale continues as it did, new technological breakthroughs like LLMs allow not only to scrape the websites better but also make better use of the extracted data.
Erez Naveh spoke to us about the future of web scraping and said“We found so many useful use cases of LLMs that I believe that the next year or a couple of years will be just figuring out how to leverage it and optimize it to the benefit and value of our customers. For instance – a fun example. In the pre-collected datasets, users can press a button and add a new smart column and assign a prompt to it. The new column will be filled with data in an AI-enhanced way almost in an instant, without having to spend time training any new models.”
Full disclosure: Bright Data is a sponsor of Software Engineering Daily.
There are hundreds of observability companies out there, and many ways to think about observability, such as application performance monitoring, server monitoring, and tracing. In a production application, multiple tools are often needed to get proper visibility on the application. This creates some challenges. Applications can produce lots of different observatory observability data, but how should the data be routed to the various downstream tools? In addition, how can data be selectively sent to different storage tiers to minimize costs?
Calyptia is a service that helps manage observability data from source to destination. Eduardo Silva is the founder and CEO of Calyptia and he joins us in this episode.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
The Hypertext Transfer Protocol, or HTTP, is used to load webpages using hypertext links, and it’s the foundation of the web. Tim Berners-Lee famously created HTTP version 0.9 in 1989, and defined the essential behavior of a client and a server. Version 1.0 was eventually finalized in 1996, and its secure variant called HTTPS is now used on more than 80% of websites. HTTP continues to undergo intense development and version 3 in now being actively adopted across the tech industry.
Nick Shadrin is a Software Architect at NGINX, and Roman Arutyunyan is a Principal Software Engineer at NGINX. Nick and Roman are experts in HTTP and they join the show today to tell the history of its evolution since 1989, and how NGINX is implementing support for HTTP/3.
Full disclosure: NGINX is a sponsor of Software Engineering Daily.
Mike Bifulco is CTO and co-founder of Craftwork. He’s also a developer advocate, writer, podcaster and serial startup founder. In past lives, Mike worked for Google, Stripe, Microsoft, and Gymnasium. Mike is also co-founder of APIs You Won’t Hate, a community for API Developers on the web. Mike’s publishes a weekly newsletter for product builders called Tiny Improvements at mikebifulco.com. Mike is on Mastodon at https://hachyderm.io/@irreverentmike
A platform as a service, or PaaS, is the concept of a complete development and deployment environment in the cloud. One of the best examples is Heroku, which was created in 2007 and later acquired by Salesforce. Although these services are great for helping startups get off the ground quickly, they can ultimately become a form of technical debt because of issues with cost, control, scale, and reliability.
Today we’re speaking with Brandon Bayer. Brandon is a licensed aircraft pilot, and he’s also the CEO of Flightcontrol, a platform as a service company that he co-founded. Flightcontrol is built on top of AWS, and allows users to deploy it on their own AWS infrastructure. In this episode, Brandon talks about how his team engineered Flightcontrol, and how it was designed for small and large teams, with scalability and maintainability in mind.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
It’s now clear that the adoption of AI will continue to increase, with nearly every industry working to rapidly incorporate it into their systems and applications to provide greater value to their users. Business analytics is a key domain that promises to be radically reshaped by AI.
Alembic is an AI platform that integrates web data, product conversion metrics, and social media to guide business decision making. John Adams is the Co-founder and Chief Innovation Officer at Alembic, and he joins the podcast to talk about data and engineering at the company.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
When StackOverflow launched in 2008 it lowered the barrier to writing complex software. It solved the longstanding problem of accessing accurate and reliable programming knowledge by offering a collaborative space where programmers could ask questions, share insights, and receive high-quality answers from a community of experts.
Generative AI has impacted the way programmers want to consume this knowledge. It has also opened new possibilities in terms of getting a personalized and real-time response.
StackOverflow has decided to put a fifth of its organizational effort in Generative AI to improve the user experience of the website.
Ellen Brandenberger leads the Product Innovation team at Stack Overflow and she joins us in this episode.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
Picture a user interacting with a web app on their phone. When they tap the screen the app triggers communication with a server, which in turn communicates with a database. This process then happens in reverse to eventually update what the user sees on-screen. The latency for this round trip depends a lot on the physical distance between the user and the server. In a traditional web app, latency can be especially poor when the user and the server are separated by an ocean, and they must communicate through an underwater trans-oceanic link. However, with edge computing, multiple servers and data centers create a distributed system that greatly improves response times.
Turso is an edge-hosted, distributed database based on libSQL, an open-source and open-contribution fork of SQLite. It was designed to minimize query latency for applications where queries come from anywhere in the world. Glauber Costa is the Founder and CEO of Turso and he joins us in this episode.
Full disclosure: Turso is a sponsor of Software Engineering Daily.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
Developer experience, or DevX, is a critical aspect of modern software development that focuses on creating a seamless and productive environment for developers. It encompasses everything from the tools and technologies used in the development process to the documentation, libraries, and frameworks available to streamline coding tasks. An emphasis on DevX can enhance individual developer productivity and, as a consequence, boost the overall success and innovation of software projects.
Amazon Web Services recently created the Next Gen DevX Division. This new division is using generative AI and foundation models to reimagine the experience of all builders on the AWS platform. Deepak Singh is the Vice President of Next Gen DevX at AWS and he joins us in this episode.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm.
Infrastructure as code refers to the use of software and configuration files to convey infrastructure specifications. This is in contrast to the traditional approach of manually provisioning servers, operating systems, storage, and other infrastructure components. With the growth of cloud computing, the infrastructure as code paradigm is becoming more integral to managing compute resources.
AutoCloud is a platform for automating resource provisioning, and is at the forefront of the move towards infrastructure as code. Tyson Kunovsky is the founder and CEO of AutoCloud, and he joins us in this episode.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
Charlie Gerard is a highly accomplished software engineer and technologist. She’s worked at Stripe, Netlify, and Atlassian and authored the book, Practical Machine Learning in JavaScript. In her spare time, Charlie explores the field of human-computer interaction and builds interactive prototypes using hardware and machine learning. Some of her recent projects include building a DIY aircraft radar system, and creating a gesture-based computer interface using Bluetooth earbuds and machine learning.
Charlie joins the podcast today to talk about her work and latest interests.
Mike Bifulco is CTO and co-founder of Craftwork. He’s also a developer advocate, writer, podcaster and serial startup founder. In past lives, Mike worked for Google, Stripe, Microsoft, and Gymnasium. Mike is also co-founder of APIs You Won’t Hate, a community for API Developers on the web. Mike’s publishes a weekly newsletter for product builders called Tiny Improvements at mikebifulco.com. Mike is on Mastodon at https://hachyderm.io/@irreverentmike
Incident management is the process of responding to unplanned events or service interruptions, and then restoring service to an operational state. Having robust incident management is vital to many software teams. Blameless is a framework designed to help software companies manage their production incidents effectively. It provides a workflow for managing the incident response as well as the post-incident retrospective process. In this way, Blameless aims to give a unified learning and process feedback framework that focuses on actionable insights.
Ken Gavranovic is the Chief Operating Officer at Blameless and he’s our guest today.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
Since its release by Facebook in 2013, React has etched itself into the canvas of web development. Fast-forward a decade, and it’s no exaggeration to say that React has become the most popular web framework, globally. The majority of the world’s web apps now have a consistent and composable approach for managing state in which the apps react to state changes by re-executing a declarative render tree of pure components.
The evidence of React’s dominance can be seen in StackOverflow trends but also in the vibrance and growth of its ecosystem. Although it is now over 10 years old, it shows no signs of relinquishing its dominance in the frontend development space.
While the core principles of React have been adapted widely within the frontend world, not every backend solution aiming to support a reactive application is built with reactivity in mind. As a result, developers have to often use state management tools and techniques to bridge the gap between the two. Within this article, we will have a look at different methods of how React applications can manage their state when working with a non-reactive backend. Also, we will review several serverless backend platforms that allow developers to build web applications that are scalable, reactive, and real-time.
Choosing the Stack
Developing a new web application is a process that involves a series of critical decisions. The developer might begin by defining the features of the application, deciding how to structure the frontend code, and choosing the toolkit to style the UI elements.
While this may seem like a substantial part of the process, an essential decision still awaits – choosing a method for the application to communicate with the backend and store its internal state. React itself doesn’t dictate this aspect, leaving the developer with the freedom to choose the most suitable approach depending on the type of backend that the application has to deal with.
One option, naive but illustrative, might be to use the built-in fetch API to request the data and store it within the component’s state. Whenever the data is refreshed, the component will render again. This straightforward method carries a notable drawback as the data won’t be shared between different components. Moving one step forward from this solution, a developer could choose to use React Context which is a popular approach to managing a React’s application state. It comes out of the box with React and can be easily coupled with the useState hook to facilitate sharing data between different components. Nevertheless, the React Context solution doesn’t always scale well for large, complex applications. In such situations, developers often turn to a state management package like Redux or MobX.
State management tools can provide a “bridge” between a non-reactive backend and a React.js front-end. To illustrate that, let’s take an application built with React, Redux and a REST API as an example. Architecting an app that uses Redux as a state management involves organizing it into 3 separate parts – the state, the view and actions. The data flow between those is one-directional. A user might trigger an action within the UI which fetches data from a backend API. The received data ends up in a reactive state. The state then notifies the view about the changes and it is up to the view to decide about what to render. The data received is kept in the local memory until refreshed again.
Navigating the Challenges of Cache Management
As mentioned in the above example, the application might keep a copy of the data received from the backend to be able to serve it to the user faster and to eliminate the need to send another network request. In computing science, this concept is called a cache – a small memory store that keeps frequently used information readily available. In the example, the cache is Redux’s state.
Keeping a copy of the data locally, while handy, creates a potential for new issues to emerge. Now, the system needs to determine whether the cached data is current – should we keep it, remove it, or render it in the UI immediately? Rendering data that is readily available in cache but potentially not current can lead to confusion on the user side. On the other hand, fetching it again each time will lead to potential delays and give ground to a poor user experience.
A first and most intuitive solution to resolve this problem might be to associate a maximum age (max-age) value with each piece of data that we store locally. Once the max-age limit is exceeded, the system would fetch the data again. However, determining an appropriate max-age can prove challenging, and sometimes, depending on the data’s nature, it might be impossible.
Here’s where the concept of Stale-While-Revalidate (SWR) comes to the rescue. In practice, SWR first returns the locally cached data to be rendered while simultaneously revalidating the data in the background. If the data has indeed changed, the UI is then rendered again. This technique ensures that the cached data remains both immediate and fresh. Note that SWR doesn’t concern itself with how requests are made – it’s merely a strategy of storing and updating the data.
Two popular libraries that currently support SWR are React Query and SWR (developed by the same team that made Next.js). Both come packed with valuable features, such as caching, deduping multiple requests into a single one, memoizing query results, and a host of performance optimizations. Along with other solutions like swrv and Apollo, they present a comprehensive solution aiming to bridge the gap between a web application’s front-end and back-end.
The Rise of Serverless Backend Platforms
The space of serverless backends is growing rapidly due to developer’s demand. One could argue that as a developer building an application, the main goal is to deliver a high-quality product that solves a problem. This raises a question – do users really care what runs the app on the backend, or do they care more about the utility and frontend experience?
Offloading the challenge of building a backend to a serverless platform can make development faster and more efficient. Both established and upcoming modern backend platforms can provide not only a database – they’re often full-suite of products that can handle the most common application needs from user authentication to data analytics.
The most well known example would be Firebase – an app development platform founded in 2012 which is now part of Google. Firebase provides real-time databases, document store (Firestore), authentication and a variety of tools that are useful for development of web and mobile applications.
Google is not the only provider in this market. Some of the most popular databases are also available in a serverless fashion. Those include MongoDB (Atlas Serverless), MySQL (PlanetScale), PostgresQL (AWS Aurora) and Redis (Uptash).
While the space has a lot of established players, it’s still developing rapidly. Many new innovative startups, like Supabase or Convex are entering to bring new functionality, improved developer experience as well as various functionalities that allow developers to do more in less time.
Convex: A perfect Real-time Backend companion for a Reactive App in 2023
Convex is a full-stack, real-time development platform using 100% TypeScript that has been built with modern reactive UI frameworks like React in mind. The team behind Convex has built and maintained some of the most popular apps, backends and exabyte-scale storage projects at companies like Dropbox and Google.
At the core of Convex lies Reactor – an innovative database that can be accessed with TypeScript cloud functions as its interface. It tracks all dependencies for every query function and whenever any of them updates, Convex reruns the query and notifies any active subscription about the changes.
The above mentioned data is stored not only on the server side. Earlier in the article, we looked at the challenges of managing cache in frontend applications and how much additional work has to be done by developers to ensure that the fresh data is rendered without too much delay.
Querying data in Convex has two awesome attributes that resolve it – caching and reactivity. Clients of the application can subscribe to queries and receive new results whenever the underlying data changes. As the client is notified immediately, the cache can be updated at the exact right spot in time when it needs to be. It’s like having an always ready state that is automatically managed and ensures immediate and fresh data availability, thereby significantly enhancing user experience and development efficiency.
Unlike Firebase, Convex uses relational data models to store application’s data while keeping the flexibility to store documents in semi-structured JSON files. The database is 100% ACID compliant and data is accessed through TypeScript backend functions which eliminates the need to write complicated and tedious security rules.
Convex provides a generous free quota for developers who want to try it out for their next project. Get started with Convex here.
Closing words
While the landscape of web development continually shifts and evolves, it’s clear that React’s ethos of reactivity has penetrated deeper than the frontend. The technologies supporting web applications are steadily advancing towards providing full-stack solutions that are efficient and intuitive. This helps developers save time and to focus on the main challenge they have – building great applications. New platforms like Convex are pioneering this movement, offering a seamless developer experience that aligns the backend’s reactivity with React’s principles.
Full disclosure: Convex is a sponsor of Software Engineering Daily.
Cloud applications continue to grow in popularity, but ensuring the security of these applications often presents a formidable engineering challenge. This challenge motivated the creation of Jit. Jit is a continuous security platform for developers, and seeks to enable every cloud app to start with minimum viable security, or MVS, without slowing development velocity.
David Melamed is the Co-founder and CTO of Jit and he joins us in the episode to talk about his platform.
Jeff is a DevSecOps engineer with experience in security, the software development life cycle, and cloud technologies. His advanced expertise in HashiCorp technologies places him as one of the most sought after trainers in the Europe, Middle East, and Africa regions and beyond. Jeff has worked for a range of different companies – from small startups to some of the biggest financial institutions. He now successfully runs his own consultancy that provides services in DevSecOps, Cloud and Security. You can find Jeff at hemmen.lu.
ScyllaDB is a fast and highly scalable NoSQL database designed to provide predictable performance at a massive cloud scale. It can handle millions of operations per second at a scale of gigabytes or petabytes. It’s also designed to be compatible with Cassandra and DynamoDB APIs. Scylla is used by Zillow, Comcast, and for Discord’s 350M+ users, and it continues to gain popularity in other cloud-native and highly scalable cloud-centric workloads.
Dor Laor is the Co-founder and CEO of Scylla, and he’s our guest today.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
React is an immensely popular JavaScript library that is used to build website user interfaces. A key feature of React is that it uses a virtual Document Object Model, or DOM, to selectively update the desired regions of the web page, which provides major performance advantages. Million.js is an open source project that provides an optimized virtual DOM. Remarkably, these optimizations make React up to 70% faster and the code weighs in at less than 4 kilobytes in size.
Aiden Bai is the creator of Million.js and he joins us in this episode.
Mike Bifulco is CTO and co-founder of Craftwork. He’s also a developer advocate, writer, podcaster and serial startup founder. In past lives, Mike worked for Google, Stripe, Microsoft, and Gymnasium. Mike is also co-founder of APIs You Won’t Hate, a community for API Developers on the web. Mike’s publishes a weekly newsletter for product builders called Tiny Improvements at mikebifulco.com. Mike is on Mastodon at https://hachyderm.io/@irreverentmike
Cox Automotive has been at the frontier of digitizing the automotive space with popular products such as AutoTrader, Kelly Blue Book, and Dealertrack. To deliver these products the company manages large quantities of data and diverse engineering teams. Scaling its operations required an engineering transformation of the company.
Chris Dillon is the VP of Architecture and Engineering Enablement at Cox Automotive. He joins the podcast to talk about how the company accomplished this transformation.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
Serverless backend platforms are cloud services that simplify the process of building a backend. These platforms are growing rapidly in popularity because they can greatly accelerate application development, and improve the developer experience. Convex is a real-time backend platform that uses 100% TypeScript and is designed with reactive UI frameworks in mind. The team behind Convex has built and maintained some of the most popular apps, backends and exabyte-scale storage projects at companies like Dropbox and Google.
James Cowling is the Co-Founder and CTO at Convex, and he joins us to discuss how Convex offers a simpler full-stack developer experience.
Full disclosure: Convex is a sponsor of Software Engineering Daily.
Jeff is a DevSecOps engineer with experience in security, the software development life cycle, and cloud technologies. His advanced expertise in HashiCorp technologies places him as one of the most sought after trainers in the Europe, Middle East, and Africa regions and beyond. Jeff has worked for a range of different companies – from small startups to some of the biggest financial institutions. He now successfully runs his own consultancy that provides services in DevSecOps, Cloud and Security. You can find Jeff at hemmen.lu.
Exafunction is a leader in deploying deep learning models at scale. One of their products is Codeium, a coding assistant for software developers based on Exafunction’s deep learning technology. Codeium provides AI-assisted autocompletion in your IDE, making it easier for you to incorporate deep learning technology in your software development workflow.
Varun Mohan is the CEO and Co-founder of Codeium, and he is our guest today.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
Tom Preston-Werner is a renowned software developer, inventor and entrepreneur. He co-founded GitHub and is the creator of the avatar service Gravatar, the TOML configuration file format, and the static site generator software Jekyll. Tom is currently working on the full-stack web framework, RedwoodJS. He joins us today to tell us the latest about RedwoodJS, including its new support for server-side rendering. RedwoodJS conference
Paweł is the founder at flat.sociaI the world’s first ‘flatverse’ start-up. Pawel’s background is as a full-stack software engineer with a lean and experimental approach towards product development. With a strong grounding in computing science, he spent the last decade getting early-stage products off the ground – both in startup and corporate settings. Follow Paweł on Twitter, LinkedIn and his personal website – pawel.io.
Incident management is the process of managing and resolving unexpected disruptions or issues in software systems, especially those that are customer-facing or critical to business operations. Implementing a robust incident management system is often a key challenge in technical environments. Rootly is a platform to handle incident management directly from Slack, and is used by hundreds of leading companies including Canva, Grammarly, and Cisco.
Ashley Sawatsky leads Developer Relations at Rootly and previously led Shopify’s Incident Communications team. Niall Murphy is Co-founder and CEO at Stanza. He has written extensively about reliability engineering and is the co-author of the best-selling book, Site Reliability Engineering. Ashley and Niall join us in this episode to discuss how engineers and their non-technical counterparts can successfully approach incident management together.
Jeff is a DevSecOps engineer with experience in security, the software development life cycle, and cloud technologies. His advanced expertise in HashiCorp technologies places him as one of the most sought after trainers in the Europe, Middle East, and Africa regions and beyond. Jeff has worked for a range of different companies – from small startups to some of the biggest financial institutions. He now successfully runs his own consultancy that provides services in DevSecOps, Cloud and Security. You can find Jeff at hemmen.lu.
AI-assisted software delivery refers to the utilization of artificial intelligence to assist, enhance, or automate various phases of the software development lifecycle. AI can be used in numerous aspects of software development, from requirements gathering to code generation to testing and monitoring. The overarching aim is to streamline software delivery, reduce errors and, ideally, reduce the time and costs associated with software development.
Birgitta Böckeler is the Global Lead for AI-assisted Software Delivery at Thoughtworks and she joins us in this episode. We discuss how the latest advances in large language models are revolutionizing software development.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm.
The Buy Now, Pay Later model, or BNPL, is traditionally a business-to-consumer model that is gaining traction in the business-to-business domain. Adoption of BNPL in business-to-business scenarios offers benefits like improved cash flow management for buyers, and increased sales for suppliers, often leading to higher order value.
However, this model introduces complexities, including the need for advanced technical platforms for managing such transactions. There are also risks associated with delayed or defaulted payments, and potential accumulation of interest and fees.
As business-to-business entities consider BNPL, they must strike a balance between its potential advantages and its inherent risks and operational challenges.
Tranch is a London-based startup that is focused on providing a payments-checkout platform for businesses that want to pay easier and get paid faster.
Sohil Pandya is the Head of Engineering at Tranch and he joins us in this episode.
Paweł is the founder at flat.sociaI the world’s first ‘flatverse’ start-up. Pawel’s background is as a full-stack software engineer with a lean and experimental approach towards product development. With a strong grounding in computing science, he spent the last decade getting early-stage products off the ground – both in startup and corporate settings. Follow Paweł on Twitter, LinkedIn and his personal website – pawel.io.
Database caching is a fundamental challenge in database management and there are hundreds of techniques to satisfy different caching scenarios.
PolyScale is a fully automated database cache. It offers an innovative approach to database caching, leveraging AI and automated configuration to simplify the process of determining what should and should not be cached. Ben Hagan is the founder and CEO of PolyScale and he is our guest today.
Full disclosure: PolyScale is a sponsor of Software Engineering Daily.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
In memory of Software Engineering Daily Founder, Jeff Meyerson.
1988 – 2022
Jeff founded Software Engineering Daily in 2015 and hosted the podcast until 2022. He was willful, hard working, and a strong advocate for self-direction.
If you’d like to listen to some of his work beyond the podcast, his You Are Not A Commodity keynote from 2019 gives a glimpse into his philosophy, and his music is available on Spotify.
In the realm of modern software development, clear and comprehensive documentation is essential for guiding users and contributors alike. Developers often struggle to create, manage, and maintain documentation that is both user-friendly and visually appealing.
In this episode, we sit down with Sebastian Lorber, the lead maintainer of Docusaurus, a widely acclaimed open-source static site generator, purpose-built for documentation sites. Sebastian sheds light on how Docusaurus empowers developers to effortlessly build and deploy documentation websites that are functional, maintainable, and easy to brand.
Mike Bifulco is CTO and co-founder of Craftwork. He’s also a developer advocate, writer, podcaster and serial startup founder. In past lives, Mike worked for Google, Stripe, Microsoft, and Gymnasium. Mike is also co-founder of APIs You Won’t Hate, a community for API Developers on the web. Mike’s publishes a weekly newsletter for product builders called Tiny Improvements at mikebifulco.com. Mike is on Mastodon at https://hachyderm.io/@irreverentmike
APIs are ubiquitous and critical to building modern software, and developers must frequently develop custom APIs to streamline user access to their services.
However, making an API that provides a great developer experience can be a time-consuming endeavor. As a result, API teams often leave the final mile of integration up to their users.
Speakeasy aims to provide a product and a pipeline that makes it easy to create and consume any API.
Sagar Batchu is the co-founder and CEO of Speakeasy and he joins us in this episode.
Paweł is the founder at flat.social the world’s first ‘flatverse’ start-up. Pawel’s background is as a full-stack software engineer with a lean and experimental approach towards product development. With a strong grounding in computing science, he spent the last decade getting early-stage products off the ground – both in startup and corporate settings.
Generative pre-trained transformer models, or GPT models, have countless applications and are being rapidly deployed across a wide range of domains.
However, using GPT models without appropriate safeguards can lead to leakage of sensitive data. This concern underscores the critical need for privacy and data protection.
Skyflow LLM Privacy Vault prevents sensitive data from reaching GPTs. Amruta Moktali is the Chief Product Officer at Skyflow and she joins us today. We discuss generative AI, how the technology is different from other AI approaches, and how we can use this technology in a safe and ethical manner.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
The CAP theorem, also known as Brewer’s theorem, is a fundamental principle in distributed systems that states that it is impossible to simultaneously achieve three desirable properties in a distributed data system: Consistency, Availability, and Partition tolerance.
Eric Brewer is the VP of Infrastructure & Google Fellow at Google and he joins us today.
This Interview was recorded at Open Source Summit North America and wouldn’t have been possible without the generous support from
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
Companies have high hopes for Machine learning and AI to support real-time product offerings, prevent fraud and drive innovation. But there was a catch – training models require labeled data that machines can digest. As data volumes increase, the opportunity to get great ML results rises, but so does the problem of labeling all the data to get that excellent result.
Enter Snorkel AI’s programmatic data labeling and MLops platforms like Snorkel Flow. Today we are interviewing Alex Ratner, one of the founders of Snorkel AI. Snorkel AI evolved from research Alex led as part of his Ph.D. research at Stanford, focused on programmatic data labeling to enable much faster and more accurate ML training and retraining.
Alex is a born teacher who always has enthusiasm for the topic. Today he will share the newest evolutions of the product at Snorkel, shed light on why doing ML well requires programmatic data labeling, and talk about foundation models in actual enterprise settings and generally.
Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.
Elasticsearch is the most established solution today to search and analyze large amounts of logs. However, it can be costly and complex to manage. Quickwit searches large amounts of append only cloud data like logs or ledgers in a fraction time with significantly less cost than Elasticsearch. In this episode, we interview Paul Masurel, one of the founders of Quickwit and the creator and maintainer of Tantivy, the open source full text search engine written in Rust.
Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.
As Companies scale and their codebase becomes large, it can become difficult to measure team effectiveness. With so many moving pieces, and an increased lack of visibility, Engineering Leaders mostly rely on gut feelings to guide decisions and determine impact. Uplevel pulls meta-data from programs like Git, Slack, Jira and public calendar events and uses machine learning and organizational science to generate actionable insights. With Uplevel Engineering teams can increase velocity, decrease burnout and plan and execute more predictable releases. Christina Forney is the VP of Product at Uplevel and she joins us in this episode.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
RudderStack is a warehouse-native customer data platform (CDP) that helps businesses collect, unify, and activate customer data from all their different sources.
In today’s episode, we’re talking to Soumyadeb Mitra, the founder and CEO of RudderStack.
We discuss the importance of activating all your data, how RudderStack can help you activate your data, the challenges of integrating different data sources and how to build a data-driven culture in your organization.
Full disclosure: RudderStack is a sponsor of Software Engineering Daily.
Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.
Hyperscalers refer to expansive cloud service providers capable of delivering enterprise-scale computing and storage services. These Hyperscalers like Google, Amazon, Facebook that have huge data centers and are either running their own software or renting out this infrastructure realized a long time back that the traditional Network, Storage & Compute server racks were not enough to fulfill the requirements of a modern enterprise at scale. So they built custom solutions for their data center requirements.
For all of the other companies and organizations that have their own data centers, they are still running legacy old technology. Oxide Computer is a company aiming to democratize access to rack-scale technology and provide a seamless software stack to facilitate its implementation.
Bryan Cantrill is the Co-Founder and CTO of Oxide Computer and he joins us in this episode.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
Available as a cloud service, Red Hat Trusted Software Supply Chain provides a DevSecOps framework to create applications more securely. Vincent Danen is the VP of Product Security at Red Hat and joins us in this episode.
Red Hat has been a Secure Open Source software provider for very long. We discuss how the Red Hat Trusted Software Supply Chain product allows enterprises to adopt DevSecOps practices successfully, safely consume open-source code and third-party dependencies, and build security into the software development life cycle.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
The state of Data inside most companies is chaotic. It takes significant time and investment to tame this chaos. When you are a platform provider you are gathering tons of data from the developers using your platform. These developers building products on your platform need insight into that data to better understand how their application is performing or to troubleshoot it. Most Platforms or SaaS application providers find it both difficult and expensive to build customer-facing analytics and data applications into their platforms. In fact most companies don’t know what to do with the data they are gathering and continually postpone future product roadmap features aimed to unlock this data. This data can be a crucial part of the developer experience and can empower your customers. It can save you countless hours of handling support tickets, and increase overall stickiness on the platform.
Propel is a GraphQL API platform ideal for powering customer-facing analytics use cases, from customer dashboards and analytics APIs to product usage or in-product metrics. Tyler Wells is Co-founder and CTO at Propel and he joins us today. We discuss how the customer-centric experiences at Twilio lead his team to the journey they are on today.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
The Internet of Things (IoT) has the potential to revolutionize many industries, but its full potential has yet to be realized. One reason is that the focus has been on the sensors rather than the data. IoT devices collect data from various sources, including machinery, customer behavior, and environmental conditions. This data can be used to improve efficiency, optimize processes, and make better decisions. However, collecting and processing this data can take time and effort. Businesses need to invest in specialized platforms and processes to make the most of IoT data. These platforms can collect, process, and analyze data in real-time, providing businesses with the insights they need to make better decisions.
Tangible benefits for real-time business objectives
Integrating and combining sensor data from multiple streams and sources multiplies that benefit through:
Optimized operations, system monitoring, and predictive maintenance. Sensors can help identify patterns, find anomalies, and suggest real-time changes that save money, prevent failure, and keep customers happy.
Personalized experiences and products. Customers who become more sophisticated and connected expect real-time, personalized products and marketing messages. Sensors offer a view into actual customer behavior by observing what people do in the real world, not relying on what people say they want.
Real-time decision-making. From getting a loan to a dinner reservation, users and businesses want to get and make decisions immediately based on the best available and freshest data. Real-time decisions that incorporate sensor data are more accurate and lower risk.
Extracting the full value of sensor data
Data collection. Sensors gather data from their environments generating raw data streams based on various parameters such as system health, temperature, pressure, or location. These raw streams contain raw data, which can be messy, including some duplicates, and overwrites, and each collection stream will have its own description of the data in the stream that computers and people need to analyze and move that data.
Data transmission. Significantly, IoT sensors rely on wireless protocols such as Bluetooth, Wi-Fi, or cellular networks to send data to a central system hosted by a company, to cloud data platforms like Snowflake or Databricks, or cloud platforms like AWS, Azure, or GCP. Wireless protocols for data transfer are more reliable and secure than ever but still suffer from drops and transmission gaps that can cause delays and inaccuracy as data is sent to its destination for processing.
Data ingestion. Central systems onboard the raw stream and prepare it for downstream consumption by people and other machines. Often overlooked, this process is another unique source of delay and possible inaccuracy as raw data moves through gateways, into central systems, and through baseline data quality checks.
Data processing and analysis. Streaming sensor data finally flows to analysts who can extract insights in real-time using modern analytics platforms for streaming. Often, at this stage, the process can include integrating third-party or batch data. That can be challenging as each data source has different descriptions and schemas, making apples-to-apples analysis and integration hard.
Data Storage and visualization. Processed data is stored and available for data consumers across the enterprise. Dashboards and reporting tools deliver insights and allow users to monitor trends, fix issues and expand capabilities. These tools constantly change as users discover new data needs and the underlying data changes.
Pillars of IoT Analytics Platforms
IoT data has special requirements to ensure the data and resulting analytics are timely, trusted, and consistent.
Speed and performance – Sensor data needs to be fast and trusted. Processing and delivering sensor data is unlike any other data analytics activity and has particular latency concerns. Distributed cloud computing, highly distributed data collection over wireless networks, and multiple hops inside the enterprise can delay data arrival and erode trust.
Interoperability – The power of sensor data comes from combining streams and adding context. When all the streams and third-party contextual data have different data definitions and schemas, humans often have to hand-tune and connect the correct data for the complete analysis. Making it easy to add, combine and review many data sources is essential to using sensor data.
Scale – Sensor processing platforms must address data scale at a level no other data function can imagine. The velocity and volume of data are often discussed, and nowhere is it more visible than in the arena of IoT. In addition to specialized hardware and cloud computing, sensor data networks require highly specialized architectural and software solutions to support real-time outcomes.
Flexibility – The often forgotten but essential player in the workflow from data collection to insight is the human and the user experience. Sensor data that is quickly moved, combined, and delivered will not have an impact without giving humans the dashboards and analytic tools they want and need to extract value.
Datorios has a Solution for Real-Time Sensor Data Processing
The shift to real-time data sources is across all domains and will grow even faster with the availability of Generative AI. That growing demand is amplified by the economic need for higher productivity and expense reduction. Datorios, a leading provider of real-time sensor data processing solutions, has announced the launch of its new solution for sensor events, transactions and IOT data.
The Datorios serverless-like solution is built on a cloud-native architecture designed to scale elastically with demand. Businesses can easily add or remove capacity without worrying about infrastructure management.
The solution also includes many features that make building and deploying real-time sensor data processing applications easy. Buyers and implementers should consider these features as they evaluate Datorios:
Combination of declarative code and visual layers both for the pipeline specifications and the data itself.
Built-in streaming engine that can process data from millions of devices in real-time.
Unique interactions between the pipeline logic and actual real-time events for the acceleration of development and debugging.
Comprehensive set of analytics and visualization tools that can be used to gain insights from sensor data.
The Datorios serverless solution helps businesses process massive volumes of sensor data in real time. The solution includes a comprehensive set of features that make it ideal for a wide range of applications, with sensor data as the key to the first use cases.
Business Benefits of the Datorios Serverless-like Solution
Time to market – By changing the way code interacts with data in real-time, Datorios can significantly accelerate the development cycle for data and change the typical delivery time from weeks to hours.
Free up the experts, save time on DevOps – Customers report and detailed case studies document up to 70% less time spent on integration and testing code reviews and documentation. That is a plus for data experts and modelers who want to spend most of their time on module development.
Save money with elastic scale – The solution is designed to scale elastically with demand, so businesses can easily add or remove capacity as needed. This can help businesses save money on infrastructure costs.
Performance delivers the speed customers expect – The solution is designed to process massive volumes of data in real-time. This can help businesses gain insights from data faster.
Trust and security – The solution is designed to be secure, so businesses can be confident that their data is safe.
Cost reduction through saved people time and architectural design – Datorios architecture is designed to reduce costs through pre-processing of high data capacity functions, moving data quality resolution closer to the source of any issues and compute optimization in a specialized event-based architecture.
There are numerous challenges encountered in processing sensor data. From grappling with the sheer volume and variety of data to ensuring its accuracy and reliability, these hurdles demand innovative solutions. These solutions must address the complexities of real-time data streaming, the necessity of robust data storage and management systems, and the intricacies of integrating diverse sensor networks.
As industries increasingly rely on sensor-generated data, it is imperative to devise comprehensive strategies and leverage cutting-edge technologies to surmount these challenges. The Datorios solution is one to consider for the scalability, ease of use, performance, and security sensor data demands. By doing so, companies can unleash the full potential of sensor data and pave the way for transformative advancements in various fields.
Full disclosure: Datorios is a sponsor of Software Engineering Daily.
The company Continue has a new take on the resale marketplace. By partnering with brands and pulling post-purchase data onto their platform, this innovative solution simplifies and enhances the resale process, empowering consumers to participate in a more sustainable and circular economy. Richie Ganney and Harry Riley are the Co-founders of Continue and they join us in this episode. We uncover how this startup is revolutionizing the way we consume and recycle products.
Jeff is a DevSecOps engineer with strong experience in Security, The Software Development Life Cycle and Cloud Technologies. His advanced expertise in HashiCorp technologies places him as one of the most sought after trainers in the EMEA region and beyond. Jeff has worked for a range of different companies – from small startups to some of the biggest financial institutions. He now successfully runs his own consultancy that provides services in DevSecOps, Cloud and Security. Check it out on hemmen.LU and follow Jeff on LinkedIn @jeffhemmen
CRaC (Coordinated Restore at Checkpoint) is a new technology that can improve startup and warmup times by orders of magnitude. It is a project of OpenJDK that was proposed and led by Azul. The CRaC Project defines public Java APIs that allow for the coordination of resources during checkpoint and restore operations. With CRaC, a checkpoint can be set at any point where an application can be safely paused. Azul is releasing a reference implementation of CRaC in JDK 17 with the Azul Zulu Build of OpenJDK for x86 64-bit Linux update. CRaC allows for an “instant” start at any point in the application lifecycle at an optimal speed. It also work’s really well with Azul’s ReadyNow! feature that optimizes warm-up. For the first time, developers will have a TCK-tested, CRaC-configured, production-ready JVM with commercial support available for their use.
Gerrit Grunwald is a Java Champion & Principle Engineer at Azul and he joins us in this episode.
Full disclosure: Azul is a sponsor of Software Engineering Daily.
Alex is an AWS Data Hero, an independent consultant, and the author of The DynamoDB Book, the comprehensive guide to data modeling with DynamoDB. He was an early employee at Serverless, Inc., creators of the Serverless Framework, and was an early community member in the serverless space. His consulting and training work focuses on serverless architectures and database optimization. You can find him on Twitter as @alexbdebrie or on his site, alexdebrie.com.
WebAssembly is a low-level binary format for the web that is compiled from other languages to offer maximized performance and is meant to augment the places where JavaScript isn’t sufficient. High Performance applications like AutoCAD, Figma and Photoshop are now leveraging the capabilities of WebAssembly to provide native experiences on the web. Matt Butcher is the CEO at Fermyon Technologies and he joins us in this episode.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
In this episode, we are talking to Robert Cooke, founder and CTO of 3forge. He has spent the last decade creating a full stack software platform that revolutionized enterprise real-time data management, visualization, and workflows through its inventive “high impact code” concept. With offices in New York, London, and Singapore, 3forge has been serving a global clientele including Tier 1 banks and large financial institutions. He talks about his life long passion for computing, the challenges of creating a complete high performance platform, and the scaling principles needed to exceed the requirements of the most demanding global firms.
Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.
Usage-Based Pricing is becoming more and more popular. Led by the wild popularity of cloud service providers such as AWS, customers are demanding more visibility into usage information and more accountability for the dollars they spend on a service. But historically service oriented products have shied away from usage based pricing because of the complexity involved in managing the data involved. Amberflo is a service that provides a platform for building and deploying usage based pricing. Puneet Gupta is the CEO and Founder of Amberflo and he is our guest today.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
Feature flags also known as feature toggles, release toggles or feature flippers are a way to enable or disable a particular feature from your app without making any changes to the source code. You can turn on or off a particular functionality without deploying new code. Feature flags can also be used to serve different features to different subset of users.
The company Flagsmith provides you a platform for developing, implementing and managing your feature flags. Ben Rometsch is the Co-founder and the CEO of Flagsmith and he joins us today.
As companies depend more on data to improve digital products and make informed decisions, it’s crucial that the data they use be accurate and reliable. MonteCarlo, the data reliability company, is the creator of the industry’s first end-to-end data observability platform.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
Programming robotics software has traditionally been a specialized field. The software industry has seen rapid progress, the operating system that provides the foundation for our software applications is taken care of by companies like Google, Microsoft, and other players like Canonical, Amazon, etc. The robotics industry still needs that OS layer that handles the complexities so that engineers can build robots that serve their business needs. VIAM aspires to be the Operating System for robotics that makes it easy to turn great ideas into production robots.
Eliot Horowitz is the Founder and CEO of VIAM and he joins us today. Eliot was previously the CTO at MongoDB, a database company. We discuss with him the different challenges one faces when trying to build a robot and how VIAM is trying to solve that.
Paweł is the founder at flat.sociaI the world’s first ‘flatverse’ start-up. Pawel’s background is as a full-stack software engineer with a lean and experimental approach towards product development. With a strong grounding in computing science, he spent the last decade getting early-stage products off the ground – both in startup and corporate settings.
Mach5 Search is a slide-in, cloud-native replacement for Elasticsearch and OpenSearch that immediately saves up to 90% in operating cost. Mach5 Search can run on top of Google BigQuery, Snowflake, and Databricks, or natively on Object Stores in all the major clouds. Vinayak Borkar is the CEO and Co-Founder of Mach5 Software and he joins us today.
Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.
Metrist is an observability platform designed to commoditize application observability. Jeff Martens is the CEO of Metrist, and he joins us today.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
In this podcast episode, we take a look at the intricacies of low-code data pipelines with Raj Bains, the founder of Prophecy.io. Raj shares valuable insights into how performant low-codedata pipelines are revolutionizing industries and transforming everyday operations. Raj discusses the founding story of Prophecy.io, the company’s mission, and its approach to democratizing the creation of efficient data pipeline solutions visual design and code generation. We also discuss technical concepts/conundrums such as data lineage, schema evolution, and metadata management, which are critical in addressing the challenges faced by data pipeline developers and businesses. The episode concludes with Raj’s thoughts on the future of low-code data pipelines, the Prophecy.io roadmap and its potential impact on various industries, from healthcare to finance.
Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.
Crossplane is an innovative open source control plane framework that helps companies provide managed access to cloud native control planes. Upbound provides a single global platform to build, deploy, and operate these internally managed control planes that are powered by cross plane.
Bassam Tabbara is the CEO of Upbound, and he joins us today.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
When your application fails, finding the reason quickly is essential for limiting downtime. Often, most of the time it takes to repair a problem is taken by figuring out what exactly went wrong. PromptOps is a service designed to make tracing the answer to why something happened back to the original cause.
Dev Nag is the CEO of PromptOps, and he joins us today.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
In the Software Industry, Conferences are crucial for developers, offering learning, networking, and collaboration opportunities. Attendees stay updated on the latest technologies, gain new skills, and connect with peers. These events expose developers to industry trends, inspire innovation, and foster professional growth. Overall, Conferences play a vital role in advancing knowledge, building connections, and driving success in the developer community.
In this episode, we discuss what makes a great Conference and how Shift Conference became of the largest events in Europe. Ivan Burazin is the Chief Developer Experience Officer at Infobip and the CEO of Codeanywhere, and he joins us today.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
The software supply chain refers to the process of creating and distributing software products. This includes all of the steps involved in creating, testing, packaging, and delivering software to end-users or customers.
Socket is a new security company that can protect your most critical apps from supply chain attacks. They are taking an entirely new approach to one of the hardest problems in security in a stagnant part of the industry that has historically been obsessed with just reporting on known vulnerabilities.
Feross is the Founder and CEO of Socket Security, and he joins us today.
This Interview was recorded at Open Source Summit North America and wouldn’t have been possible without the generous support from
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
Chaos engineering is a discipline within the field of software engineering that focuses on testing and improving the resilience and stability of a system by intentionally introducing controlled instances of chaos and failure. The primary goal of chaos engineering is to identify and address potential weaknesses and vulnerabilities in a system, ultimately making it more robust and reliable. Uma Mukkara is the Head of Chaos Engineering at Harness and he joins us today.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
The CAP theorem, also known as Brewer’s theorem, is a fundamental principle in distributed systems that states that it is impossible to simultaneously achieve three desirable properties in a distributed data system: Consistency, Availability, and Partition tolerance.
Eric Brewer is the VP of Infrastructure & Google Fellow at Google and he joins us today.
This Interview was recorded at Open Source Summit North America and wouldn’t have been possible without the generous support from
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
Astasia is a Partner on Quiet Capital’s enterprise team. Prior to joining Quiet, she was an investor on Redpoint Venture’s early stage enterprise team. Astasia joins the show to talk about Early Stage Venture Capital.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.
Ian Coldwater is a DevSecOps engineer turned red teamer who specializes in breaking and hardening Kubernetes, containers, and cloud native infrastructure. In their spare time, they like to go on cross-country road trips, capture flags, and eat a lot of pie. Ian lives in Minneapolis and tweets as @IanColdwater.
Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.
These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.
Although it improves the management and scaling of infrastructure and applications, Kubernetes frequently has challenges managing the complexity of releasing applications.
Git is the most widely used version-control system in the software industry today. GitOps is a set of procedures that uses the power of Git to provide both revision and change control within the Kubernetes platform.
Flux is an open source project that implements GitOps-driven continuous deployment for Kubernetes.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
Bug reporting hasn’t changed since the 1990’s. Despite all the technological advancements we’ve made in the rest of software development, the way we handle bugs has stayed the same.
It is common practice for non technical teams to provide bug reports that are missing vital information for developers to identify and quickly fix code, such as: network requests, console logs, HAR file, device information, video or screenshot replication, and more.
Providing a comprehensive bug report requires logging into many tools and time spent context switching, creating an environment where people are less likely to report bugs, which is not an ideal scenario for any founder or builder.
Dani & Irtefa, co-founders at Jam.dev set out to build a tool that will enable anyone, no matter their technical background, to capture rich contextual technical data about bugs, so that developers can quickly identify and resolve issues.
Jam is a browser extension that will instantly replay a bug while automatically attaching diagnostics: network requests and speed, console logs, device, and more – all in one easy click. You don’t have to log into a new tool or disrupt workflows, as Jam has direct integrations with all the first class bug reporting tools to seamlessly drop diagnostics into a ticket.
In this episode, Dani & Irtefa share their experience building greenfield products for Cloudflare’s Emerging Technology and Incubation team, how bug reporting was the common pain-point that impacted their product velocity, and the technology decisions they made when building Jam.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
In this episode we have two guests Ramiro Berrelleza is the founder and CEO of Okteto, a developer tool that helps you create production like dev environments in the cloud for every developer. Ramiro will tell us how to easily make replica environments for your day to day development , but computing related infrastructure is only half the battle. It also helps to have realistic data when you’re developing. John Craft is the CTO at Privacy Dynamics, a company that helps you create anonymized data. You can use Privacy Dynamics to create realistic data from your production database without compromising your customer information.
Alex is an AWS Data Hero, an independent consultant, and the author of The DynamoDB Book, the comprehensive guide to data modeling with DynamoDB. He was an early employee at Serverless, Inc., creators of the Serverless Framework, and was an early community member in the serverless space. His consulting and training work focuses on serverless architectures and database optimization. You can find him on Twitter as @alexbdebrie or on his site, alexdebrie.com.
The cloud has become an all-encompassing platform for running diverse applications and enabling individuals and teams to add value by utilizing services and infrastructure that streamline the process of software building and operation. Nonetheless, the cloud has presented new hurdles for developers as it is intricate, and application development demands comprehension of cloud service intricacies. Moreover, creating portable applications across different cloud providers, as well as local testing and debugging, have proven to be difficult. Additionally, leaky abstractions and inadequate tooling have made it challenging to repurpose cloud architectures into reusable components.
To address these challenges, a cloud-centric programming language called Wing has been introduced. Wing enables developers to create distributed systems that fully capitalize on the cloud’s capabilities without the need to concern themselves with the underlying infrastructure.
Jeff is a DevSecOps engineer with strong experience in Security, The Software Development Life Cycle and Cloud Technologies. His advanced expertise in HashiCorp technologies places him as one of the most sought after trainers in the EMEA region and beyond. Jeff has worked for a range of different companies – from small startups to some of the biggest financial institutions. He now successfully runs his own consultancy that provides services in DevSecOps, Cloud and Security. Check it out on hemmen.LU and follow Jeff on LinkedIn @jeffhemmen
The goal of having a single pane of glass that allows us to see what is happening with our organization’s IT operations has been a long-standing goal for many organizations. The goal makes a lot of sense. Without a clear end-to-end picture, it is hard to determine where your problems are if you can’t determine whether something happening upstream is creating significant knock-on effects.
When we have these high-level views, we are, of course, aggregating and abstracting details. So the ability to drill into the detail from a single view is an inherent requirement. The problem comes when we have distributed our solutions across multiple data centers, cloud regions, or even regions with multiple vendors.
The core of the challenge is that our monitoring through logs, metrics, and traces accounts for a significant amount of data, particularly when it isn’t compressed. An application that is chatty with its logs or hasn’t tuned its logging configuration can easily generate more log content than the actual transactional data. The only reason we don’t notice it is that logs are generally not consolidated, and log data is purged.
When it comes to handling the monitoring in a distributed arrangement, if we want to consolidate our logs, we’re potentially egressing a lot of traffic from a data center or cloud provider, and that costs. Cloud providers typically don’t charge for inbound data, but depending upon the provider, it can be expensive for data egress; it can even cost to transmit data between regions with some providers. Even for private data centers, the cost exists in the form of bandwidth of connectivity to the internet backbone and/or the use of leased lines. The numbers can also vary around the world as well.
The following diagram provides some indicative figures from the last time I surveyed the published prices of the leading hyper scalers, and the on-premises costs are derived from leased line pricing.
This raises the question of how on earth do you create a centralized single pane of glass for your monitoring without risking potentially significant data costs. Where should I consolidate my data to? What does this mean if I use SaaS monitoring solutions such as DataDog?
There are several things we can do to improve the situation. Firstly, let’s look at the logs and traces being generated. They may help during development and testing, but do we need all of it? If we’re using logging frameworks, are the logs correctly classified as Trace, Debug, and so on? When logging frameworks are being used by applications, we can tune the logging configuration to deal with the situation when one module is particularly noisy. But for those systems that are brittle, people who are nervous about modifying any configuration or a 3rd party support organization will void any agreements if you modify any configuration. The following line of control is to take advantage of tools such as Fluentd, Logstash, or Fluentbit, which brings with it full support for OpenTelemetry. We can introduce these tools into the environment near the data source so that they can capture and filter the logs, traces, and metrics data.
The way these tools work means they can consume, transform and send logs, traces, and metrics to the final destination in a format that most systems can support. Further, Fluentd and Fluentbit can easily be deployed to fan out and fan in workloads – so scaling to sort out the data comprehensively can be done easily. We can also use them as a relay capability so we can funnel the data through specific points in a network for added security.
As you can see in the following diagram, we’re mixing Fluentd and Fluentbit to concentrate data flow before allowing it to egress. In doing so, we can reduce the number of points of network exposure to the internet. A strategy that shouldn’t be used as the only mechanism to secure data transmission, but certainly one that can be part of an arsenal of security considerations. It can also be used as a point of failsafe in the event of connectivity issues.
As well as filtering and channeling the data flow, these tools can also direct data to multiple destinations. So rather than throwing away data that we don’t want centrally, we can consolidate the data into an efficient time-series data store within the same data center/cloud and send on the data that has been identified as high value. This then gives us two options; in the event of investigating an issue, we can do a couple of things:
Identify the additional data needed to enrich the central aggregated analysis and ingest just that additional data (and possibly further refine the filtration for the future) needed.
Implement localized analysis and incorporate the resultant views into our dashboards.
Either way, you have access to additional information. I would opt for the former. I’ve seen situations where the local data stores have been purged too quickly by local operational teams, and data like traces and logs compress well in greater volume. But remember, if the logs include data that may be sensitive to location, pulling them to the center can raise additional challenges.
While in the diagram, we’ve shown the monitoring center to be on-premise, this could equally be a SaaS product or one of the clouds. The key to where the center is comes down to three key criteria:
Any data constraints in terms of the ISO 27001 view of security (integrity, confidentiality, and availability).
Connectivity and connectivity costs. This will tend to bias the location for monitoring to where the largest volume of monitoring data is generated.
Monitoring capability and capacity – both functional (visualize and analyze data) and non-functional factors, such as how quickly inbound monitoring data can be ingested and processed.
Adopting a GitOps strategy to help ensure that we have consistency in configuration and, therefore, data flow from software that may well be deployed across data centers or cloud regions and possibly even multiple cloud vendors can be kept consistent because the monitoring sources are consistent in configuration If we identify changes to the filters (to remove or include) data coming to the center.
Incidentally, most stores of log data, be that compressed flat files, databases can be processed with tools like Fluentd not only as a data sink but also as a data source. So it is possible through GitOps to distribute out temporary configurations for your Fluentd/Fluentbit nodes which can harvest and bulk move any newly required data for the center from those regionalized staging stores rather than manually accessing and searching them. But if you adopt this approach, we recommend creating templates for such actions in advance and use as part of a tested operational process. If such a strategy were to be adopted at short notice as part of a problem remediation activity, you could accidentally try and harvest too much data or impact current active operations. It needs to be done with awareness about how it can impact what is live.
Hopefully, this will help offer some inspiration for cost-efficiently handling hybrid and multi-cloud operational monitoring.
Chroma is an open source embedding database that is designed to make it easy to build large language model applications by making knowledge, facts and skills pluggable.
Anton Troynikov is the co-founder of Chroma and he is our guest today.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
The cloud has become an all-encompassing platform for running diverse applications and enabling individuals and teams to add value by utilizing services and infrastructure that streamline the process of software building and operation. Nonetheless, the cloud has presented new hurdles for developers as it is intricate, and application development demands comprehension of cloud service intricacies. Moreover, creating portable applications across different cloud providers, as well as local testing and debugging, have proven to be difficult. Additionally, leaky abstractions and inadequate tooling have made it challenging to repurpose cloud architectures into reusable components.
To address these challenges, a cloud-centric programming language called Wing has been introduced. Wing enables developers to create distributed systems that fully capitalize on the cloud’s capabilities without the need to concern themselves with the underlying infrastructure.
Jeff is a DevSecOps engineer with strong experience in Security, The Software Development Life Cycle and Cloud Technologies. His advanced expertise in HashiCorp technologies places him as one of the most sought after trainers in the EMEA region and beyond. Jeff has worked for a range of different companies – from small startups to some of the biggest financial institutions. He now successfully runs his own consultancy that provides services in DevSecOps, Cloud and Security. Check it out on hemmen.LU and follow Jeff on LinkedIn @jeffhemmen
Code editors are complex and demanding pieces of software that are both essential tools and constant companions for developers. Today, we are interviewing Nathan Sobo to hear about Zed, a code editor focused on “multiplayer” experiences, performance and a streamlined, minimalist design.
Nathan Sobo was a member of the Atom editor team at GitHub, which worked on the (now-deprecated) Atom code editor. He will share his thoughts on Rust, the impact of very low-latency collaborative coding on happiness, and how to turn an authoring environment into a first-class concern.
Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on Linkedinor Twitter @jocelynbyrne
Most companies aren’t in the building business yet they need to deal with pricing and billing as a core piece of technology whether they want to or not. Stigg is an easy to implement headless pricing and packaging platform that takes a lot of the hassle out of pricing and billing for your SaaS application. Their APIs help developers launch new pricing plans faster that lead to better customer buying experiences.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
Finout helps FinOps, DevOps, and Finance to manage & reduce cloud spend and improve the company’s profitability without adding code or changing existing tags.
Roi Ravhon is the Co-founder and CEO at Finout and he joins us today.
Full disclosure: Finout is a sponsor of Software Engineering Daily.
Alex is an AWS Data Hero, an independent consultant, and the author of The DynamoDB Book, the comprehensive guide to data modeling with DynamoDB. He was an early employee at Serverless, Inc., creators of the Serverless Framework, and was an early community member in the serverless space. His consulting and training work focuses on serverless architectures and database optimization. You can find him on Twitter as @alexbdebrie or on his site, alexdebrie.com.
Data Activation is the method of unlocking the knowledge sorted within your data warehouse, and making it actionable by your business users in the end tools that they use every day. In doing so, Data Activation helps bring data people toward the center of the business, directly tying their work to business outcomes.
Hightouch is the simplest and fastest way to get started with Data Activation. As a Data Activation Platform, Hightouch uses Reverse ETL to sync data from the warehouse to 100+ different integrations. With Hightouch companies can leverage their existing data models and easily view and monitor all of their data syncs in a single platform. Better yet, Hightouch offers a visual audience builder that makes it easy for non-technical users to create custom audiences at moment’s notice.
Tejas Manohar is the CEO at Hightouch and he joins us today.
Full disclosure: Hightouch is a sponsor of Software Engineering Daily.
Alex is an AWS Data Hero, an independent consultant, and the author of The DynamoDB Book, the comprehensive guide to data modeling with DynamoDB. He was an early employee at Serverless, Inc., creators of the Serverless Framework, and was an early community member in the serverless space. His consulting and training work focuses on serverless architectures and database optimization. You can find him on Twitter as @alexbdebrie or on his site, alexdebrie.com.
Enabling data engineers to create data pipelines easily while delivering data streams that meet low-latency, production requirements is a difficult balancing act. David Yaffe and Johnny Gaettinger join us today to share how they have created that balance at Estuary.
Estuary is a data operations platform that synchronizes data across the systems where data lives and delivers it where you want it to live with sub second low latency. Dave and Johnny will share the technical choices that support sub second performance, use cases for batch and streaming data applications and some of their business perspective as an early stage start-up.
Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on Linkedinor Twitter @jocelynbyrne
A data catalog provides an index into the data sets and schemas of a company.Data teams are growing in size, and more companies than ever have a data team, so the market for data catalog is larger than ever.
Mark is the CEO of Stemma and the co-creator of Amundsen, a data catalog that came out of Lyft. In today’s show Mark shares how his history as a software developer, data engineer and product leader at Lyft have helped him create a vision for a modern data catalog.
Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securit.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on Linkedinor Twitter @jocelynbyrne
Streaming analytics refers to the process of analyzing real-time data that is generated continuously and rapidly from various sources, such as sensors, applications, social media, and other internet-connected devices. Streaming analytics platforms enable organizations to extract business value from data in motion, similar to how traditional analytics tools derive insights from data at rest. DeltaStream is a unified serverless stream processing platform to manage, secure and process all your event streams and is based on Apache Flink. Hojjat Jafarpour is the Founder and CEO at DeltaStream and he joins us today. Before joining DeltaStream, Hojjat was at Confluent, the company behind Apache Kafka, he built a product called ksqlDB, ksqlDB is a database built to do Stream processing on top of Apache Kafka.
Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on Linkedinor Twitter @jocelynbyrne
Observability refers to the capability of comprehending the internal functioning of a complex system by observing its external outputs. Observability tools help developers understand the entire scenario of an application by digging into the data that is being used and recorded. And can help pipe data across various solutions used within enterprises to read and discover what’s going on from analytics. Rookout is an observability platform with a developer-first approach that empowers you to collect any data point, even from the most intricate levels of live code, in real time and as needed. Liran Haimovitch is the CTO at Rookout, and he joins us today. We discuss about Rookouts recent offering called Snapshots. Rookout calls this the fourth pillar of Observability. With Snapshots, developers can accurately and quickly capture application state with outstanding performance and built-in security.
Mike Bifulco is CTO and co-founder of Craftwork. He’s also a developer advocate, writer, podcaster and serial startup founder. In past lives, Mike worked for Google, Stripe, Microsoft, and Gymnasium. Mike is also co-founder of APIs You Won’t Hate, a community for API Developers on the web. Mike’s publishes a weekly newsletter for product builders called Tiny Improvements at mikebifulco.com. Mike is on Mastodon at https://hachyderm.io/@irreverentmike
Distributed databases are necessary for storing and managing data across multiple nodes in a network. They provide scalability, fault tolerance, improved performance, and cost savings. By distributing data across nodes, they allow for efficient processing of large amounts of data and redundancy against failures. They can also be used to store data across multiple locations for faster access and better performance.
Turso is an edge-hosted, distributed database based on libSQL, an open-source and open-contribution fork of SQLite. It was designed to minimize query latency for applications where queries come from anywhere in the world. In particular, it works well with edge functions provided by cloud platforms such as CloudFlare, Netlify, and Vercel, by putting your data geographically close to the code that accesses it.
Glauber Costa is the Founder and CEO of ChiselStrike the company behind Turso, and he joins us today.
Full disclosure: ChiselStrike is a sponsor of Software Engineering Daily.
Alex is an AWS Data Hero, an independent consultant, and the author of The DynamoDB Book, the comprehensive guide to data modeling with DynamoDB. He was an early employee at Serverless, Inc., creators of the Serverless Framework, and was an early community member in the serverless space. His consulting and training work focuses on serverless architectures and database optimization. You can find him on Twitter as @alexbdebrie or on his site, alexdebrie.com.
The increasing complexity of modern cloud-native architectures has led to the emergence Platform Engineering. This practice involves the development and upkeep of an integrated product, known as an “Internal Developer Platform,” which serves as a flexible and supported abstraction layer between application developers and the underlying technologies. Luca Galante leads Product at Humanitec and he joins us today.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
C++ is a powerful programming language that has been in use for several decades. Its importance lies in its versatility and efficiency, making it a popular choice for developing software and systems across different domains. The impact of C++ is significant, as it has been used to create numerous high-performance applications, including operating systems, browsers, gaming engines, and financial systems. It has also played a crucial role in the development of the internet and the growth of modern computing.
In a recent development the NSA indirectly asked developers to use memory safe languages whenever possible because hackers frequently exploit memory issues. In this episode, we’ll discuss these safety issues with the creator of C++ Bjarne Stroupstrup.
Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm
A SIEM platform provides organizations with a powerful tool for improving their security posture, by providing insights into potential security threats and enabling proactive security measures.
Panther is a Cloud based security monitoring platform that helps teams detect and respond to security breaches quickly and intelligently. Jack Naglieri is the Founder and CEO of Panther and he joins us today.
Jeff is a DevSecOps engineer with strong experience in Security, The Software Development Life Cycle and Cloud Technologies. He’s an expert in HashiCorp Technologies and is one the most sought after trainers in EMEA and beyond. Jeff has worked for companies ranging from small startups to the biggest financial institutions, he runs his own consultancy that provides services in DevOps, Cloud and Security. Follow Jeff on LinkedIn @jeffhemmen
DataSet is a log analytics platform provided by Sentinel One that helps DevOps, IT engineering, and security teams get answers from their data across all time periods, both live streaming and historical. It’s powered by a unique architecture that uses a massively parallel query engine to provide actionable insights from the data available.
John Hart is a distinguished engineer leading the Event DB team, where he’s responsible for the time series database that powers the Dataset product. John is our guest here today.
Full disclosure: SentinelOne is a sponsor of Software Engineering Daily.
This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.
Vercel is the platform for frontend developers, providing the speed and reliability innovators need to create at the moment of inspiration.Vercel’s Edge Functions enable you to deliver dynamic, personalized content with the lightweight Edge Runtime.
What are the reasons why we need to improve performance? The JVM is a powerful piece of software – this is the reason why Java has maintained its popularity over the past 26+ years. But… there are some ways that the JVM works that can impact the performance of your applications.
One of the biggest challenges is latency – how quickly can we respond to a request from a user to access your application. That comes down to garbage collection – which leads to pauses and a bad customer experience. Garbage collection pauses can vary. But they impact application performance. The bigger the heap, the longer your pauses…but how do you optimize? What about warm-up time? That’s also a big challenge. We’ve addressed these problems without re-writing code from scratch – and that leads to better Java performance without any code changes.
This episode explores how you can use a performance JVM to make applications start fast, go fast, and stay fast.
Full disclosure: Azul is a sponsor of Software Engineering Daily.
Acme Corp Assistant"
! \nHow can I help?"
Jordi Mon Companys
Jeff is a DevSecOps engineer with experience in security, the software development life cycle, and cloud technologies. His advanced expertise in HashiCorp technologies places him as one of the most sought after trainers in the Europe, Middle East, and Africa regions and beyond. Jeff has worked for a range of different companies – from small startups to some of the biggest financial institutions. He now successfully runs his own consultancy that provides services in DevSecOps, Cloud and Security. You can find Jeff at 
AI-assisted software delivery refers to the utilization of artificial intelligence to assist, enhance, or automate various phases of the software development lifecycle. AI can be used in numerous aspects of software development, from requirements gathering to code generation to testing and monitoring. The overarching aim is to streamline software delivery, reduce errors and, ideally, reduce the time and costs associated with software development.
