NOW Insurance Uses Data and AI to Revolutionize an Industry

‍NOW Insurance is using lakehouse-powered data analysis and AI-driven innovations to revolutionize insurance for medical professionals. In an event hosted by Onehouse on May 16, 2024, we learned about NOW Insurance’s work and how Onehouse contributes to it. Continue reading for an overview, or view the full talk.

Jonathan Sims, VP of Data and Analytics at NOW Insurance, and Andy Walner, Product Manager at Onehouse, discussed in depth how they came to adopt a Onehouse-provided data lakehouse. The data lakehouse, which is powered by Hudi, provides the foundation for NOW Insurance’s differentiated analytics and AI-powered insurance offerings. 

Sims’ team has built a data processing system that is much more efficient, responsive, cost-effective, and granular than what is currently available from anyone else in the industry. The system produces tangible benefits to those insured by NOW Insurance: medical practitioners receive differentiated, cost-effective, personalized, and modern insurance coverage, without having to deal with archaic bureaucracy.

Revolutionary Data-Driven Insurance Products

NOW Insurance primarily provides professional insurance for medical practitioners. Physicians, nurses, nurse practitioners, home health aides, and others enjoy a highly differentiated experience:

• No paperwork: They feature a no-paperwork application process where users just sign up for service through a digital portal.
• Fast insurance decisions: They issue insurance quotes and certificates in less than three minutes, instead of days or weeks.
• Customized and unique coverage: They insure under many circumstances that other providers can’t, such as telehealth across state lines, and independent work, all while providing unusually flexible coverage and deductible options. 

Cutting the application time from weeks, or even months, to mere minutes requires NOW Insurance to process significantly more data than other providers. NOW can cover as many as 15,000 data points per application in near real-time. 

To provide fast quotes and personalized coverage, NOW Insurance leverages a large amount of data, as much as 500 tables’ worth, from difficult-to-access datasets. The data is used to build both client-facing AI models in support of end-user products, such as classification and risk assessment, and internal-facing AI models used in business process optimization. The NOW Insurance data team faced engineering challenges typical of products built on data of such scale and complexity.

NOW Insurance’s Data Engineering Challenges

Sims’ data team at NOW Insurance began adopting a data lakehouse architecture in early 2020, and grew to need a lakehouse service provider in late 2021. Their systems were ingesting large CSV data files, using AWS Glue crawlers to try to keep their schema up to date, and Airflow DAG code to handle table and schema changes. They decided on Apache Athena as their standard query engine, with Apache Superset providing business intelligence (BI) analytics support. After moving to a custom-built tracking system for analyzing user behavior, they started experiencing three very familiar growth challenges: frequent scheme changes, slow and inefficient data analytics, and resource-intensive data management. 

Frequent Schema Changes 

Alternative data sources, as well as public- and government-provided data sources, tend to be fragile and unreliable. The formats change frequently, and without warning, requiring schema changes to adapt to them. And there is a significant problem with late facts—information about a client, such as their existing coverage, historical claims, or payouts, may arrive after some decisions have been made, and will need to be applied retroactively. In addition, sometimes information about insurance policies and other qualifications will change retroactively.

Slow and Inefficient Data Analytics 

Insurance is a low-volume, high-value business. An individual client or user is difficult to acquire and has a potentially high lifetime value (LTV) to a company. Presenting users with accurate, up-to-date quotes and related information is crucial to winning and then keeping their business. As a result, user acquisition systems, as well as the services providing data to users in NOW Insurance’s web portal, must react in as near to real-time as possible to a user’s actions in the app, to their insurance decisions, and to newly arrived information, while integrating often-voluminous historical data. 

Resource-Intensive Data Management 

Before switching to a data lakehouse system, NOW Insurance was dependent on hundreds of data pipelines for building AI models and for managing and processing data. Data freshness was a challenge—data was frequently a day or so out of date. And there was significant cost, in the form of time spent by the data engineering team to perform manual data management.

The most obvious solution at the time was to move toward a formal lakehouse solution.

NOW Insurance chose Apache Hudi

When deciding which lakehouse solution to use, they considered Apache Hudi, Apache Iceberg, and Delta Lake. Hudi compared favorably to Delta Lake—Hudi is open source, just like the rest of NOW Insurance’s data software stack. Both Iceberg and Delta Lake had higher upfront costs. The larger open-source community backing Hudi spoke to its potential longevity, and the solution seemed to be technically a better fit for their needs. 

Sims considered having NOW Insurance build and manage all the systems required for a Hudi-powered lakehouse, but the effort seemed daunting. Besides adopting Hudi, his data team would have had to set up, configure, and maintain complex subsystems, including Debezium for data capture, Kafka for processing, and Deltastreamer for incremental changes; they also would have had to stand up an end-to-end change data capture (CDC) pipeline. The decision to move to a lakehouse stalled until Onehouse gained popularity in early 2022, and NOW Insurance saw a potential solution. 

Onehouse’s Managed Offering

Sims recounts how in 2022 he found out that Vinoth Chandar, the CEO at Onehouse, promised to take care of all of the setup and admin work required to set up Hudi for Onehouse customers in each customer’s own AWS environment. Onehouse seemed like a perfect solution for NOW Insurance and, after an initial demo, NOW Insurance became one of the first adopters of Onehouses services. Some of the main points in favor of Onehouse include:

• Provides a fully managed solution: Onehouse cleanly integrates into NOW Insurance’s architecture, with no need for significant work from an in-house engineering team.
• Reduces engineering overhead: The integration requires no engineering time be spent on setting up and managing data pipelines.
• Supports a robust range of features: Features offered by Onehouse can optimize NOW Insurance’s data flows, with support for automated CDC streaming, file sizing, catalog syncing, and storage utilization improvements.
• Enhances data governance: Onehouse can run inside a private virtual private cloud (VPC), supporting the strict data security and compliance requirements of the insurance industry. 

Benefits Of Building With Onehouse

‍

The switch to using Onehouse was a tremendous success. NOW Insurance’s data team saw multiple benefits:

• Significant cost savings: With Onehouse, NOW Insurance lowered storage costs and substantially reduced both engineering time and operational costs.
• Enhanced data freshness and access: Onehouse supports near real-time data analysis and data that is fresh—-in minutes, instead of days —allowing NOW Insurance to improve customer experience, standards compliance, and strategic decision-making.
• Streamlined model training: Onehouse efficiently manages large data models, leading to faster AI model training and development.
• Complex data analysis: Machine learning models can use more intricate data analysis and data management techniques, such as snapshots and time travel, to become more accurate and specific.
• Future-proof data management: Proven and established growth paths mean the system is expected to scale up to the point where it will easily support expected data volume and complexity changes.

As an early adopter of the lakehouse architecture pattern, and specifically of Apache Hudi and the Onehouse managed service offering, Sims was pleasantly surprised by the speed of adoption and general success of the technologies. What, at the time of the adoption, seemed like a new and cutting-edge technology and service provider, has since turned into a core component of their offering, setting a new standard for the industry.

The Universal Lakehouse for Your Data—Onehouse

Following Sims’ part of the talk, Andy Walner from Onehouse took a deeper dive into everything that Onehouse has to offer. Onehouse was created by the people who have been building Apache Hudi since the project’s original roots when it was built by Uber’s data team in 2016. Their team includes recognized industry veterans, and their services have seen wide adoption. Some of the largest data processing warehouses in the world now run on Onehouse-managed services. 

Today, Onehouse can be counted on as:

• Fast: Onehouse provides near real-time data analysis, scaled to the largest data platforms in the world.
• Cost-efficient: Data ingestion and ETL pipelines are 10x cheaper to run than the alternatives.
• Easy to use: Re-usable, low-code pipelines and a fully managed service are assembled to create a data lakehouse solution in days, instead of months.
• Universal: Onehouse supports all data formats, storage options, and data analysis compute engines.

Whether it’s a matter of continuous data ingestion, table management services, data lifecycle administration, or managed transformations, Onehouse provides a full set of cutting-edge managed services. Onehouse powers many modern, data-based companies and services, such as the ones NOW Insurance is using to revolutionize the insurance industry.

For more details about NOW Insurance’s implementation of Onehouse, you can find a complete recording of the online event we’ve reviewed in this talk or download our case study. If you’re looking to learn more about the various services offered by Onehouse, check out our extensive resource library.