AI Assistant for Cyber Insurance Submissions (2024)

My Overarching Mission

• Help brokers break into Cyber Insurance without prior experience
• Help brokers scale (faster submissions, compare policies, build presentations…)

Problem: More Options of Providers = More Questions to Answer

Cyber Insurance is technical, difficult to sell. Connecting insurance seekers with multiple providers increased their chances of a quote BUT entailed answering more questions:

Evolution of UX Strategy for Filling Cyber Submissions

The timing was right for the AI features:

• Product Vision: Long-standing design principle seeing Relay as an “Assistant” to brokers
• Customer Value: Customers’ ongoing struggles integrating cyber into their workflow
• Technical Feasibility: Engineering brought emerging AI capabilities to my attention

I was tasked with investigating how we could embed Gen AI into our product.

Feature 1: AI Submission Assistant

I sketched multiple concepts and decided on a slide-out AI-Assistant. To keep the user experience simple and control quality of responses, I designed a short set of presets for the user to choose from:

Shipping to learn: I expected users to suggest more things over time.

Feature 2: Email Quote Bot – Meeting Customers Where They Are At

During customer calls, we heard repeatedly that brokers were “portaled out” from using multiple insurer portals.

We needed to meet users where they felt at home: email and spreadsheets. Over time, I came to appreciate that certain user personas just wouldn't log into our software. For example, "producers" (sales people) spent all their time on phone calls with customers and traveling. There were hints of opportunities to engage them and give them tools, but they would never learn or even log into any new system. This made the team realize that we need to think beyond the UI we'd invested so much time into. 

I designed an Email Bot workflow, so users didn’t have to log in to get a quote. The bot consumed documents that contained client information (e.g., last year’s PDF submission) over email and replied with quotes. We used an LLM backend to parse the document, automatically fill a cyber submission, fill gaps with assumptions, and reply with quotes:

Impact of Email Bot on Customer Outcomes

Several customers came to rely entirely on the new Email Bot. One of the pilot customers was able to quote 20% more business:

From our Beta release, we saw we weren’t hitting our 80% submission-to-quote success ratio. I identified the causes, and we fixed it in the next release:

My Role as Blend of Designer and PM

• Designed mocks and prototypes
• Defined scope of MVP and planned next releases
• Managed team of 2 developers
• Presented progress updates at company all-hands
• Ran Wizard of Oz user research to validate MVP prior to build
• Persuaded AI vendor to build enhancements for us
• Lead prompt engineering and established test strategy
• Interviewed customers post-deployment for testimonials and feedback

Challenges Designing AI Submission Assistant

Here are some challenges I negotiated in our first foray into AI:

Technical challenge	ChatGPT wasn’t yet ready for open-ended queries; we also had no experience with LLMs in-house
Usability challenge	We needed this to be dead simple – our users were often not tech savvy and avoided learning new technology whenever possible
Implementation challenge	We needed this to fit seamlessly into our product and give us the biggest bang for the buck; no edge cases or dealing with users going off-script
Sales/marketing influence	Head of Product pushed for “in-your-face” approach, so AI could feature more prominently in marketing and sales presentations. I didn’t want to overcommit our UI to this until we got some real-world feedback; and I didn’t want it to disrupt our core flow. I negotiated a compromise with the Head of Product to ensure the feature is discoverable but lived out of the way in a side-bar. This pattern was also reusable in other contexts.
Legal influence	Legal wanted to put the feature behind a wall with a large disclaimer. I wrote and negotiated the wording and flow with legal to keep it unobtrusive.

User Research via “Wizard of Oz” Prototype

For two weeks, I monitored my inbox and impersonated a chatbot (assembling submissions and attaching quotes manually).

My research objectives included:

• What’s maximum time we can take to respond so the solution is acceptable?
• Would users be comfortable sending follow-up questions to a chatbot?
• Would users even remember to use the new workflow? (old habits die hard in this industry)
• How would users react to assumptions? What information would users need to verify?
• And most importantly: would the document extraction tech be good enough to produce quotes most of the time?
• Also: collect more real documents to refine our prompts for data extraction

I varied my responses to test various hunches. I’d make successful responses but also reject a submission on purpose to see what the user would do instead.

Conflicting Mental Models: Learning how brokers think and make trade-offs a key part of my job. For example, brokers traditionally want to send insurers strong deals to gain preferential treatment in the future. If they send weak deals, it might damage their reputation. With APIs and Email Bots, they didn't need to worry about this, but old habits may have caused people to avoiding sending greater volumes through the email bot.

Learnings From Prototype

• We needed to respond within 5 min in order not to block other steps in users’ workflow
• To some users it felt natural to ask the chatbox follow up questions, but it wasn’t cruicial; it was enough for v1 to create a “shell” submission, which users could edit via link instead
• We measured the time saving of 10-20 minutes per submission from data entry alone; this translated into one customer being able to place 20% more business
• Needed data prediction improvements: e.g., 15% of submissions couldn’t go through because of missing industry code
• Most importantly: we kept seeing submissions despite users being told they could stop at the end of our pilot

The following quote further illustrates how this feature validated our principle of meeting users where they are (i.e. shifting from web UI to their inbox):

Interestingly, this customer even formed their own garden path by doing extra work to put information into a PDF just so they can submit it to the bot (early version of bot wasn’t able to read email body). This is an example of how prototyping lets us observe real user behavior.

MVP Scope and Feature Roadmap

After the prototyping and consulting with engineering, I defined the roadmap for this feature, setting MVP constraints as:

• Bot can process 1 document at a time
• Bot cannot read email body
• Bot cannot reply to follow ups (user must hit Edit link)

Following that release, we started work on enhancements (not covered here).

Example of User Testing the Bot Response

To help refine the bot response, I asked users over email to choose among variants:

One user replied: “B is better” while another said “Example A is better.” So I followed up with more questions and uncovered interesting insights regarding:

• How granular the breakdown should be
• What layout entails lowest mental load (it wasn’t what I expected)
• How many bullets are too many
• Which items are most crucial to review

In the end, I reused the existing email “digest” template I had designed for another feature. The bot summary was simplified and tacked on at the bottom:

Prompt Engineering & Test Strategy

To ensure we had an 80%+ success rate converting submissions into quotes, I had to get creative with designing and testing prompts. The AI vendor did not have a mass-test feature, but I wanted a systematic approach. I built a JavaScript browser automation for mass prompt testing and established a process to regression test prompt changes. Here is an example of my log, showing pass/fail with various prompt variants across dozens of document samples:

Alignment With Relay’s Design Principles

Whenever I and the team made a decision impacting UX, I captured it as a principle so we could be consistent. They held up well, even as we pivoted our business:

Relay's  Design Principles:

➤ Virtual Assistant - The right help at the right time
➤ Meet Customers Where They Are At - Don't fix what isn't broken
➤ Single Entry - Efficient data entry
➤ It's Your Decision - We can suggest but the customer is in control
... and others

Relay’s foray into AI was an extension of prior decisions. For example, the Virtual Assistant principle initially described features that replaced common Admit Assistant tasks like:

• Create me a proposal automatically
• Remind me of upcoming renewals

LLMs merely allowed us to take this principle more literally. By focusing our first AI features on the submission form and email workflow (the salient and familiar), we were able to get a positive response to AI from a non-tech-savvy crowd.