Tech Blog Menu

Introduction

My name is Nicolas Juneau and I am Coveo’s CFO (Chief Failure Officer). As the blog review team has yet to unpack a huge backlog of articles to review, I took this opportunity to write on this blog about a subject nobody wants me to talk about: patterns for project failure.

We all heard the conferences, we all read the articles: we know how to ensure a project’s success. After all, software engineering is a tried and true discipline as old as civil engineering. Julius Caesar successfully designed, wrote, and deployed his very own cipher back in the Roman empire, so we should have this figured out by now. Today, let’s take a break from articles teaching us what to do and let’s focus on something more entertaining: striving for failure.

Let me do what we always try to do on Star Trek: hopefully entertain you, perhaps even make you laugh a couple of times. And when your guard is down, slip in a heavy idea or two…

– Gene Rodenberry, “Inside Star Trek”, 1979

As you may have heard, Coveo recently released its Generative Answering solution (also called “GenAI”). It’s been all the hype internally at Coveo, as well as externally with multiple customers and partners approaching us to play with this new product and implement it on their end.

Similarly, there’s recently been a lot of hype about the release of Baldur’s Gate 3, the Larian Studios video game based on Wizards of the Coast’s classic TTRPG Dungeons and Dragons.

As a big fan of D&D and a big fan of new tech, I thought it would be a great idea (and a great way to sink my teeth into a new Coveo product) to create a Coveo GenAI-powered bot that can answer questions about Dungeons and Dragons.

In the past few years, our cloud service provider, AWS, has been overall pretty reliable. But like everything in life, nothing is perfect and as Werner Vogels (CTO of Amazon) repeated many times - “Everything Fails All the Time”.

Over the years, we have seen regional outages affecting a subset of the services that we leverage. When those outages occurred, we often relied on another AWS region to quickly spin something up that allowed us to continue delivering our services. About 4 years ago, we delivered our multi-regions feature to reduce latency, and just recently we worked towards leveraging those regions in an active-active way for our search infrastructure. The main driver was to improve resiliency and to handle those outages the same as any given Tuesday.

There are only two hard things in Computer Science: memory problems, error handling, and of course off-by-1 errors.

For years, I’ve felt uncertain about what to do when something unexpected happens in a program I wrote. Should I return an error code, crash, crash in debug builds only, throw an exception… This uncertainty lit up my curiosity, and slowly, while I accumulated the years of experience, I became more aware of the tradeoffs behind each strategy. Because, of course, the answer is, as always, it depends.

I spend most of my days inside a code editor, Jira, Outlook, or in meetings. Now that one of our projects won the Identity Management Project of the Year – SMB award, it’s a great opportunity to take a look back at what the Coveo R&D Defense team built.

The Coveo infrastructure is constantly growing. DevOps engineers add new regions and services, which leads to more systems that can break, more complex access management, and more complex audit logging. If I tell stakeholders that the entire R&D department needs always-on access to all the services they deploy and own in a production environment, some of those stakeholders will tell me that the risks are too high and that it is not acceptable. On the other hand, if only a handful of people can help when there is an incident in production, the on-call access management person will have to be woken up every time an engineer needs access to a specific resource. This makes access management unhappy, and increases the time to resolution, potentially even causing a breach of our service level agreement. Leadership won’t like that.

This is why Coveo needed a good middle ground. The R&D department needed a system that allowed selected employees to gain privileged access on systems they own for a short period of time, fix the incident, and follow up with a post-mortem. Back in 2020, Coveo adopted strongDM to manage privileged access rights. While it already supported granting temporary privileges, it lacked a way to allow employees to quickly request a temporary privilege, without waking up the strongDM administrator at 3 AM. From the strongDM APIs, the R&D Defense team built that system.

Everybody loves a fast, responsive continuous integration (CI) setup. While it’s fun to battle each others on rolling chairs, having quick feedback on pull requests builds and fast deployments is an important part of a good development environment and is also something we constantly have to invest time in.

In this post, we will show you a neat way to safely use your local Maven repository across multiple build processes using the Named Locks feature of Maven Artifact Resolver. This will speed up your concurrent builds by reducing the required time to download dependencies and will also help maximize your CI instances usage.

There is an old adage that says, “Hardware eventually fails. Software eventually works.” As the Coveo Blitz competition is approaching, we are ramping up the platform to receive more than 135 participants who will produce software bots that fight against one another for the prestigious Coveo Cup. One of the cool features of the platform, that we’ve created and improved over the years, is the ability to visualize past fights by downloading a replay file.

But this year, like every year, was different. This year, the replays were up to 50MiB large and the download time was suffering from it. By suffering I mean 50 seconds to download 40MiB; waiting for a mp3 to download in the 2000’s kind of suffering. The replays are stored in S3, proxied by our servers written in go, so let’s see what is going on in the go code, and maybe it will eventually work!

In my last blog post, I described a practical way of integrating Sentry with Prefect 1 flows. Since then, Prefect 2 was released, and with it, many interesting capabilities were introduced. I won’t spend time explaining why you should migrate to Prefect 2, but I suggest you have a look at the new features presented here. If you’re interested, I also created a step by step GitHub repository showcasing some of the new features available so that you can try it yourself.

Use case

Following the newly launched Coveo Slack Application, it is now possible to search your whole Coveo index directly from Slack with a single click or slash command. This is definitely a great approach to search, but what about Slack users that are not aware of this feature but actually need it? This is the use-case we will tackle in this project: automatically answering questions in targeted Slack channels.

It is not a secret that some Slack channels become a search dump and it is not always easy to address them all, even when the answers are sometimes easy to find. A great way to tackle this problem is to try to find questions and send them to the Coveo Search API. A simple way to do this is to listen to all the messages sent on a targeted channels, and only trigger a search on the ones containing a question mark.

A simple architecture for this idea would be to host a serverless function in AWS Lambda, and send the detected question to Coveo’s Search API. For better versioning, deployment, and monitoring approach, a serverless Application will be used for this project.

At Coveo, we deal with an enormous amount of data on a daily basis. With data growth, our data platform has also grown from a single team a few years back, to more than 3 teams and 20 employees. With this growth, we also gave ourselves the mission of democratizing data across our organization and allowing more and more external teams to access and experiment with the data we capture.

The challenge we rapidly faced was that we had to offer more and more support to these external teams on how to automate some of these applications and scripts they were developing over the data. Most of these stakeholders are often really proficient with SQL and Python, but have less knowledge and experience with CI/CD, infrastructure, and monitoring.

To solve this problem, we started looking at some solutions that would allow these teams and individuals to easily deploy and run these different workloads in production, without having to develop an in-house solution that would require a lot of engineering time and maintenance.

After investigating multiple solutions to solve this problem, a clear winner stood out for us: Prefect.