Engineers at Agari Solve ‘Interesting Problems Every Day’ in their Quest to Make Email Safe
Last time we talked with the Agari Engineering team, they were working on a product to help eliminate email as a channel for cyberattacks. Two years later, the team has successfully spun up an additional product and the email security Agari offers is more sought-after than ever — thanks to a slew of recent high-profile phishing attacks, including on the DNC last summer. For this story, we asked Chris Haag (Director of Engineering), Scot Kennedy (Director of Analytics), Gabriel Poon (Data Scientist), and Natia Chachkhiani (Software Engineer) to tell us how the team has changed (Kanban AND Scrum) and what fun challenges they’re focused on today.
Interested in joining the team? Check out open roles or get in touch with the team via jobs@agari.com to learn more.
Let’s start with the basics. What does Agari do?
Chris: We’re working to make email the most effective and safest form of communication, and we do that in a couple of ways. First, we help domain owners like Facebook tell the world what a legitimate message from them looks like, and what to do if it looks inauthentic. Second, we filter out any malicious emails sent to our customers’ employees.
A lot of security companies claim they stop targeted phishing — we’re different in that we actually do it. Most email security companies look for patterns of malicious messages. The reason that usually doesn’t work is that phishing, unlike spam, requires relatively few emails to do damage. For instance, The Economist reported that it took the Russians fewer than 30 emails to the Democratic National Committee to hack John Podesta. To solve that problem, we analyze good emails rather than just the bad ones. Since most email communication is legitimate, that gives us much more data to work from, and machine learning is excellent at making predictions based on those patterns. Once we’ve moved aside everything that’s clearly legitimate, the rest is either the start of a new pattern of legit communication, or malicious.
Tell us more about the data science approach you’re using.
Scot: When we evaluate an email, we look at two dimensions — authenticity and intent. Authenticity is simply, “Is this really from who it says it’s from?” If I get an email from PayPal but the “y” is replaced with a “7” then I score its authenticity very low. For intent, we’re asking the question, “Does this sender mean me harm?”
We also look at who’s receiving the email, not just who is sending it. For example, an email from Taco Bell asking for your bank account number is an obvious red flag. But if Taco Bell is one of your company’s clients, that might be more authentic. So part of what our machine learning does is characterize what personal communication looks like, versus what marketing or phishing looks like. Each of those has a different social graph and pattern.
“A lot of security companies claim they stop targeted phishing — we’re different in that we actually do it.” — Chris
Gabriel: Splitting our method across those two axes also helps our customers grasp what we do more easily. We could explain how we threw a machine learning model at an email and it made this prediction, but it’s much simpler to say, “This email scored really low in authenticity.”
What interesting projects have you been working on?
Natia: I’ve been enhancing our integration with Office 365’s email APIs, figuring out how to load the elements so we have permissions to move the messages and can make the setup seamless. We want to simplify it to one click, where everything else happens in the background, and we need to figure out exactly what those background processes should be. It’s been a challenge, but it’s fun; I get to work with a lot of different parts of our product, and I’m responsible for the entire lifecycle.
Gabriel: I’m working on changes to our authenticity model. One of the challenges is that we don’t have a ton of labels to work with. Traditionally, you’d have some giant data set, or a reasonably sized one at least, for what you want to classify. You’d randomly sample that data, train your model on it, and then test the model on a new piece of data. But when we look at malicious emails, we might get one phishing message for every thousand messages we receive, and only a couple of those are labeled. So part of the long-term project I’ve been working on is how to teach the system to pick up on signals from a small number of labels.
What were you doing before Agari and why did you decide to join the team?
Scot: I studied applied math in grad school, and started out in bioinformatics after graduation. One of my first jobs was in internet security at a company called IronPort, where we were dealing with large, messy datasets and trying to identify patterns. I realized a lot of the principles of biotech carried over to internet security, and I was excited about applying that background to fix common, everyday problems.
Agari was particularly interesting to me because when I met the management team, I could tell they were people who I could trust to make good decisions and who I’d enjoy seeing every day. To have a great startup, I think you need an interesting idea, a great team, and a product that makes people’s lives better — and that they’ll pay for. Lots of places have one or two of those things, but it’s rare to find a place like this, where that whole chain of wonderfulness happens.
Chris: I’ve been at various startups for the past 17 years working in engineering and devops roles. I was skeptical when I first heard about Agari, because to me, working in internet security had this element of selling fear. But the more people I met here, the more I wanted to join. Now I’ve been on board for almost two and a half years, and every day I see how what we do serves a purpose.
“To have a great startup, I think you need an interesting idea, a great team, and a product that makes people’s lives better — and that they’ll pay for.” — Scot
Gabriel: I was at a company called TrueCar before I came here. I thought the problems Agari was solving were meaningful and interesting, and like everyone else, I liked the vibe I got from the team. Plus, there are a lot of people with advanced degrees here, and I knew if I joined I’d learn a lot.
Natia: I second the learning opportunities; we have so many great engineers here. When I ran into problems on my latest project, I went around and talked with people, and they were more than willing to help. We get to learn outside of the office too; Agari pays for each of us to attend one conference every year on a topic related to what we do or what we want to learn.
As for how I ended up here, I moved to the U.S. from Georgia eight years ago and started at Agari last year, immediately after I finished my master’s in computer science at UCLA. I joined for a couple of reasons; one, the team was awesome when I interviewed, and two, everything we do is so relevant.
Tell us about a time when you realized the Engineering team was struggling, and how you fixed it.
Scot: For a while, we were so focused on meeting specific customer needs we weren’t making time to improve foundational processes that would serve the long-term success of the product. Our management team saw that and did something about it. The three directors and the CTO formed a scrum to do customer escalation management, which allowed one team each quarter to step away from working on customer needs and focus entirely on an internal project. It was an organizational risk, but it really paid off.
Another example that comes to mind is the effort we’ve made to increase interaction between teams. It’s easy to hang out with the people you see day in and day out, and you’ll always be close to them, but we wanted people to know what other teams were doing too. Things like playing Ultimate Frisbee together might sound a little goofy, but it’s really helped, along with bi-weekly tech talks, where teams share the cool things they’ve worked on and what they’ve learned. We also bring in outside experts to teach us cool stuff; we recently had one of the people behind Empire come in to talk with the team.
How is the Engineering team organized and what are the challenges you face?
Chris: We’re split into small teams of about six engineers, and each of those teams has a lot of autonomy over how they work and which technologies they use. For instance, last time we talked, we were adhering to Agile pretty closely across the board; now some teams still have scrum but some have shifted to Kanban. That has its benefits, but also its challenges. We’ve had to circle around a few times to decide which processes our teams should have in common, like how we use Terraform or do quality tests.
It’s particularly important that we understand where we need to be consistent, because we’re working across multiple products, handling our customers’ outbound and inbound security. We had an infrastructure as code project, for example, that we ended up sort of doing twice and then bringing back together. We’re launching an Engineering Enablement team in Summer 2017; their customer is the rest of Engineering, and they focus on things like continuous integration and better tooling and test suites.
“We used to adhere to Agile pretty closely across the board; now some teams still have scrum but some have shifted to Kanban.” — Chris
Natia: I think a team like that will really help speed up deployments, which is a pain point. It’s a lot of hours that we’ll be able to spend on other things, and definitely an area where we can be more efficient.
Scot: It’s really important to look for those opportunities for improvement, and to push for them. In the last six months, we’ve seen the two branches working across each product start using a lot of the same tools, and we’ve seen some of the cultural divide evaporate. That requires conscious effort.
What’s your collaboration like with other teams?
Natia: We work closely with Customer Success. They talk directly with customers so they understand what’s going well and what’s not. We have a Slack channel dedicated to the feedback they hear. I’ve been able to hop on a few customer calls too; it’s always good to hear from the customers directly. That’s another benefit of being at a smaller company.
“It’s always good to hear from the customers directly. That’s another benefit of being at a smaller company.” — Natia
Chris: Yeah, our Customer Success team is extremely helpful; they have so much insight into how our customers or prospective customers are looking at an issue. I’ve reached out before and said, “Can you give me a list of the customers dealing with this specific issue?” They always give great, specific feedback.
Let’s talk about location. What’s it like working in San Mateo?
Scot: We’re seeing a lot of startups come here, but it still feels like a quieter town. I like that you meet people here who don’t work in tech at all. This office also has an embarrassment of culinary riches nearby, and there’s a great Japanese garden we like to walk through during one-on-ones. Plus, from where I live in San Francisco, it can be faster to get to San Mateo than a lot of parts of the city.
Gabriel: Yeah, the commute is easy. I live in San Francisco, like the rest of the data team, and I take Caltrain; you can literally see the stop from the Agari office. I also like being able to take a break from the city.
What are you looking forward to in the near future?
Gabriel: I’m excited about our upcoming hackathon. I’m planning on a couple of projects related to deep neural networks.
Scot: Yeah, we’re unveiling a new and improved hackathon this year. We’d stopped doing it, which I think actually speaks to the fact that we’re already solving interesting problems every day. The hackathon didn’t seem that special by comparison, and people had kind of stopped getting excited about it.
Chris: Right; it wasn’t a perk anymore. So a bunch of us went to lunch and brainstormed what a great hackathon at Agari would look like. We’re going to spend two and half days on projects that never quite hit the priority list, but that’ll be awesome when they’re done.
Scot: We’re also keeping it within business hours. A lot of us have families, and we all have personal lives, so “hacking all weekend” wasn’t that appealing to us. That’s something we value here; people will tell you, “Hey, go home.”
What interesting problems could someone help solve if they joined this team?
Chris: One challenge is that we’re currently down revved on both Ruby and Rails. Ruby won’t be that tough, but we’re still figuring out how to get up to the latest version of Rails. Also, scaling in an enterprise market is based on the customers you acquire. We can increase the number of events per second we need to process by an order of magnitude just by adding a particularly large set of new customers.
Scot: As far as what we’ve already achieved, we’re very cloud native now. Everything we do lives there, and moving into that more abstract space has been amazing. Things that someone used to have to spend all their time building — now you can just click and it’s there. Because of the cloud, our second product runs in real time; it used to run once per day. That’s changed how we interact with our customers, because now we are constantly looking at the messages coming in. When we’re asked a question, we can look at it from a data science perspective and send an answer back in less than a second. So we’re doing data science in a real-time, pure cloud environment, which is different than anything I’ve done elsewhere.
Curious about engineering jobs at Agari? Check out open roles or get in touch with the team via jobs@agari.com, to learn more.
