ZipMatch at AWS Summit Manila

I had the pleasure of presenting ZipMatch’s technology on April 20th at the AWS Summit Manila. You can check out on the Summit On Demand page, or better yet, right here. 🙂

Since the notes on the slideshare were just talking points, I thought I’d post a more complete write-up here. Enjoy!

 

Intro

Good morning everyone. Let me begin by saying its great to be here today. My name is Kyle, and I’m CTO at a real estate marketplace based here in the Philippines called ZipMatch.ZipMatch is about 4 ½ years old right now, and we’ve been running on AWS for about 4 of those years.Today, I’ll be sharing our journey with AWS. Why we chose it, how our business has grown with it and what’s next for the future.

Who is ZipMatch?

ZipMatch is real estate marketplace based here in the Philippines. We focus on data and transparency, which enables renters and buyers to compare properties against each other in greater detail than classified sites.Using our unique algorithms, we can score and rank properties based on factors such as neighborhood, amenities, affordability, rental yield, price per square meter and many other factors.We’re backed by a great group of Silicon Valley and Regional investors which include 500 Startups, Monk’s Hill Ventures and Kickstart.

2013 – Our move to the cloud

When I joined ZipMatch in 2013 we were a small team, and our website was hosted on a single server in a datacenter in Makati. The server worked fine, but simple changes for security, networking, or storage would take days or sometimes weeks to be handled. Each change required multiple phone calls, e-mails and usually paper contracts to be signed. Not exactly how a tech-startup prefers to do business.After researching a few solutions, we agreed to setup a proof of concept on AWS.None of us in the team had any experience with AWS at the time, but our reading told us a lot of the systems were based on existing standards, so we agreed to run a proof of concept.

Things went quite smoothly. Within a week we had a fully functional copy of ZipMatch running in the Japanese AWS region. So we scheduled a time to move production to the cloud.

Then we realized, latency to Singapore would be faster for most of our users. We had deployed to the wrong country!

After some reading, we realized the issue was trivial. We followed a guide to take a snapshot of our working EC2 instance, redeploy it in Singapore. This all got done in about an hour. This made for some very happy geeks.

This was probably the first in a long list of wins where AWS and Cloud Infrastructure saved our business a lot time and money, and we knew we had made the right call.

2017 – Where we are today

So, that was roughly 4 years ago. From that single, humble server we’ve grown a lot. Here’s how things look now.We currently leverage about 22 AWS Technologies. I don’t have enough time today to go into each one, but I’d like to share a few highlights.We run our development, staging and production environments in the cloud using some of these technologies:

Compute / EC2

Half of our product is still driven by EC2 instances. We need multiple now because we have much more product complexity and about 100x the traffic. The instances are based in different availability zones for added redundancy. The traffic is split between them using the Elastic Load Balancer service.

Database / RDS

Our data is our biggest asset. We now run our primary databases in RDS in a master-slave configuration. RDS handles a lot of the housekeeping a DBA’s or sysadmins would typically have to do. We’re a big fan of RDS because aside from the initial migration, we’ve basically spent zero hours on it. It just works.

Static Content / CDN / S3 and Cloudfront

We use S3 for nearly all of our static content. CSS, JS, fonts, images are all stored here. This includes our virtual 360 tours and VR assets.

We add over 15,000 images a month to S3 from property listings alone. We leverage cloudfront, AWS’s CDN on all of our S3 buckets to put content closer to our users and cut down load times. So although our infrastructure is based in Singapore, our Philippines users will load about 95% of the website from an edge node in Manila.

Lead Lifecycle / Event Bus

Each inquiry that comes in on ZipMatch is subject to many various business rules.

Where leads are assigned. SMS and anti-spam checking. Quality scoring and routing etc.

As this system grew in complexity we moved it away from monolithic PHP code, and into a more robust event-driven system. The business rules and logic are python code running in dockers containers, and the systems uses SQS as an event bus and Dynamo for stateful storage.

We’ve had fun a bit of fun load testing it with a few thousand inquiries per minute. It handled it with ease, which means we all sleep better at night than we used to.

Data Science

Data is our edge at ZipMatch. We use it to understand our customers behaviour. We use it to spot trends in the market. We use it to avoid fraud, and bogus information. We use data to dive into the statistics of each province, city and barangay. We are a data driven company through and through.Our analytics and data visualization is done through AWS’s data warehouse product, Redshift. We run ETL processes nightly which dump and transform data and upload it to redshift. We also do periodic extracts from other 3rd party data tools (i.e. mixpanel) and upload it into redshift for analysis / correlation with other data.Redshift allows us to centralize all of our data in one place, and run giant queries we wouldn’t dare in production.

Our data visualization tool of choice is a piece of software called Spotfire. In Spotfire we can build live, real time dashboards for whoever needs them in the business. They are very powerful, and the end result is that most questions we have can answered right away with a pre-built dashboard.

It was actually through our Data analytics setup that we spotted a great market opportunity. Brokers often complain about a lack of data in the market. Government agencies do not publish many real estate statistics, and many records are on paper, making true market information difficult. We realized with our millions of data points from years of listing and project data, we could a lot of great information about Philippines real estate that would be a first in our industry.

2017 – Rankings and Statistics

In March of this year we released a new section of our site dedicated to ranking all the projects in the Philippines by various criteria. Our rankings section allows you to sort various projects or locations by investment data such as rental yield and per SQ/M prices. Lifestyle data like the proximity to points of interest such as schools, shops, and hospitals. Affordability data, looking for the best per SQ/M deals in the country. This was not easy. Our rankings section provides a live summary of information from over 100,000 property listings and 2000 projects. That’s over a million data points to crunch. The vision was to provide a dashboard into the Philippines real estate industry and rank properties by the factors our buyers and sellers value.

We delivered this by building a whole new web front-end and data-API. We run these services in docker containers, which is managed by an ECS cluster. We cache data in redis which relieves pressure from our backend databases.

We send traffic to the rankings section using the new Application Load Balancer service. ALB allows us to do layer-7 traffic routing, sending specific HTTP requests to the new web front end.

We did a soft release in March this year. So far, the industry has been very excited about the data we’ve released through our rankings section. Buyers, renters and even developers themselves have shown a huge interest in the statistics, and data from our web analytic tools shows very high engagement so far.

All in all, it was an ambitious project and I’m confident in saying we could not have spotted the opportunity and delivered it so quickly without the tools AWS provides.

What’s next for ZipMatch?

Ultimately, our goal is to connect buyers and sellers. It sounds simple enough, but all the difficulty lies in the middle. The buying decision is so large, and transaction so complicated that the average renter or buyer is often thinking about dozens of factors in their purchase. Is it within budget? What’s the commute time like? Is it close to schools for kids? What sort of common areas does a project have, etc.We have already begun down the path of leveraging data to help customers make better decisions, how can we ease this further?Perhaps smarter recommendations, using machine learning to constantly improve the algorithms which match buyers and sellers?

Or AI frameworks which helps understand customer needs, and automate repetitive processes?

AWS has existing technologies for both AI and Machine learning. So if and when we choose to leverage these technologies we would certainly look to AWS solutions first.

Running our business on the cloud with AWS has been a cornerstone of our product and IT strategy. It is a big part of why we’re here today. We’ve only scratched the surface of the many opportunities in our industry and we’re excited for what the future has in store.