Why You Shouldn’t Integrate with a RETS Server

Real estate is a large industry, and the agents who are a part of it are busy people. At Lucid we built out a solution that helps real estate agents save time when trying to sell properties. We did this by connecting the MLS data stored in sources like RETS servers directly to Lucidpress where the agents can create content for their listings. While the end result is a simple and smooth experience for the agents, the process we took to integrate with the sources and get the data there wasn’t as easy.

Real Estate Data Import (REDI)

  By using the Real Estate Data Import (REDI) feature in Lucidpress, real estate agents can quickly create materials to advertise the properties they are selling. We pull in information for their properties from their various data sources, and within a couple of clicks, they can apply a property to a template and be ready to print/share flyers, postcards, or anything else they create.

  While implementing REDI, we decided on a few data sources to integrate with. Lucidchart's data linking feature supports Google Sheets, and with some small modifications we were able to integrate with it for REDI as well. Zillow was an easy second data source to add because it provides a clear XML specification for parsing the data. A Multiple Listings Service (MLS) integration seemed like another good data source for REDI. MLS is a data source for real estate brokerages to share information with each other. However, as we explored integrating this data source we discovered interesting challenges

Multiple Listings Service (MLS)

MLS is a suite of services that has origins as far back as the late 1800s, and is used by real estate brokerages to share property listing information with one another. This information is typically stored in some kind of database. The incentive for sharing this information is that agents from other brokerages will find buyers for the properties an agent is trying to sell. There is no single nationwide MLS, and each MLS is controlled by a local or regional association of realtors. Since there is no single nationwide MLS, there is also not a uniform standard for the data stored.

Real Estate Transaction Standard (RETS)

Real Estate Transaction Standard (RETS) is about as close as you get to a standard in MLS. RETS is an XML-based standard that specifies how to interact with a server storing MLS information. For example, a few standard transactions that are supported with RETS include: Login, GetObject, Search, and GetMetadata. The disappointing thing about RETS is that it doesn’t specify a standard for how data is stored. Underlying data such as address, agent name, or list price does not have a standard format.

Integrating with RETS

We decided to implement the MLS integration with a single MLS at first. Right from the start I ran into trouble. I couldn't log in to the RETS server with the credentials we'd acquired for the MLS. After a couple of days of debugging and digging through a lot documentation, I discovered you needed a specific custom user-agent header in order to even authenticate to the RETS server. As I later discovered, this header isn’t required by all RETS servers, which makes things even less consistent from MLS to MLS. Once I successfully logged in, combing through the data was another adventure entirely. The response for the login gives you a list of “capability urls.” These are the set of functions that you can perform on the server (Search, GetMetadata, etc). Once you successfully log in, these capability urls are the top-level endpoints to hit in order to retrieve additional information from the RETS server. For REDI we wanted around 20 basic pieces of information about the property listings for a specific broker. In order to find this information, I had to do a fair amount of manual digging. The first MLS we integrated with had over 400 fields of data for each property listing. You could find out anything about a property, from how many bathrooms there were to whether or not the barn on the property had electricity. There is no standard between different MLS sources on how these data fields are named or structured, so the first step to retrieving the information you want is to hit the GetMetadata endpoint. This endpoint gives you a list of resources, their data fields, and properties of each data field. After manually searching through this metadata for a day or two, I found almost all the fields we wanted to retrieve about a different property listing. Once I found the data we needed, I implemented a solution to connect to the server, retrieve the data, parse the data, and store it on our end. After almost a month of research into RETS and understanding the structure of the data for the MLS we were integrating, I finally had the data pulled into Lucidpress for our customers to use. After all that work, I felt pretty good about our solution. That changed as soon as we were asked to integrate with a second MLS. After a couple of days of investigation into the second MLS, I quickly realized the two MLS sources were quite different from one another. I had a few options. I had already done some of the heavy lifting the first time around. I knew what needed to be done to build a second implementation to retrieve and parse this new data format, and I could have gone that route. However, I could see that the variety of data formats between servers would mean this approach of integrating with RETS servers would quickly become a full-time job of its own. Instead, the better solution was to find a more generalized and scalable way to integrate with MLS data sources.

SimplyRETS

Thankfully there was a scalable way to implement our MLS integration. After further investigation, we decided to integrate with SimplyRETS, a third-party software that connects to various RETS servers and normalizes the data for developer use. As a result, we could build a single implementation to connect to SimplyRETS and pass it the right parameters for the various MLS sources we wanted to access. So we scrapped my month of work to integrate with the first MLS and connected to SimplyRETS API instead. Now when customers have new MLS data sources to connect to, there is little to no engineering work required.

Conclusion

Through the process of bringing MLS data into Lucidpress, there were a few lessons learned:

1. Standards in tech are important. RETS is good because you can know how to authenticate to an MLS server, but without a standard data format, it becomes unscalable to build out an integration.
2. Scalability for processes and systems is important. Just because something works a certain way (like integrating with the RETS server directly), it doesn’t mean you should use that same approach indefinitely as you grow.
3. Often the first approach you take isn’t the right one, but it is still worthwhile. The month I spent connecting directly to a RETS server was helpful in understanding the data we retrieved from SimpyRETS later on.

Overall, integrating with a RETS server was a great experience in trial and error. As many things are in software, iteration was key in coming to the best solution to an interesting problem.