How to Find Off-Market New Constructions

Using open building permits data

Wherever you look, housing market is crazy. This is me pouring oil onto the flame — how you can use permit data to find new constructions near completion so you can put in an offer before it lists. You are welcome. It’s insanity out there.

Let’s use Seattle as an example, where homes sell within 7 days on the market.

Seattle actually has a really decent open data program.

The data you want to look for is the building permits. You can download the data and analyze/filter it with your favorite tool. But the built-in building permit map offers decent visualization and filtering off the bat.

Say you are interested in townhomes being constructed — we can set the filter to search for these keywords, and limit the permit issue date to be 2020.

Voila you have all the townhomes that has a permit issued in 2020. Sometimes the descriptions will even contain the orientation if you care about that very much. E.g., “Construct east single-family residence…”.

Once you narrow in on a particular construction, you can then pop over to the Permit and Property Records website and look up the permit number. This will give you a list of all permits filed associated with this address so you can stalk the progress and details of the construction.

Usually if sewer permit is already in place — it may be close to completion. But don’t quote me on that.

You can probably replicate the same process for other cities if they have a decent permit online system. And you can definitely write scripts to crawl/analyze/automate to create listings before they hit MLS/Redfin.

Good luck out there! It’s nuts.

Finally, for fun — a totally unscientific overlay of the townhomes construction permit map (issued in 2021 so far) overlaid with the light rail map. Ballard/Fremont/Wallingford, as well as Central/Beacon Hill definitely seeing a lot of actions.

Alright hope this gets more people interested in data science at least!

COVID Reported Crime Analysis [Part 1] Seattle

Overall crime down. Downtown is actually a lot safer.

In response to the recent news, I want to do a series of data analysis on the crime rate changes in the U.S. during COVID (focusing on NYC and Seattle where I spent time during the pandemic) [1]. I hope to first establish the overall picture of crime rate changes during COVID, and then if we somehow manage to find better data [2], track hate crimes against Asian American Pacific Islander communities.

(Feel free to reach out if you spot any bug in my code/analysis or have thoughts for gathering more accurate hate crime data.)


In Seattle area:

  1. Overall crime reduced by 8.5% YoY for 2020 (22.25K v.s. 24.31K in 2019). Crime against society (e.g., gambling, prostitution, and drug violations) had the biggest decline (-37.4%); followed by crime against person (-9.2%). Crime against property (e.g., robbery, bribery, and burglary) had the least reduction (-3.2%).

  2. Crime rates are hyper-local: it appears that crime rates in residential neighborhoods are more stable (e.g., Queen Anne); while more commercial areas see bigger changes (e.g, downtown, Capitol Hill).

  3. Media focuses on reporting increase in homicides (e.g., Massive 1-Year Rise In Homicide Rates Collided With The Pandemic In 2020). While homicides are horrifying, they are rare compared to “petty crimes” such as Burglary/Breaking & Entering, Theft From Motor Vehicle, Destruction/Damage/Vandalism of Property.


Overall Crime

Data Source: SPD Crime Data: 2008-Present [3].

Each crime is categorized by into crime against “person”, “property”, and “society”. See details here.

“Crimes Against Persons, e.g., murder, rape, and assault, are those whose victims are always individuals. The object of Crimes Against Property, e.g., robbery, bribery, and burglary, is to obtain money, property, or some other benefit. Crimes Against Society, e.g., gambling, prostitution, and drug violations, represent society’s prohibition against engaging in certain types of activity; they are typically victimless crimes in which property is not the object. ”

Below is the tracking of crime since 2008 by crime against category for the overall Seattle area. Crime against property, person and society actually all dropped in 2020 in Seattle.

Zooming in on 2019-2021. Overall crime reduced by 8.5% YoY for 2020 (22.25K v.s. 24.31K in 2019). Crime against society had the biggest decline (-37.4%); followed by crime against person (-9.2%). Crime against property had the least reduction (-3.2%).

When we plot the YoY change per month, specific events can be mapped onto the month. For example, crime against society clearly began to drop in March 2020 when the lockdown started. This trend is consistent with other studies such as “In Chicago drug arrests fell 42% since the lockdowns, compared with the same period in 2019” [4].


The impact of COVID on crime rate is also hyper-local — meaning that different neighborhoods have different patterns. Below are crime rate count for a few neighborhoods in Seattle.

Commercial neighborhoods such as downtown and Capitol Hill see more changes (not necessarily in the same direction); while residential neighborhoods such as Queen Anne seem to be more stable.

For example, Capitol Hill saw overall reduction of crime by 1% in 2020 (crime against property +10%, person ~0%, society -40%). There are spikes in June for crime against person and property mapping to the Capitol Hill Organized Protest time window (Jun 8, 2020 - Jul 1, 2020), and again in Sep and Oct 2020. But since things have quiet down.

The neighborhood saw biggest change is downtown commercial area. In 2019, downtown commercial area had the highest crime rate, followed by Capitol Hill, Northgate, and Queen Anne. In 2020, the ranking flipped to Capitol Hill > Downtown Commercial > Queen Anne > Northgate.

There is an overall 37% reduction in crime in downtown in 2020 YoY across all three categories (-39% against property; -26% against person; and -43% against society). The simple explanation seems to be that there is just no one there. Specific offenses that had a big decrease include: Theft From Motor Vehicle (-64%); Shoplifting (-60%)'; Trespass of Real Property (-50%).

In comparison, for the more residential neighborhood Queen Anne, there is a slight uptick in overall crime YoY by 3.7%, with increase in crime against property +5.6% and person +3%, and decrease in against society (-12.6%). The highest increase for crime against property comes from Theft of Motor Vehicle Parts or Accessories (+243%), Robbery (+46%), and Burglary/Breaking & Entering (+45%).


  1. Data and code available at


  3. There is a data anomaly in the crime dataset that had a spike in Fraud Offenses in May 2020. I have excluded it from the analysis.




  7. Massive 1-Year Rise In Homicide Rates Collided With The Pandemic In 2020

  8. How a pandemic and social unrest possibly drove up Seattle homicides in 2020

  9. Changed title from “Crime Analysis” to “Reported Crime Analysis” per suggestion by @heycori and reddit users.

Two Ways to Grow: Learning and Unlearning

Just as the cool Internet kids say, bundling and unbuilding are the only two ways to make money in business. In the past three months, I spent a lot of time thinking about a framework for personal growth. And after listening to too much Ben Thompson during long walks, the framework of learning and unlearning keeps coming back to me.

And, in short, I think unlearning is way underrated as a key way to grow.

The Learning

This is the easy part to sell in a framework of personal growth. There is no shortage of inspirational quotes to keep us motivated to continue to learn. In particular, those of us who choose to pursue a PhD is by definition very committed to learning. In particular, in the PhD program, we have a very particular way of learning, which is best illustrated by Matt Might’s famous illustration:

A fun paradoxical effect of learning is that the more you learn, the less you feel you know. Of course, this feeling is strictly wrong. But part of what’s beautiful about learning is learning to be humble and recognize the vastness of human knowledge and most importantly the limitation of that knowledge. That’s why inevitably we learn to use very hedging language in writing, such as, “there is strong evidence that…”. Part of the learning process is to recognize that things are complicated and having the capacity to understand the nuances.

The Unlearning

What is understated, therefore, is the process of unlearning.

tweeted a couple weeks ago that I had an aha moment at work (product engineering). In research, what makes a “good” problem is completely different from what makes a “good” engineering problem. In research, if the solution is obvious, it is not a very interesting problem to solve. However, in engineering, if the solution is obvious, that is one of the best problems to pursue because it is so feasible and potentially reliable and easy to maintain. But when I came in wearing the lens of a researcher as I was trained, none of the engineering problems felt very exciting — the solution is obvious, I would shrug and complain. I had to unlearn and adapt to see things in a different light. Now I find a lot of problems super exciting because I can see where I need to go and it’s only a matter of joyful heads down implementation from here.

So the point I am trying to make is — at certain point we need to unlearn the things that once we spent so much energy to learn and hold dear to. At certain point the things that we tried so hard to learn become a thing that holds us back from seeing other possibilities. If there is no conscious effort to unlearn things, there is the danger of getting stuck in a very weird place trying to fight an impossible battle while tiring ourselves out.

Learning and Unlearning HCI

As a thought experiment, let’s see how one can learn and unlearn Human-Computer Interaction (HCI). In building a new socio-technical system such as a social network, one way to break the problem down is what we should v.s. can build:

An engineering-centric view would lead to the focus on can outweigh should:

Learning HCI means recognizing that this imbalance causes a lot of societal problems, and that these technical systems do not live in a vacuum and we should consider the human factors while not getting too excited by what we can build. Learning HCI means being more thoughtful in thinking about what we should build.

However, after a year in industry, when I read my past CHI papers, especially the design section (which in all honesty, I usually threw up my hands in the air and hope my advisor could fill them in last minute), I can’t help but find the ideas in there full of naiveté. When I call on practitioners to do certain things, I didn’t even consider that a) only a very small fraction of these practitioners have even heard of CHI and b) there are realistic constraints about the metrics, the OKRS, the performance reviews, the politics, etc. on the line that may all occupy priority before my little design recommendation.

Unlearning HCI means recognizing and owning the fact that to make a real impact, a lot of other factors need to be considered as well, e.g., business constraints.

As much as it is tempting to criticize big tech for only caring about profits, there is a super power that those of us with research background aspiring to make real-world impact needs to learn — to actually find that product-market fit for our research insights or prototypes. Smart Reply is s great example.

It is a super power to jump through all the hoops to get an idea up and running in the real-world. And to learn this super power, we need to do at least a little bit of unlearning of the research purity. Maybe a little bit of Machiavellianism is useful. But then the question to ponder next will be:

What about principles and ethics? This might be the question I will chew on in my long walks to come. Let me know if you have thoughts.

Thanks for reading!


What a PhD Has and Hasn’t Taught Me

A (weird) year into industry

A year ago, I took a “leap of faith” to join a product engineering team instead of doing research right out of PhD (after figuring out that this is the hat that suits me best — see The HCI PhD’s Guide to Choosing a Hat). After such a weird year (thanks 2020), I am surprised by how many things are transferable out of PhD that I didn’t expect, and at the same time, how many skills that are lacking from the training. So here is my best attempt to provide some observations and comparisons between industry and academia (in particular, the PhD training). I am particularly incentivized to write this after coming across Syllabus for Eric's PhD students. Regardless of which side you are on, I hope you find the other side’s perspective interesting and useful.

First, on transferable skills — the fact that I am writing this on a Friday evening drinking wine and finding it actually really enjoyable is the result/proof of the most useful skill out of PhD — writing, fast and well, in that order. According to my advisor, I was a beautiful writer before the PhD program but I digressed too much. After I began to write paper with my advisor, however, the floral style gradually went away and I finally ended up writing most similarly to my advisor — and I suspect also my advisor’s advisor. There is a certain concise and logical style to it, although these days, I cut myself loose a bit. The secret to good writing, really, is quantity over quality. The fact that I am used to putting ideas down into words gives me a huge advantage in industry. This is especially true at a big organization such as Google, when people jokingly call themselves “doc”-engineer. You come up with ideas, great. Anyone in academia comes up with hundreds of ideas every day. It is only when you publish that you solidify these ideas under your authorship, your name, at the same time contributing the collective knowledge of science. Similarly, at big companies, writing is also a very scalable way of collaborating and leading.

Image may contain: 2 people, text that says 'The manuscript is very interesting, very well-written and well-argued. Recommendation: REJECT'

Via Facebook Group: Reviewer 2 Must Be Stopped

So writing was the obvious one. There was another transferable skill that surprised — peer review, on both ends. Let me explain. Now being the performance review season at companies, people seem to overall hate and struggle with the process much. At its core, performance review is similar to a peer review publication process, where you have peer reviewers (other researchers or colleagues, anonymous or not), meta reviewers (your manager or above), and meetings where people discuss the cases (calibration meetings and promotion committee meetings at companies where ratings and promo cases are discussed, and program committee meetings at conferences where borderline papers and awards are discussed). As a PhD student, you get familiarized, quickly hurt (unless you are just a genius in which case my hat tip to you), and then somewhat jaded about the entire process. Along the way you probably produce some memes about it too and there is a certain degree of dry humor and randomness to the process. On the flip side, as a peer, you also need to write paper or peer reviews, which take a significant amount of time and learning to do well (I still find Dan Cosley’s How I review papers to be the most useful guide.) All this is to say, in the peer review process you learn that presenting oneself accurately isn’t easy; there is some randomness to the process; you learn to be part of the process that tries the best to evaluate your peers and hopefully become a useful and constructive reviewer and eventually one of the people running the show. There is no need to take it personal. But at the same time, there are tricks to do it well.

There are other similarities too. Small details such as knowing how to produce and present a solid graph (always have a title; label your axes; always cover your axes if presenting; give people time to read the graph, etc.). Bigger things like time management; how to be productive in solitude (which turns out to be a very useful skill during a pandemic); how to do literature/background review so that when you face a new problem, you know where to begin; how to give a good talk and handle Q&A; how to network (going to conferences, elevator pitch, cold emailing); how to synthesize smaller ideas into a coherent story (your paper v.s. research statement/dissertation); how to come up with ideas; risk management (think about that paper that had so many rejections); handling rejections (this is a big one) not personally; community building (mentoring others and serving on conference committee); perseverance; open-mindedness and curiosity; taste.

On the flip side, I recently noticed how inadequately I am trained for other useful, or even essential life skills. I am still working on coming up with a comprehensive list but here are some starter ideas:

  1. How to have difficult conversations.

  2. Negotiation.

  3. Conflict management.

  4. The art of loving (reading the book by Eric Fromm now.)

  5. Personal finances and investing.

So these are the things I am spending time reading about and thinking these days. Let me know if you have pointers or thoughts. As I am truly stunned about how little attention I paid to these aspects of the adult development compared to the intellectual pursuit during PhD years.

Thanks for reading my rambles. I hope you have a great weekend. And cheers.

The HCI PhD’s Guide to Choosing a Hat

HCI PhD students wear many hats — including design, engineering, data science, product management on top of research. It is a blessing and a curse. The interdisciplinary nature of HCI gives freedom to do innovative research, not limited by artificial boundaries of disciplines. But at the same time, when interacting with industry — regardless of just dabbling for summer internships or deciding for the non-academic career path — it can be very confusing.

Which hat do I want to wear from now on?

And how do I make that decision?

I Want My Hat Back, Book by Jon Klassen

I couldn’t choose. So I tried them all. In my three internships, I tried qualitative UX, quantitive UX, data science/research. I ended up becoming an engineer and have been pretty happy in the role so far.

When I talk to friends, they often ask me about this decision. So here are my two cents to help anyone in similar situations to navigate. Hopefully you can find the hat that suits you the best.

What NOT to base your decision on

Let’s start with the easy ones. Do NOT base your decision on what you can do or good at now. Do NOT limit your options by thinking to yourself that you don’t have experience in a specific role. Compared to the entire career that lies in front of you (possibly decades long), the difference between having a few more years of experience prior or during PhD isn’t that big. This is a time to try new things, and gain experience. Do NOT set boundaries for yourself.

Second, do NOT think that certain roles are harder than others. They all require different skill sets and there are very successful people at the top in each role.

Last, do NOT be intimidated to think that others will always be better than you in certain roles. Luckily, we don’t have to be a genius in any of these roles to be successful. The skills and experiences in these types of roles are more learned than born with. If we are talking about arts, it’s a different game. But for most tech roles, talent won’t be a limiting factor at least couple more years down the road.

What I based my decision on

I don’t feel I have enough information to recommend what you should base your decision on. Instead, let me tell you about my considerations.

I have a rule that when I make small decisions, I base them on analytics — I look at reviews when picking restaurants, hotels, or a certain model of tech gear to buy. When it comes to big decisions, though, I base them on visualization. If I close my eyes, and visualize what myself and my life would be like in five years, in each path, which one does my gut feel I would be most happy in?

Then it became clear to me that I don’t want a path where when I look into the future, I can predict and see exactly what myself and my life would be like in five or ten years. (Side note — this made academic life a less exciting option for now.) For both qualitative and quantitive UX, I can see myself become reasonably good in the trade, maybe even leading a small team. Maybe I will teach on the side also at HCI departments in near by universities if that ever becomes a thing. But I can’t see uncertainties. It is too clear-cut to me and I can see to the end of it. (To clarify I find UX extremely important and respect people in this role a great deal.)

For data science, I can see myself gaining more experience, again, maybe leading a small team, publishing papers and maybe contributing to open-source projects, teaching, etc. But there is this urge to change something about the world, to build something, to be that change I want to see for tech, for the way products are built and designed. And sure, I can inform a lot of decisions, especially in data-driven organizations, but I know I will really itch for getting my hands dirty and building things from scratch.

Then there is the design. Honestly this is where I contradict myself a bit. I am no good at producing design, period. This is a classic case where my taste far exceeds my ability to execute. I love and appreciate good design, so much that I fully recognize I won’t be great making design my career.

Finally this leaves us engineering. I never imagined engineering to be my first real job. I had many misconceptions. In undergrad in China, people openly say that girls won’t be good at it. In grad school, engineering somehow is perceived as inferior to research. This is especially true for internships. Your advisor may encourage you to take a research internship, but maybe not necessarily a pure engineering internship where you just implement stuff. But when I pause and visualize five years down the road, engineering gives me a HUGE amount of freedom. I can continue to do research on the side and apply that in the products I build. I can decide to build something else. I can probably lead a team and build something that changes the world in certain ways. There are also many unknowns and possibilities that can open up down the road, such as product management, investing, etc. I don’t know for sure, which makes it the most exciting.

I will be honest. The first six months was tough. Coming to engineering from research is such a humbling experience. Last moment I was an expert on a (albeit) niche field. The next moment I was this dumb person who knows nothing and someone way younger than me has to teach me everything. That’s for a separate post though. But now close to a year on the job, I feel great. There are enough challenges and unknowns ahead and I feel very excited to learn and grow.

Hopefully this is helpful for you. I know these are tough times but please trust yourself. Good luck and let me know which hat you end up choosing. I hope you find the best one — or even better, find ways to not having to choose and continuing wearing them all.

Loading more posts…