Data Science, Data Visualization, Programming

Recently I got myself obsessed with a Japanese tv show. I found myself cannot stop checking on Twitter, Instagram, and a Chinese app called Douban for updates and discussions about the show. In the meantime, I ran into an introduction article about the Python library Twint, which is very convenient in gathering twitter data. While it is torturing waiting for the new episode to come out every week, I decided to use the waiting time exploring Twint and derive some insights about the show from Twitter.

This article will discuss the use of twint, including how to installtwint, how to…

It is estimated that 80% of the world’s data is unstructured. Thus deriving information from unstructured data is an essential part of data analysis. Text mining is the process of deriving valuable insights from unstructured text data, and sentiment analysis is one applicant of text mining. It is using natural language processing and machine learning techniques to understand and classify subjective emotions from text data. In business settings, sentiment analysis is widely used in understanding customer reviews, detecting spam from emails, etc. This article is the first part of the tutorial that introduces the specific techniques used to conduct sentiment…

Office Hours

I have listed the technical questions to practice in machine learning, statistics, and probability theory in my previous articles regarding data science interview preparations. I have also discussed the strategies that can be used to prepare case study questions before and during data science interviews. This article is the fifth article of the data science interview preparation series, and it will be focused on behavioral questions. I will first discusses how to prepare the behavioral round and then list some frequently asked questions for you to practice the techniques.

I like the behavioral round the most in the interview process…

You have probably heard about Bayes’ theorem from statistic classes. The magic formula based on conditional probability unlocks a novel way of interpreting the probability of an event. In this article, I will discuss the basics regarding Bayesian Statistics, and what is the difference between Frequentist and Bayesian in terms of statistical inference. Moreover, some applications of Bayesian statistics.

What is Bayesian Statistics

Bayesian statistics is a statistical theory based on the Bayesian interpretation of probability. To understand Bayesian Statistics, we need to first understand conditional probability and Bayes’ theorem.

1. Conditional Probability

Conditional probability measures the probability of an event occurring based on the…

Making Sense of Big Data

Seasonality, as its name suggested, refers to the seasonal characteristics of the time series data. It is the predictable pattern that repeats at a certain frequency within one year, such as weekly, monthly, quarterly, etc. The most straightforward example to demonstrate seasonality is to look at the temperature data. We always expect the temperature to be higher in the summer while lower in the winter in most places on Earth.

Taking seasonality into consideration is very important in time series forecasting, such as demand forecasting. For example, we may expect ice cream sales to have seasonality since the sales will…

Gradient Descent is an algorithm that solves optimization problems using first-order iterations. Since it is designed to find the local minimum of a differential function, gradient descent is widely used in machine learning models to find the best parameters that minimize the model’s cost function. This article will discuss the gradient descent algorithm in detail, and the next article will discuss its application in linear regression models.

The algorithm

As mentioned above, gradient descent solves minimization problems, suppose we have an optimization problem:

This article is the fifth part of my “nail the data science interviews with confidence” series. In the previous articles, I was mostly focused on discussing technical questions in machine learning, statistics, probability theories, and case studies during the data science interview. It is important to not only showcase your hard skills but also essential to demonstrate soft skills like communication skills during interviews. As traditional in-person interviews have been widely suspended due to the pandemic, how to present the best of yourself over the screen is a challenging task for all interviewees. During the past months of job-hunting, I…

2020 was probably the worst year for job hunting, especially as a new grad. Being in the same job-hunting pool with many experienced talents competing for limited job opportunities has been a nightmare for me. Luckily, after 7 months of constant job hunting, I finally got an offer I am super excited about and satisfied with. I want to thank myself for always having the strength to put myself together, even facing the rejections over and over again. How to deal with failures is the most important lesson I have learned from 2020. …