Review timestamp, datetime data in Python

How many times did you get confused with datetime data? Have you been struggling too much every time data appears and you rigorously search on Google to find a solution or post your question on StackOverflow? The idea of this review is to provide a useful list of the frequently appeared issues with datetime. A gift for the new year 2021, working faster and with fewer problems.

Image by Chinh Le Duc on Unsplash

Types of Time?

In Python or SQL, the types of time we often see can be divided into a simple 2 categories: Datetime, Timestamp. Other materials or sources divide Datetime into smaller categories, but I prefer to keep things simple at this first step.

UTC or GMT?

Mentioning UTC or GMT, they belong to datetime type. The idea behind relates to timezone and standard time.

UNIX or EPOCH?

At this point, we are mentioning about Timestamp type. I will just quote a nice recap from textmagic.com:

import datetime as dt
import Pandas as pd

Convert datetime to timestamp

In some situations, you really need to convert datetime to timestamp. For example: I build a model in which I need to use a function in which the parameter only allows me to put timestamp format. In this case, I need to find a way to systematically /automatically convert datetime in my model.

dt.datetime(2020,1,1,0,0).timestamp()
making a list of timestamps
making a list of timestamps
Image by author

Convert timestamp to datetime

This point would be helpful when you want to convert your timestamp to datetime so that your data is more human-readable.

dt.datetime.fromtimestamp(1577829600.0)

How to get the time of the present

This is simple as it is. Depending on the time types you want, pass the code:

dt.datetime.now()
dt.datetime.timestamp(now)

In Pandas, changing the data type of a column to datetime

Very often you have to convert the data type to datetime. In many cases, the display of data looks like datetime, but it is understood by the system as ‘object’. It is important to convert the data type to datetime as it truly reflects the essence of our data, but more importantly, it influences our later analysis which requires datetime as the input. There are several useful ways to do this.

Force data type to datetime when you load the file

It is always simple and useful when you load the file into Pandas dataframe, you specify the data type using parameter ‘parse_dates=[column index]’.

pd.read_csv('data.csv',parse_dates=[2])

Change the data type of a column in Pandas data frame to datetime:

We have a Pandas data frame call df, column ‘Date’ which is now ‘object’ type, and we want to convert it to datetime. There are 2 ways to conduct it.

Method 1:

Change datatype using series.astype and overwrite back to the original data frame. This method is that it is easy to understand and compute. The downside of this method is that some report this way only works with Windows but not on Linux.

df.Date=df.Date.astype('datetime64')
df['Date']=df['Date'].astype('datetime64')

Method 2:

Using pandas.to_datetime method applied to our data frame, passing the code as below:

pd.to_datetime(df.Date)

Useful methods in Pandas to analyze time series

Resample method applied when you want to tidy your data for analysis by time. One point to note about resample is that it requires datetime data in data frame row index. This method is worth checking out HERE.

Visualization of data after “resample year”(Image by author)
visualization of data after “groupby hour” (Image by author)

I am Data Analyst and interested in logical thinking, technology & programming.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store