Data partitioning is a strategic approach used in database management, which involves segmenting a large dataset into smaller, more manageable subsets for improved efficiency and accessibility.

Think of it as an organized library; instead of having all the books randomly scattered, they’re sorted into categories and sections, making it easier to locate a particular book you need to use in your research.

Similarly, data partitioning helps categorize and store large volumes of data, enhancing query performance, reducing system load, and facilitating more effective data management. Partitioning the data you import to Pecan can help your queries run faster, and your models generate predictions quicker.

To maximize these benefits, it is important to select the right partitioning strategy.

You can partition your data in Pecan regardless of whether your data is partitioned in your organization and what is your partitioning strategy.

Pecan allows you to partition your data horizontally, meaning - divide your table into subsets based on the values of a specific column.

Pecan supports independent partitioning by columns of type Timestamp, String, and Date.

We’ll use the following table as an example. The table holds three columns, each of a type Pecan supports for partitioning.

Note: a real table might hold additional columns with different types. As long as you have a column of one of these data types, you will be able to use them for partitioning.

Columns of type timestamp can be used to split your data into partitions (subset), where each partition will hold one day of data.
For example, if you select transaction_time as your partitioning column, the data will be organized as follows:

Columns of type date can be used to split your data into partitions (subset), where each partition will hold one day of data.
For example, if you select install_date as your partitioning column, the data will be organized as follows:

Columns of type string can be used to split your data into partitions (subset). In this case, every unique value in the column would create its own partition.
For example, if you select username as your partitioning column, the data will be organized as follows:

In cases like an especially large table, you might want to use more than one level of partitioning.
Pecan allows partitioning by up to two data types.
Note the order of the columns is important. To make sure you maximize the benefits of this feature, Pecan recommends the first column to be the one of the two that will result in a smaller amount of partitions.

Following the above examples, between partition_time (4 partitions) and username (7 partitions), it would be best to select partition_time as the first partition column.

You can decide to partition your data once you import it for the first time or apply partitioning to data you've already imported - the process is the same.
​

First, go to the connections tab, and select the connection with the table you'd like to partition.
​

Remember: when using two columns, the order has an impact.
​
​

If you’d like to customize the way your data is partitioned, open a support ticket with details on how you wish your data to be organized, and the Pecan team will handle it for you.