What Is Data Engineering?

Core concept

Preparing data for meaningful use

Data engineering connects the data produced by everyday activities with the people and applications that need it.

Raw data passing through data engineering work to become useful information — Figure 2. Data engineering prepares raw data so that it can be used reliably.

A working definition

Data engineering is the practice of designing and managing the processes that collect, organise, prepare and deliver data for meaningful use.

Raw data may be incomplete, inconsistent, scattered across different systems or difficult to access. Data engineering helps make that information dependable and available to authorised users.

Four ideas to remember

Data is generated

People, applications and devices continuously create records.

Data needs preparation

Raw information is not automatically ready for use.

Data must be dependable

Users need accurate, complete and timely information.

Data creates value

Prepared data supports services, decisions and innovation.

Think about it: If a university has thousands of student records but cannot find the correct information when needed, does it truly have useful data?