I wrote this in early 2019, about a year after joining a mobile game studio as its first data scientist. The studio had no internal data science or business intelligence setup yet, and I was part of a small team building one.

The work put me into digital marketing for the first time. I was pulling rich user-level and device-level data from marketing platform APIs and from the games themselves. It was my first real experience building data pipelines, from collecting the data to building a warehouse, dashboards and predictive models. Here is what I learned.

1. Choosing tools is a balancing act

You need some knowledge of what technology is available before you can plan how a pipeline fits together. I had to weigh time, people and cost when choosing tools and planning the work. These were the questions I asked before building anything:

Taking ownership of the pipeline meant making decisions based on my own technical skills and on what the organization needed. Buying tools that fit the studio’s existing workflows, while staying within budget, mattered just as much.

2. Understand your files and how you store them

Anyone who works with data may find this obvious, but I learned not to take it for granted. I started paying much closer attention to how a CSV file separates one value from the next and how it marks where each value sits.

CSV files are often messy. A column can hold unpredictable strings, and those strings can contain the same character as the delimiter. You either handle that across the whole file, or you check whether your ingestion tool can tell a real delimiter from the same character inside a quoted string. The comma is the most common delimiter, and I have lost count of the times I have seen it inside a string as well.

Good file storage means organizing data so it is easy to find and load for the way it will be used. If the data is time-based and likely to feed a time series analysis or a historical chart, put the date in the file name. This pays off as the data grows. A live sales database probably will not hold data from three years ago, but you may want that range for a later analysis, and a dated file name makes it easy to find and load.

3. Learn the metadata

When you load data into a database, you need to know the data that describes your data as well as the data itself.

For example, one table I built held phone model names. I assumed no name would be longer than 15 to 20 characters, something like “iPhone X” or “Samsung Galaxy Note”. I was wrong. One device name was 72 characters long. Building tables properly means understanding what each column holds, its data type, and how much space it needs.

When the data comes from another system or an API, read the documentation. Learn their metadata in detail so you can map their types and schemas correctly into your own tables.

The next point is the opposite, and it is something I would not recommend in practice. I did it while learning because I had nothing better.

Sometimes you get data you have never seen and cannot work out the type ahead of time. In one case I tried to store a 90-character string in a varchar(75) column, and the load failed. I had no way to know the maximum length in advance. With poor documentation or a shaky upstream design, it can make sense to leave room. If I had never seen a string longer than 100 characters, I would set the column to varchar(200).

You might wonder how a business survives without proper data architecture. Many businesses do well on their core product alone. Rebuilding internal data infrastructure can take years, and some companies do not see the benefit or do not treat it as urgent. If it is not broken, why fix it?

4. Build error handling and alerts into the pipeline

One of the better parts of pipeline work is being able to see whether each load succeeded. I learned to set things up so that when an error happened, I heard about it right away and could fix it quickly.

One pipeline I built downloaded data from an API and loaded it into a database table. Sometimes the Python script could not reach the provider’s server and got a server error. I added error handling to the script so the API call could simply be retried without touching the code.

I also found that monitoring could fit into the way the team already worked. Most of our communication happened in Slack, and I realized Slack could be how my scripts talked to me too. I added a small snippet to each ingestion script that sent me a Slack message whenever an error occurred, with the source of the error, the reason, and how to fix it.

5. Validation is a team effort

There are many points along a pipeline where people can protect the quality of the data. Building on the previous point, alerts on the engineering side can also confirm that the right number of rows and columns arrived in the database. Any load errors can be sent as alerts too, so you know all the data made it.

On the analysis side, it helps to have experienced, numbers-minded people on the team looking at the data through dashboards and reports. An experienced person can spot a small shift in a trend quickly. It is good practice for them to speak up when their gut says the data looks wrong.

The fewer manual checkpoints a pipeline needs, the more time goes into actually using the data. Still, checks at several stages do not hurt. Done well, they make the data more accurate and more trusted by the people who rely on it.

Looking back from 2026

The questions in the first section still hold up well. Who owns it, what it costs, who uses it and how fast they need answers are still the questions that decide most data platform choices. The difference is that the answers are cheaper now. A managed cloud warehouse such as Snowflake, BigQuery or Redshift Serverless is the default, and a small team can stand one up in a day.

On ingestion, I would no longer hand-write a script for each marketing API. Managed connectors (Fivetran, Airbyte, or the platform’s own export) handle the common sources, including retries and schema changes. Custom scripts are for the sources nobody else supports, and those would run under an orchestrator such as Airflow or Dagster rather than on a timer.

On files, I would land raw data as Parquet in object storage, partitioned by date, rather than relying on dated CSV file names. The idea of making time easy to find is the same, but the format carries its own types and avoids the delimiter problem entirely. When a CSV is unavoidable, a proper parser with quoting rules handles commas inside strings.

The varchar(200) workaround is mostly unnecessary now. Modern warehouses store strings without a meaningful length penalty, so I would use an unbounded string type and put the effort into data contracts and schema tests instead. Tools like dbt tests, or a check that compares row counts between source and destination, make section five’s validation automatic rather than something a person has to notice.

The Slack alerts were a good instinct and still the right idea. Today they come from the orchestrator’s failure hooks or an observability tool, so every pipeline gets them by default. The human check in section five still matters, though. Automated tests catch what you thought to test for, and experienced people still catch the rest.