Ahh…documentation. It may not be the most exciting topic, but everyone notices when it’s badly done. Documentation often gets neglected because we’re all pushing to move on to the next project. But it can be a powerful tool when done well – demystifying a data platform is one of the single most important things we do when we first come into a business. Seeing a crisp set of documentation that has all the vital information just makes you want to dive in and be part of the team.
That’s why it’s so important to build good documentation practices in your data engineering practice. No more information silos. (No more panicking because Steve’s last day was Friday and he was the only one who knew what was going on with that 5,000-line SQL script.) Newcomers to the team get onboarded smoothly and ramp up more quickly and interrupt experienced team members less often.
There’s no doubt about it – great documentation is a huge trust-builder that encourages excitement and engagement for what is being built. But does all of the above sound like an impossible dream? Read on, it can happen if you keep these tips in mind!
Best practices for documentation
1. Centralise your documentation so it’s easy to search and discover
Whatever tool or platform you end up using, make sure it’s a single, centralised tool that is accessible for its intended users.
2. Use clear and consistent formatting.
It also helps to maintain a logical information hierarchy and folder structure and have nifty features like a clickable table of contents, keywords and search functionality to keep documentation easy to navigate. Use consistent and clear headings, bullet points and formatting to make it succinct and readable.
3. Know and prepare the right level of documentation for the situation.
Generally, there’s two main types of documentation data teams need to consider and prepare.
Business processes
This might be built out by a project team, business analyst or stakeholder, usually written in plain english. If it’s available, a data team should reference this rather than reinvent the wheel. If not available, the data team should produce a high-level one-pager that business stakeholders can understand with a process diagram.
Technical processes
In order to support the above business process, we need a series of steps that may not be obvious to data/business consumers to support the data enrichment with accurate information.
a) Firstly, we need a mechanism to ingest the payload from the webhook and route the event data from the lead generation campaign and send it to a place where the enrichment takes place.
b) The source of the data enrichment will be from a data warehouse - there’s no point creating this documentation for each example so we should have that specified somewhere once and linked to wherever it is used in a process.
c) Then, we articulate the process of enrichment.
d) From here, we add the email to the lead nurture sequence again via some form of API interaction.
4. Surface the most important information.
Keep documentation succinct and practical, focusing on what's necessary for understanding and using the data systems. The below are some of the types of information I like to document each time I start a project – but of course, your mileage may vary.
How much information to include, and how detailed it needs to be will vary depending on the length, complexity and scope of the project.
Overview
In many cases it is Confluence, though it could be Notion or any other wiki-style platforms. Paint a picture of the technology landscape. What cloud service is used? What database technologies, what orchestration tools are in place? This is a good place to start. Try to keep it at a high level and provide succinct information that a business executive can quickly understand.
Stakeholders, key contacts, SysAdmin support
It’s always a good idea to have a functional org chart of who is in the team. List out stakeholders and customers (internal or external) as well as any other teams that the project has interdependencies with.
Business problem that the product/solution is trying to solve
In the overview, we provided a bird’s eye view. In this section, go a level deeper and provide information at a level of detail that a business practitioner such as a Product Manager would want to know.
How solution/product operates
At this level, we start getting a bit more technical and gets into more granular detail around business processes and logic. Rather than reinvent the wheel, see if you can leverage and link to documentation already in existence that may have been created by the Business Analyst/Project team.
Technical solution architecture overview
This piece of documentation outlines the high-level technical solutions that are needed without going into the specific technologies (e.g. a queue is needed, rather than SQS specifically, or an orchestration tool, rather than Airflow or Mage).
Technology evaluation
This part is not always necessary, but if the situation calls for it provide an evaluation of what vendor or technology to use, comparing any features, benefits and pros and cons. e.g.
Snowflake vs Databricks vs BigQuery
PostgreSQL vs MySQL vs Oracle
Airflow vs Mage
Solution design
This is where you’ve locked down the architecture and design choices and are ready to start building. By this point, you should have nailed down a fairly accurate plan, even if variations may occur later on. Think of it as architectural drawings at the start of a home building project.
Business As Usual (BAU) documentation
Data sources and pipelines
Provide a succinct explanation of what the pipeline does.
Lay out all of the various data sources you have in the business - this could be SaaS apps like mailchimp and salesforce or databases within your business like Postgres, Oracle or SQL server. Tabularise this and add important information, such as what domain and business unit the data is for.
Link to the repo(s) where the code is stored.
If available, link to the platform engineering wiki where team owner information and KPIs are stored.
Explanation of procedures Here is a good opportunity to outline playbooks and ceremonies within your team. For example:
If there’s an outage or incident, how do you capture what happened? How do you organise retrospectives to learn from it? What is your continuous improvement plan?
What are your team rituals? Standups, code reviews, one-on-ones, quarterly reviews, sprint planning and retrospectives.
What is the process for onboarding new work or pipelines?
Diagrams, screenshots and visual aids
Throughout your documentation, include diagrams when helpful. A visual cue is often a great way to communicate complexity in a simple manner.
Types of diagrams include:
Architecture
Process flow
Data flow
Technical flow.
Include screenshots and code snippets liberally while ensuring not to expose any credentials.
SLAs
It’s a good idea to articulate your SLA’s and how you are able to support them. This often requires a negotiation of team capacity and on-call capabilities.
Security information
While we should never store credentials in documentation, security is a day 0 concern and must be considered prior to starting on the project. it’s a good idea to articulate security practices that are in place, including:
What password manager are you using?
Where do you store credentials for use in applications (for example, is it Secrets Manager or Parameter Store in AWS?)
What least privilege do you use?
What do you do when a breach occurs?
Templates and example code and payloads
The way to continuously improve as a team is to keep things DRY: “Don’t Repeat Yourself”. Leverage templates for frequently used code such as:
Frontend templates
Backend ci/cd pipelines
A standardised way to deploy terraform and to name aws services
Code blocks for things like exponential backoff, which can be leveraged throughout your code base.
Good documentation is ultimately down to good culture and leadership
In the typical busy day in the life of a data engineer, good documentation practices won’t magically happen unless it is made an explicit and integral part of the engineering culture. This means that engineering leadership should set clear expectations for contributing and maintaining documentation. If you don’t stress the importance of documentation to your team members, or do a good job of modelling this practice, you can’t expect your team to build this habit.
Below are some ideas to help nurture a robust documentation culture.
Training and knowledge sharing: encourage knowledge-sharing sessions to discuss best practices and tools for documentation.
Documentation testing: Don’t let the first time that documentation is used be in an emergency. Test and tweak your documentation. I’ve heard of teams who implement game days where they have engineers unfamiliar with the particular project or service test how effective the documentation is in a real-life setting, which is a great way for the team to have fun while achieving pragmatic outcomes.
Continuous improvement: Regularly review and refine documentation practices to adapt to new challenges and technologies.
Cultivating a documentation culture doesn’t just improve operational efficiency and reduce knowledge silos; it builds a foundation for sustained success, a shared language that helps teams navigate the complexities of data engineering. Good luck as you continue to write, build, and repeat!
Further reading and resources
If you’re after further reading or inspiration about documentation, I’ve found the following resources helpful.
At Cloud Shuttle, we create effective, best-in-class documentation as part of all our consulting engagements. If you need support on a project with documentation that needs a complete overhaul (or even just some fine-tuning), we can help. Contact us today and we can help you build a solid foundation for your data engineering projects to thrive.