Skip to content

Amundsen Frontend Strategy

A vision and strategy for Amundsen’s Frontend

1. Overview

Amundsen’s UI hasn’t had a lot of updates on our front-end application in the last year. As a side effect, the code is suffering from stagnation. The team had to hold off on solving some of the user experience and quality issues due to limited resourcing. This is a problem because this can slow velocity and critical updates become more urgent if we don’t focus our limited resources in the right direction.

One thing we can do is to create a technical strategy that sets a direction for contributors and maintainers on the front-end that allows them to align, make quick, confident decisions and improve the effectiveness of the resources we have.

Table of Contents

2. What

A technical strategy is a tool of proactive alignment that empowers teams to move quickly and with confidence. A great technical strategy identifies a problem with the current situation, proposes a principled approach to overcome it, and then shows you a coherent roadmap to follow.

This Strategy will break down into three parts: * Where are we now (Current State) * Where do we want to get to (Target State) * What are we doing next (Next Steps)

3. Where are we today

To create a strategy for Amundsen’s UI in the near future, we need to know where we are today. Here we’ll describe it with the following considerations: * User Experience - How our final users experience the UI and how well they can perform their tasks. * Codebase Quality - The quality of the codebase, how fast we can extend it with new features without hurting stability. * Developer Experience & Collaboration - How easy is for new contributors to start helping in the project.

3.1. User Experience

We lack consistency and a polished user experience. We can summarize it on these points: * Inconsistent typography * Inconsistent buttons and links * Not a11y WAG 2.1 compliant * Inconsistent error handling * Limited “browsing” experience * Bulky components & layout * Homepage page is not tailored to user role * Cluttered metadata on the left panel of the Table Detail page * Differing list components across different pages and even within the same page across different tabs

3.2. Codebase Quality

There are some aspects where the current codebase is sub-par: * Many old ESLint issues still around, like: * Not destructing props * Usage of setState within componentDidUpdate * Human-readable strings hardcoded on JSX code * Sub-optimal state management * Scattered and inconsistent Redux logic for extracting info * Complex and boilerplate full Redux logic * Incomplete testing * Outdated unit testing library (Enzyme, 107 files) * Incomplete unit test coverage * No end to end testing * Scattered URL logic * Sub-par API * No API type Verification * Non-cleaned deprecated endpoints * Inconsistent functionality between endpoints for different resources * Many bugs in the codebase * No code health monitoring * Deprecated component library * Outdated React Version (v16)

3.3. Developer Experience and Collaboration

Developers bump into friction when dealing with the following issues: * Scattered fixture data for our unit tests * Hard to figure out typography * Inconsistent and incomplete instrumentation * No test coverage tracking * Incomplete Documentation * Incomplete frontend configuration documentation * No code style guide * No naming conventions * Outdated Docker image for getting started with Amundsen for OSS * Missing showcase of configurable features * Outdated versions of all packages * No PR Previews * Complex and brittle developer environment setup * Custom React application on top of Flask

4. Where we want to be

Now that we have established where we are today (as of Jan 2023); let’s now describe where we want to be within a medium to long term. We will repeat the previous grouping:

4.1. User Experience

  • Consistent typography
  • Consistent buttons and links
  • A11y WAG 2.1 compliant
  • Consistent and efficient error handling
  • Delightful “browsing” experience
  • Information-dense components & layout
  • Tailored and configurable homepage
  • Tidy Table Detail page metadata
  • Consistent list components

4.2. Codebase Quality

  • No ESLint warnings
  • All human-readable strings as constants
  • Optimal state management
    • Testable and extensible Redux logic for extracting info
    • More maintainable Redux logic
  • Thorough testing
    • Using React Testing Library
    • Fully tested codebase (80%+)
    • With end to end testing
  • Centralized URL logic
  • Pristine API
    • Tested API type verification
    • Non-deprecated endpoints
    • Consistent functionality between endpoints for different resources
  • Bug-free code
  • With codebase health monitoring
  • Up to date component library
  • Up to date React (v18)

4.3. Developer Experience and Collaboration

  • Extendable fixture data we can use to manually test the UI
  • Limited typography options
  • Thorough and extensible instrumentation
  • Easily tracked test coverage that breaks builds when reduced
  • Thorough Documentation
    • Thorough frontend configuration documentation with examples
    • Complete code style guide
    • Naming conventions
  • Up to date Docker image for OSS
    • Including showcase of configurable features
    • Updated versions of all packages
  • Previews with PRs
  • Easy developer environment setup
  • React framework (Next.js) independent from flask API

5. Current State and Vision Summary

Consideration Current State Target State
User Experience Inconsistent typography Consistent typography
Inconsistent buttons and links Consistent buttons and links
Not a11y WAG 2.1 compliant A11y WAG 2.1 compliant
Inconsistent error handling Consistent and efficient error handling
Limited “browsing” experience Delightful “browsing” experience
Bulky components & layout Information-dense components & layout
Homepage page is not tailored to user role Tailored and configurable homepage
Cluttered metadata on the left panel of the Table Detail page Tidy Table Detail page metadata
Differing list components across pages or tabs Consistent list components
Codebase Quality Many old ESLint issues still around, like:
  • Not destructing props
  • Usage of setState within componentDidUpdate
No ESLint warnings
Human-readable strings hardcoded on JSX code All human-readable strings as constants
Sub-optimal state management
  • Scattered and inconsistent Redux logic for extracting info
  • Complex and boilerplate full Redux logic
Optimal state management
  • Testable and extensible Redux logic for extracting info
  • More maintainable Redux logic
Incomplete testing
  • Outdated unit testing library (Enzyme)
  • Incomplete unit test coverage
  • No end to end testing
Thorough testing
  • Using React Testing Library
  • Fully tested codebase (80%+)
  • With end to end testing
Scattered URL logic Centralized URL logic
Sub-par API
  • No API type Verification
  • Non-cleaned deprecated endpoints
  • Inconsistent functionality between endpoints for different resources
Pristine API
  • Tested API type verification
  • Non-deprecated endpoints
  • Consistent functionality between endpoints for different resources
Many bugs in the codebase Bug-free code
No code health monitoring With codebase health monitoring
Deprecated component library Up to date component library
Outdated React Version (v16) Up to date React (v18)
Developer Experience and Collaboration Scattered fixture data for our unit tests Extendable fixture data we can use to manually test the UI
Hard to figure out typography Limited typography options
Inconsistent and incomplete instrumentation Thorough and extensible instrumentation
No test coverage tracking Easily tracked test coverage that breaks builds when reduced
Incomplete Documentation
  • Incomplete frontend configuration documentation
  • No code style guide
  • No naming conventions
Thorough Documentation
  • Thorough frontend configuration documentation with examples
  • Complete code style guide
  • Naming conventions
Outdated Docker image for getting started with Amundsen for OSS
  • Missing showcase of configurable features
  • Outdated versions of all packages
Up to date Docker image for OSS
  • Including showcase of configurable features
  • Updated versions of all packages
No PR Previews Previews with PRs
Complex and brittle developer environment setup Easy developer environment setup
Custom React application on top of Flask React framework (Next.js) independent from flask API

6. Next Steps

We have seen where we think we are right now, and where we want to be in the future. The changes are many, so what are our next steps? In this section, we describe the changes we’ll need to introduce to take us to our vision.

6.1. Prioritized Milestones

From the next steps task prioritization (see Appendix B), we have grouped tasks into Milestones that will target specific metrics. Here is our proposal:

6.1.1. Milestone 1: Increase OSS contributions and remove critical team devex issues

Metrics: * Increased number of contributions (in PR numbers) * Team time savings (estimation)

Success Condition: We increase the monthly contributions by 20%

Tasks: 1. Update Docker image for OSS - Adding a showcase of configurable features 2. Fix developer environment setup - Create a document of the current setup that works 100% of the times using Lyft’s tooling and Python version 3. Improve error handling - Provide useful Metadata, and API error messages 4. Add end to end testing - running them on the development machine and covering the main user flows 5. Improve Documentation - Improve our frontend configuration documentation 1. Complete docs with all the rest of config options 2. Add examples to all the options 3. Structure docs 6. Get PR previews working and usable - with extensive test data 7. Improve test coverage - 100% test coverage on config-utils and util files 8. Improve Documentation - Create a code style guide and naming conventions

6.1.2. Milestone 2: Eliminate main User reported issues

Metrics: Customer satisfaction (Surveyed)

Success Condition: We increase customer satisfaction by 20% (NPS score or other metric)

Tasks: 1. Improve error handling - Provide useful or actionable UI error messages 2. Organize the left sidebar of the Table Detail Page - Move information into expandable blocks 3. Organize the left sidebar of the Table Detail Page - Declutter features 4. Squash all outstanding bugs 5. Improve error handling - Improve user experience when token expires

6.1.3. Milestone 3: Reduce dependencies issues and improve testing

Metrics: Team time savings (estimation)

Success Condition: We have no issues when bumping OSS project versions

Tasks: 1. Update Docker image for OSS - Update versions of all packages 2. Fix our developer environment setup - Simplify setup if possible 1. Align internal and external python package versions as much as possible for easier installation and pyenv setup 3. Add end to end testing - Running on CI and covering secondary user flows 4. Improve test coverage - 90% test coverage on presentational components and 80% overall test coverage

7. Appendix A: Prioritization Process

After we synced on where we think we are right now, and where we want to be in the future, we got many tasks. So what are our next steps? In this section, we describe the changes we’ll need to introduce to take us to our vision.

We will add an extra category to the improvements: their benefits to our users, and to the OSS team. For that, we will include these labels:

  1. <UT> - Improvements that save time to our users
  2. <DEXP> - Improvements that save time to our team
    • By reducing the maintenance burden
    • By reducing the time to create features and debug issues
    • By reducing friction for new OSS contributions, effectively getting more help

Let’s see the improvements with the initial categories and their benefits:

7.1. User Experience

  • Improve our typography - <DEXP>
    • Remove old classes
  • Improve our buttons - <DEXP>
    • Audit our buttons in Storybook
    • Audit our links in Storybook
    • Update all our buttons and links to be consistent
    • Add missing styles
  • Improve our a11y - <UT>
    • Audit a11y
    • Fix a11y issues
    • Setup automated testing to catch regressions
  • Improve error handling - <DEXP>, <UT>
    • Handle errors at the right level and with useful feedback
    • Improve experience when token expires
  • Improve our browsing experience - <UT>
    • Complete our transition to faceted search on the search page
    • Add a basic browse experience to complement the faceted search
  • Bulky components & layout - <UT>
    • Update our components to be more compact
    • Update our layout to be more compact
    • Update table detail page left panel to look more orderly when a table has many of the ‘optional’ fields displayed.
  • Homepage page is not tailored to user role - <UT>
    • Move our announcements section into a drawer that opens from the global header
    • Improve our homepage layout to contain more customizable widgets
  • Organize the left sidebar of the Table Detail Page - <UT>
    • Move information into blocks
    • Reorganize the blocks and open those more important for our users
  • Make the lists/tables consistent - <DEXP>
    • Decide for one way of listing elements
    • Roll it out all over the UI

7.2. Codebase Quality

  • Fix all the ESLint warnings - <DEXP>
    • Destructuring
    • SetState issues
  • Improve State Management - <DEXP>
    • Create Redux selectors whenever we can
    • Explore using Redux toolkit to simplify our Redux logic
  • Improve Testing - <DEXP>
    • Migrate Enzyme tests into React Testing Library <DEXP>
      • Audit work
      • Create tickets and find support
      • Migrate tests
    • Improve test coverage - <DEXP>
      • 100% test coverage on config-utils
      • 100% test coverage on util files
      • 90% test coverage on presentational components
      • 80% overall test coverage components
    • Add end to end testing - <UT>, <DEXP>
      • Cover main user flows
      • Cover secondary user flows
      • Ran on CI
  • Centralize URL logic - <DEXP>
    • Create util file with static URLs
    • Extend with dynamic URLs
  • Improve our API - <DEXP>
  • Squash all the bugs - <UT>
    • Critical
    • Major
    • Minor
  • Add codebase health monitoring - <DEXP>
    • Find a free solution for OSS (code climate?)
    • Introduce it
    • Break builds with degradations
  • Update component library - <DEXP>
  • Update React version - <DEXP>
    • Depends on moving to React Testing Library
    • Update codebase using codemods

7.3. Developer Experience and Collaboration

  • Fix our fixture data - <DEXP>
    • Create test data builders for all our fixtures
    • Create a debug mode to use the test data for manual testing of features
  • Easy to use text component - <DEXP>
  • Improved instrumentation - <DEXP>
    • Actions with analytics payloads
    • Make sure all feature utilization is accurately tracked 4
  • Test coverage tracking - <DEXP>
    • Use codecov to track test coverage
    • Use codecov to break builds when reducing by a given %
  • Improve Documentation - <DEXP>
    • Improve our frontend configuration documentation
      • Complete docs with all the rest of config options
      • Add examples to all the options
      • Structure docs
    • Create a code styleguide
    • Create naming conventions
  • Update Docker image for OSS - <DEXP>
    • Add showcase of configurable features
    • Update versions of all packages
  • Get Uffici tool merged and usable - <DEXP>
  • Fix our developer environment setup - <DEXP>
    • Create a document of the current setup that works 100% of the times using Lyft’s tooling and Python version
    • Simplify setup if possible
  • Migrate into using Next.js - <DEXP>
    • Pick between the many approaches to incrementally adopt Next.js
    • Move out the application part by part into Next.js keeping Flask as the API server

7.4. Next steps (by cost/impact)

We have seen the gap we’ll need to cover in each consideration group and our vision for them. Now, we’ll look at the work from an impact point of view. We will prioritize things with the highest leverage and tasks we should do first to make others easier later.

For that, we have estimated the impact (from 0 to 5) and cost (T-Shirt sizing), and prioritized the improvements in the following list:

7.5. Prioritized Improvements

  1. Update Docker image for OSS - <DEXP> | Impact - 4 5 5 5 - 5 | Cost - S

    1. Add showcase of configurable features
    2. Update versions of all packages

  2. Fix our developer environment setup - <DEXP> | Impact - 4 5 5 4 - 4.5 **| Cost - M**

    1. Create a document of the current setup that works 100% of the times using Lyft’s tooling and Python version
    2. Simplify setup if possible
      1. Align internal and external python package versions as much as possible for easier installation and pyenv setup

  3. Improve error handling - <DEXP>, <UT> | Impact - 4 4 4 3 4 - 4 **| Cost **- M

    1. Provide useful Metadata error messages
    2. Provide useful API error messages
    3. Provide useful UI messaging
    4. Improve experience when token expires

  4. Organize the left sidebar of the Table Detail Page - <UT> | Impact - 5 3 5 4 4 - 4 **| Cost - M**

    1. Move information into blocks
    2. Reorganize the blocks and open those more important for our users
    3. Later: declutter

  5. Add end to end testing - <UT>, <DEXP> | Impact - 4 5 4 - 4.5 **| Cost **- L

    1. Depends on Docker image working
    2. Ran in Dev machine
    3. Cover main user flows
    4. Cover secondary user flows
    5. Ran on CI

  6. Squash all the bugs - <UT> | Impact - 4 4 3 5 4 - 4 **| Cost **- S

    1. Critical
    2. Major
    3. Minor

  7. Improve Documentation - <DEXP> | Impact - 5 4 4 3 4 - 4 **| Cost - M**

    1. Improve our frontend configuration documentation
      1. Complete docs with all the rest of config options
      2. Add examples to all the options
      3. Structure docs
    2. Create a code styleguide
    3. Create naming conventions

  8. Get Uffici tool merged and usable - <DEXP> | Impact - 4 3 3 4 - 3.5 **| Cost - M**

    1. Depends on the docker task

  9. Improve test coverage - <DEXP> | Impact - 4 3 - 3.5 **| Cost **- M

    1. 100% test coverage on config-utils
    2. 100% test coverage on util files
    3. 90% test coverage on presentational components
    4. 80% overall test coverage components

  10. Migrate Enzyme tests into React Testing Library <DEXP> | Impact - 3 3 - 3 | Cost -

    1. Audit work
    2. Create tickets and find support
    3. Migrate tests

  11. Migrate into using Next.js - <DEXP> | Impact - 3 2 3 4 - 3 | Cost -

    1. Pick between the many approaches to incrementally adopt Next.js
    2. Move out the application part by part into Next.js keeping Flask as the API server

  12. Improve our browsing experience - <UT> | Impact - 3 2 3 2 3 - 3 | Cost -

    1. Complete our transition to faceted search on the search page
    2. Add a basic browse experience to complement the faceted search

  13. Bulky components & layout - <DP> | Impact - 4 2 3 3 3 - 3 | Cost -

    1. Update our components to be more compact
    2. Update our layout to be more compact
    3. Update table detail page left panel to look more orderly when a table has many of the ‘optional’ fields displayed.

  14. Homepage page is not tailored to user role - <UT> | Impact - 4 3 4 2 3 - 3 | Cost -

    1. Move our announcements section into a drawer that opens from the global header
    2. Improve our homepage layout to contain more customizable widgets

  15. Improve State Management - <DEXP> | Impact - 1 3 3 3 - 3 | Cost -

    1. Create Redux selectors whenever we can
    2. Explore using Redux toolkit to simplify our Redux logic

  16. Improve our API - <DEXP> | Impact - 4 3 3 3 - 3 | Cost -

    1. Runtime type checking with https://gcanti.github.io/io-ts/
    2. Clean up all deprecated endpoints
    3. Refactor resource endpoints to make them consistent

  17. Add codebase health monitoring - <DEXP> | Impact - 2 3 4 3 - 3 | Cost -

    1. Find a free solution for OSS (code climate?)
    2. Introduce it
    3. Break builds with degradations

  18. Update component library - <DEXP> | Impact - 2 3 3 3 4 - 3 | Cost -

  19. Improved instrumentation - <DEXP> | Impact - 1 3 2 3 - 3 | Cost -

    1. Actions with analytics payloads
    2. Make sure all feature utilization is accurately tracked

  20. Test coverage tracking - <DEXP> | Impact - 2 3 3 2 - 2.5 | Cost -

    1. Use codecov to track test coverage
    2. Use codecov to break builds when reducing by a given %

  21. Improve our typography - <DEXP> | Impact - 1 2 2 1 2 - **2 **| Cost -

    1. Remove old classes

  22. Improve our buttons - <DEXP> | Impact - 2 2 2 1 2 - **2 **| Cost -

    1. Audit our buttons in Storybook
    2. Audit our links in Storybook
    3. Update all our buttons and links to be consistent
    4. Add missing styles

  23. Improve our a11y - <UT> | Impact - 2 3 1 1 1 - **2 **| Cost -

    1. Audit a11y
    2. Fix a11y issues
    3. Setup automated testing to catch regressions

  24. Make the lists/tables consistent - <DEXP>, <DP> | Impact - 2 1 3 2 2 - 2 | Cost -

    1. Decide for one way of listing elements
    2. Roll it out all over the UI

  25. Fix all the ESLint warnings - <DEXP> | Impact - 2 2 2 2 -** 2** | Cost -

    1. Destructuring
    2. SetState issues

  26. Centralize URL logic - <DEXP> | Impact - 2 2 2 2 - 2 | Cost -

    1. Create util file with static URLs
    2. Extend with dynamic URLs

  27. Update React version - <DEXP> | Impact - 1 2 2 3 - 2 | Cost -

    1. Depends on moving to React Testing Library
    2. Update codebase using codemods

  28. Fix our fixture data - <DEXP> | Impact - 2 2 2 3 - 2 | Cost -

    1. Create test data builders for all our fixtures
    2. Create a debug mode to use the test data for manual testing of features

  29. Easy to use text component - <DEXP> | Impact - 1 1 3 1 3 - 2 | Cost -

8. Appendix B: Why we got here

There is no single reason for the current state of our frontend codebase, however, we can relate to the following reasons (among others): * Lack of resources * Lack of OSS contributions to the UI * General Stagnation of the community