The more your development environment looks like what you're ultimately deploying, the fewer headaches your team is going to have.

When using Kubernetes, you have a few different options of how you could have your developers work. I've built developer tools across the whole spectrum and here are some benefits and drawbacks I've seen to each.

Build Local, Run Local [without Docker or Kubernetes]

Benefits:

• No upfront migration: Continue to develop your applications exactly how you did before
• No learning curve: Developers don't have to learn new tooling
• Usually quick: Once development environment is set up and dependencies are downloaded, builds are usually quick. Can leverage native tools: compiler level caching, running on local ports, and instant changes (no networked filesystem, ssh tunnel, etc.)

Downfalls

• Parity between environments: Significant departure from how services are actually ran. Many places that things can go wrong.
• "Works on my machine": Setting up a developer environment needs to be done on a per-user basis.
• Single platform development: Development OS cannot be different from runtime environment (e.g. can't develop on a pixelbook or MacBook and deploy to a linux environment.)

Build Local, Run Local [with Docker and Kubernetes]

Benefits

• Closer to Production: Fewer differences with higher environments. Developers can catch issues in development rather than waiting for CI or QA or catch them.
• Portable: You can run Docker on Kubernetes on every major OS.
• Declarative environment: Setup and teardown development environments easily. No need for long developer environment setup documents. Applying the configuration for a cluster can be as easy as kubectl apply -f folder/.
• Reproducible: Alongside declarative environments, bugs and other issues are easier to reproduce because Docker and Kubernetes manage the immediately dependencies for an application.
• Full Control: Developers manage the entire stack and therefore have few limitations when developing.

Drawbacks

• Limited: Environment may be too large to run on your workstation. Istio suggests 8GB and 4 vCPUs on minikube. Won't work for users with high data or compute requirements (e.g. ML workloads)
• Ops work for the Developer: Developers have to manage a local cluster. Minikube and Docker for Desktop provide one-click cluster setup, but what happens when your cluster goes down? Networking issues, OOM errors, and more can require developer intervention.

Build Local, Deploy Remote [with Kubernetes]

Benefits

• Closest to Production: While it doesn't really matter what guest OS Docker uses, Kubernetes still has many host dependencies with the kubelet, which doesn't run containerized. A Kubernetes feature might work on Docker for Desktop or minikube's custom VM image but not the one your production cluster.
• More Portable: You can run Docker on every major OS.
• Managed Declarative environment: Have your ops team manage the cluster, instead of the developers. Manage O(orgs) clusters, not O(developers).
• Can support arbitrarily large environments
• Can be shared by multiple users
• Can utilize ops-managed resources (dashboard, logging, monitoring, specialized hardware like TPUs)

Drawbacks

• Cost: You have to buy hardware for your developers anyways
• Speed: Build artifacts can be large, and it takes time to move large objects across a network.
• New Development Tools: Apps aren't deploy to localhost by default like they might be locally.