Running on Databricks?

Databricks jobs slow or Unity Catalog locking you out? We sort it out.

Databricks is powerful enough to run a lakehouse, and expensive enough that waste shows up quickly. Clusters that take ten minutes to start, Unity Catalog grants that work in dev and not in prod, Delta merges that rewrite the whole table every night: we trace the Spark UI, the audit log and the lakehouse layout until jobs finish inside their SLA and the monthly bill makes sense.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

databricks-job.log

still slow

What it is

Databricks at a glance

Databricks is a cloud data and AI platform built around the lakehouse idea: open table formats like Delta Lake on object storage, Spark as the primary compute engine, and a unified governance layer in Unity Catalog. It was founded in 2013 by the original creators of Spark and now runs on AWS, Azure and Google Cloud.

By 2026 Databricks is where many large organizations park their data platform, ML pipelines and AI workloads. The core pieces are notebooks, jobs, Delta Live Tables, SQL warehouses, Unity Catalog, MLflow and the Mosaic AI stack. Its appeal is end-to-end coverage from ingestion to model serving; the tradeoff is a platform that rewards discipline and punishes casual use at the invoice.

Type
Lakehouse data and AI platform
First released
2013, founded by the original Spark team
License
Proprietary platform, Delta Lake and Spark are Apache 2.0
Maintained by
Databricks, Inc.
Typically used for
Lakehouse analytics, Spark ETL, Delta Lake tables, ML training and serving, data warehousing via SQL warehouses

The honest version

Databricks pros and cons

No technology is all upside. This is where Databricks earns its place and where it tends to cause trouble.

Where Databricks is strong

  • A real lakehouseDelta Lake on object storage with Spark compute gives you warehouse-style ACID tables without paying warehouse-style storage costs. Streaming, batch and ML all read the same tables.
  • Unity Catalog centralizes governanceGrants, lineage, data classification and audit sit in one place across workspaces, which is the governance story the raw lakehouse never had on its own.
  • Photon makes SQL competitiveThe vectorized Photon engine brings SQL warehouse performance to Databricks queries, which closes much of the gap to Snowflake and BigQuery for interactive analytics.
  • Delta Live Tables for declarative pipelinesDLT lets you describe pipelines as data transformations with built-in expectations and lineage, which replaces a lot of custom Spark orchestration glue.
  • MLflow is integrated end to endExperiment tracking, model registry, serving endpoints and feature stores work together, which makes putting an ML model into production less of a research project.
  • Works across cloudsThe same platform runs on AWS, Azure and GCP with the same control plane model, which matters for organizations that are not single-cloud by choice.

Where Databricks bites

  • Cluster startup latency is painfulA cold cluster takes minutes to come up, which breaks the flow of ad hoc work and pushes teams toward keeping clusters running, which then shows up on the bill.
  • Costs can run away fastDBUs plus cloud infrastructure, plus Photon surcharge, plus idle time, multiply quickly. Without a cost dashboard and a cluster policy you can land a four-figure surprise in a week.
  • Unity Catalog migration is a projectMoving from the legacy hive metastore to Unity Catalog touches every workspace, every job, and every permission. The payoff is real; the migration is not quick.
  • Notebook culture hides bad engineeringNotebooks are convenient and encourage code that is not version controlled, not tested, and not reusable. A platform full of production notebooks is hard to maintain.
  • Spark errors are their own dialectAnalysisException, Py4JJavaError, stage failures and executor OOMs all require reading a long Java stack trace through a Python lens. The learning curve is real.
  • Vendor gravity is strongDelta Lake is open, but much of the platform value is in the UI, orchestration and governance. Moving off is a multi-quarter project rather than a database export.

Sound familiar?

Common Databricks problems we fix

The Databricks problems that bring people to us, and what you have once each one is fixed.

DBU usage this month: 48,217 (budget: 30,000)

The bill jumps 60% in one month

An always-on cluster left running over a weekend, a job that started running hourly instead of daily, or Photon enabled across workspaces where it is not earning its surcharge.

Spend tracked and attributed

PERMISSION_DENIED: User does not have USE SCHEMA

Unity Catalog grants fail in production

Grants that were applied through the UI in dev and never committed to code, so a new workspace or a new service principal has no idea what it is allowed to touch.

Grants managed in code

Cluster startup timed out after 20 minutes

Interactive work is blocked waiting for clusters

A standard cluster taking many minutes to come up, forcing engineers into context switching and encouraging them to leave clusters running. Serverless or pool-backed clusters shorten the loop.

Cluster startup under a minute

DELTA_CONCURRENT_APPEND: ConcurrentAppendException

Concurrent Delta writes fight each other

Two jobs merging or appending to the same table at the same time, conflicting on the same partitions. Partitioning strategy, write isolation and job ordering have to line up.

Writes serialized on the right key

Delta table has 1,284 small files

Lakehouse queries get slower every week

Streaming or incremental jobs producing many small files, which Spark then has to enumerate and read individually. OPTIMIZE and auto-compaction bring query times back.

Small files compacted on schedule

Photon acceleration disabled

Queries silently run on the non-Photon engine

A UDF, an unsupported function or a specific data type that falls off Photon, so the query runs on the slower engine while still paying the surcharge. Shows up as nothing-looking performance regressions.

Photon path kept intact

AnalysisException: table not registered in Unity Catalog

Code expects legacy hive paths

A notebook that references dbfs paths or hive metastore tables that no longer exist after a Unity Catalog migration. Migration has to carry the code and the data together.

Code and catalog aligned

Cluster in TERMINATED state

A job cluster fails to come back up

A cluster policy change, a missing init script, or a Spark version removed from the available list, so the next scheduled job cannot even start.

Job clusters launch reliably

What we do

Databricks help, from cost control to lakehouse design

Call us for one slow job, or hand us the whole workspace.

Rescue
Jobs falling over, clusters that will not start, or Unity Catalog permissions blocking a whole team. We stabilize first, then write down how it happened.
Cost and cluster tuning
Cluster policies, autoscaling settings, serverless versus classic, and Photon use measured against real workloads, with the bill before and after.
Unity Catalog migration
Moving from hive metastore to Unity Catalog across workspaces, with grants in code, lineage verified and jobs tested before the switch.
Lakehouse design
Bronze, silver and gold layering, Delta table layout, OPTIMIZE and VACUUM cadence, and Delta Live Tables where they earn their keep.
ML and MLflow
Feature stores, model registry, serving endpoints and the pipeline that moves a model from notebook to production without a bespoke rewrite.
Managed Tech
A standing watch on job health, DBU spend, Unity Catalog drift, Delta table maintenance and platform upgrades, plus support when a workspace misbehaves.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Databricks alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Databricks questions

Something else? Call+1 737 727 7327

Why is our Databricks bill so high?

Usually a combination of always-on interactive clusters, jobs running on bigger clusters than they need, Photon enabled on workloads it does not accelerate, and idle time on classic clusters that should be serverless. We attribute spend per cluster and per job, then recommend which knobs to turn first.

Databricks or Snowflake for a new data platform?

Snowflake wins on pure SQL warehousing simplicity and often on cost at modest scale. Databricks wins when the workload includes ML, streaming, or data types that do not fit a warehouse neatly. Many organizations end up with both: Snowflake for the warehouse, Databricks for ML and the data lake.

How do we migrate to Unity Catalog without breaking everything?

A phased plan: new tables go to Unity Catalog first, legacy tables are mirrored and dual-written, grants move into code, and jobs are tested against the new catalog before the switch. Rushing it almost always means a week of broken permissions and a month of audit cleanup.

Should we use Delta Live Tables or regular jobs?

DLT shines when the pipeline is declarative, has clear expectations, and benefits from automatic incremental logic. Regular jobs still win for pipelines that need custom orchestration, bespoke scheduling, or integration with existing airflow dags. We mix both in most real projects.

Why do our Delta merges take so long?

Usually a combination of small files, bad partitioning for the merge predicate, and auto-compaction turned off. A table that is partitioned by date and merged on customer_id will rewrite huge amounts of data on every run. OPTIMIZE with ZORDER and a smarter write key usually recovers the time.

How much does Databricks help cost?

The first conversation is free. Tell us the workload and the bill, and we come back with a price before any work is scheduled.

Free estimate

Databricks giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.