Databricks, the lakehouse firm, launched a new development experience for the Databricks Lakehouse Platform: the VS Code Extension for Databricks. It enables developers to write their code locally leveraging the powerful authoring capabilities of the IDE, connect to Databricks clusters and run code remotely, and use the software development best practices of source code control, unit testing and CI/CD directly from their favourite IDE.
What are the new capabilities of VS Code?
Develop natively inside VS Code
Following this release, organisations can now build all of their data and AI applications while staying inside their IDE. Developers can author the code for their pipelines and jobs in VS Code, then deploy, test and run it in real-time on their Databricks cluster. This will enable teams to apply software development best practices and utilise VS Code’s native capabilities for editing, refactoring, testing, and CI/CD for data and AI projects.
“My team wants to run data workloads on the Databricks Lakehouse but develop data apps from their IDE. Now they can write code, edit, and test in their IDE and run their code on Databricks.” – Andrew Garrido, Head of Software Development for Data Processing, Kantar.
The full power of the Databricks Lakehouse in your IDE
Now that organisations can build on Databricks within VS Code, developers can perform all of their work in one location. Databricks objects can be managed inside VS Code natively with the new extension, allowing teams to stay in their IDEs and preventing context switching between applications. All of the Databricks components, such as clusters, pipelines, and tasks, are integrated into VS Code workspaces and regular workflows.
The Lakehouse’s scale can be utilised to process and analyse large data sets, use clusters for queries and visualisations, train machine learning models, and deploy jobs to production so that anyone in a firm can see and use data to make decisions, all within VS Code.
Uniquely designed to take advantage of full IDE capabilities
Developer teams can enjoy all the comforts of VS Code they are used to while building applications on Databricks. Navigating to function definitions, refactoring, using advanced find-and-replace, and utilising split windows can help drive team-wide efficiency.
Teams can also receive code completion on functions and variables, including Databricks-specific objects, to speed up discovery and development. Developers can now utilise the software development best practices already set up for the development process, and build and modularise different parts of their program across separate functions and files.
Files are local, so teams can use VS Code’s Git tools and the git CLI, while they can also modularise code into files and libraries to encourage code reuse and improve hygiene. VS Code enables developers to use their preferred testing framework to ensure software quality, while integrating with top CI/CD tools to deliver code into production faster.
“I find that the Databricks extension enables developers who prefer an IDE to be able to readily harness the power of VS Code with native and extensible functions while easily allowing me to run Databricks locally. Very helpful!” – Sam Walker, Data Engineer at Watco.
A fully supported IDE experience
The extension is available through the Visual Studio marketplace to streamline the acquisition experience and ensure that teams are getting a supported and trusted version from Databricks. It will be regularly updated to offer experience and quality improvements and CIOs can also be assured that their investment is future-proof with added support for new capabilities of the Databricks Lakehouse platform as they get released.
“We consider it a huge advance that will further boost our development process, a milestone that deserves to be celebrated and shared!” – Iago Brandão, MLOps at ViaHub.
What does the release mean for developers?
“This will be the first of many planned updates for teams who rely on IDEs for their development process,” said Patrick Wendell, co-founder and VP Engineering at Databricks.
“We’ve invested in a new team focusing exclusively on the breadth of the developer ecosystem, and will be rolling out support for other IDEs and additional tools that enable full access to the lakehouse from third-party products,” Patrick Wendell further said.
The VS Code Extension is available for download through the Visual Studio marketplace: Download the VS Code Extension. Please take a look at our documentation for AWS, Azure, and GCP for help with how to get started using the extension, and give it a try.