Azure Databricks Engineer

Job Description

As an Azure Databricks Engineer, you will play a crucial role in our data engineering and analytics efforts on the Azure platform. You will be responsible for designing, developing, and maintaining Azure Databricks-based solutions that enable efficient data processing, analytics, and machine learning for our organization.

Roles and Responsibility

• Azure Databricks Development: Design, develop, and maintain data pipelines, ETL
  processes, and analytics workflows using Azure Databricks.
• Data Ingestion: Implement data ingestion processes from various sources into Azure
  Databricks, ensuring data quality and reliability.
• Data Transformation: Perform data transformation and cleansing tasks to prepare data
  for analysis and reporting.
• Optimization: Continuously optimize and fine-tune Azure Databricks clusters and jobs
  for performance and scalability.
• Data Integration: Collaborate with cross-functional teams to integrate Azure Databricks
  with other Azure services and data storage solutions.
• Security and Compliance: Implement and maintain data security measures and ensure
  compliance with data governance standards on the Azure platform.
• Documentation: Create comprehensive documentation for Azure Databricks solutions,
  including best practices and guidelines.

Qualifications

• Bachelor’s degree in Computer Science, Data Science, or a related field (or equivalent
  work experience).
• 4-6 years of hands-on experience as an Azure Databricks Engineer or in a similar role.
• Proficiency in Azure Databricks, including data processing, cluster management, and
  notebook development.
• Strong programming skills in languages such as Python, Scala, or SQL.
• Experience with big data technologies and frameworks like Apache Spark.
• Knowledge of Azure services, particularly Azure Data Lake Storage, Azure SQL Data
  Warehouse, and Azure Synapse Analytics.
• Familiarity with cloud security and data governance practices on Azure.
• Excellent problem-solving and analytical skills.
• Ability to work collaboratively in a team and communicate effectively with technical and
  non-technical stakeholders.