Job Description

Job Description

Norfolk Southern Corporation is currently seeking a Senior Big Data Engineer with an affinity for working with others to create successful solutions.  Join a smart, highly skilled team with a passion for technology, where you will work on our state-of-the-art Big Data Platforms. This Senior Data Engineer with a balance of theoretical business intelligence and data analytics knowledge and hands-on experience and exposure to real-world BI problem-solving.  In this roll you will participate in all phases of the Data Engineering life cycle and will independently and collaboratively write project requirements, architect solutions, and perform data ingestion development and support duties.  

In this role, you will work with people from many areas of the company to understand their data and BI needs and perform detailed requirements gathering along with analysis in partnership with our team of Data Modelers.  Engage our BI development team to agree on effort estimates and then oversee the progress of the project through its completion.  

Responsibilities

Defines data requirements, gather, and wrangle large scale of structured and unstructured data, and validate data by running various data tools in the Data Environment.
Supports the standardization, customization and ad-hoc data analysis, and will develop the mechanisms to ingest, analyze, validate, normalize and clean data.
Creates data policy and develop interfaces and retention models which requires synthesizing or anonymizing data.
Implements statistical data quality procedures on new data sources, and by applying rigorous iterative data analytics, supports Data Scientists and analytics and insights creation in data sourcing and preparation to visualize data and synthesize insights of commercial value.
Develops and maintains data engineering best practices and contributes to Insights on data analytics and visualization concepts, methods and techniques.
Works closely with the data science and business intelligence teams to develop data models and pipelines for research, reporting, and machine learning.
Builds data pipelines that clean, transform, and aggregate data from disparate sources.
Employs a variety of languages and tools (e.g. scripting languages) to marry systems together.
Engages with business teams to gather requirements and design data solutions.
Understands overall architecture and industry data technology.
Mentors team of more Junior Data Engineers.
Refines and prioritizes team backlog. 
Collaborates across multiple projects to provide data engineering expertise across teams. 
Presents to leadership around solutions that are in-progress or being maintained. 
Analyzes most relevant insights and shares with leadership to provide strategic recommendations for the business.
Leads a team of data engineers and act as a key senior contributor to a data engineering project.
Drives innovative technology solutions through thought leadership on emerging trends.
Applies knowledge of Data Architecture components, leads project teams from requirements to implementation. 

Education Required

Bachelor’s Degree in Information Systems, Computer Science, Computer Information Systems or related technology field required.  

Skills Required 

5 years of Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog.
7+ years of experience in a data engineering role with a track record of manipulating, processing, and extracting value from large datasets.
7+ years of Big Data tools like Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, HDFS.
5+ years building, testing, and optimizing ‘Big Data’ data ingestion pipelines, architectures, and data sets with Tibco, IBM or others. 
Advanced SQL, data ingestion frameworks, data modeling, and working with big data.
High-velocity high-volume stream processing with Apache Kafka and Spark Streaming.
NoSQL databases, including HBASE and/or Cassandra.
ETL experience with Scala (and/or Python) and PySpark/Scala-Spark
Agile Scrum, Kanban or SAFe experience.

Skills Desired 

Python (and/or Scala) and PySpark/Scala-Spark.
Database solutions like Snowflake, Kudu/Impala, Delta Lake or BigQuery.
NoSQL databases, including HBASE and/or Cassandra.
Azure, AWS Serverless technologies, like, S3, Kinesis/MSK, lambda, and Glue.
Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series.
Strong understanding of Relational & Dimensional modeling. 
Experience with GIT code versioning software.
Experience with REST API and Web Services.