大手グローバル製薬-Platform Engineer-Data Solutions　

Responsibilities

• Create best practices and thought leadership content to be used by the federated delivery teams building data solutions and data products on Enterprise Data platforms that cater to batch, streaming and real-time data.
• Influence stakeholders at all levels through complex engagement models with the wider cloud ecosystem not limited but inclusive of AWS foundations for Infrastructure and data technologies, Databricks, Informatica, Kafka, Managed File Transfer, and 3rd party applications, ensuring they are excited by the Enterprise Data Services vision and solution strategy.

• Be a 'champion’ for both customers and colleagues by operating as an expert Engineer and trusted advisor for significant data analytics architecture, design, and adoption and scaling of the Datalake platform.

• Provide a roadmap for modernizing legacy capabilities inherent to the current platform. Support all data platform initiatives – Data Lake Strategy, Data Engineering and Platform development, Data Governance, Security Models, and Master Data Management.

• Establish a collaborative engineering culture based on trust, innovation, and a mindset of continuous improvement. Utilize Industry best practices and agile methodologies to deliver solution and extract efficiency through automations in Continuous Development and Continuous Integration. Manages efforts to problem solve for engineering challenges and coordinate with project consultants and delivery/engagement managers.
• As a leading technical contributor who can consistently take a poorly defined business or technical problem, work it to a well-defined data problem/specification, and execute it at a high level. Have a strong focus on metrics, both for the impact of their work and for its engineering and operations.
• Understand the Data Platforms investments and create data tools for consumption of services and uncover opportunities for cost optimization to assist the team in building and optimizing our platforms into an innovative unit within the company.

Must Requirements:

-Data Engineering Experience

• 8+ years of relevant work experience in data platforms, solutions, and delivery methodologies (Java, Python, Spark, Hadoop, Kafka, SQL, NoSQL, Postgres and/or other modern programming language and tools such as JIRA, Git, Jenkins, Bitbucket, Confluence).
• Familiarity with the core technology stack, including Databricks Lakehouse (Delta Lake) or equivalent such as Big Query/Snowflake, SQL/Python/Spark, AWS, Prefect/Airflow,

• Deep Specialty Expertise in at least one of the following areas:
  • Experience scaling big data workloads that are performant and cost-effective.
  • Experience with Development Tools for CI/CD, Unit and Integration testing, Automation and Orchestration, REST API, BI tools, and SQL Interfaces (e.g., Jenkins)
  • Experience designing data solutions on cloud infrastructure and services, such as AWS, Azure, or GCP(Google Cloud) using best practices in cloud security and networking.
  •  

-Software Engineering Experience

• 5+ years’ experience in a customer-facing technical role with expertise in at least one of the following:
  • Software Engineer/Data Engineer: data ingestion, streaming technologies - such as Spark Streaming and Kafka, performance tuning, troubleshooting, and debugging Spark or other big data solutions.

Nice to have:

• Experience with ETL/Orchestration tools (e.g., Informatica, and Airflow, etc.)
• Industry level experience of working with public cloud environments (AWS, GCP, or Azure), and associated deep understanding of failover, high-availability, and high scalability.

• Data ingestion using one or more modern ETL compute and orchestration frameworks (e.g., Apache Airflow, Luigi, Spark, Apache Nifi, Flink and Apache Beam).

• 3+ years of experience with SQL or NoSQL databases: PostgreSQL, SQL Server, Oracle, MySQL, Redis, MongoDB, Elasticsearch, Hive, HBase, Teradata, Cassandra, Amazon Redshift, Snowflake.

• Advanced working SQL knowledge and experience working with relational databases, authoring (SQL) as well as working familiarity with a variety of databases.

• Experience building and optimizing 'big data' data pipelines, architectures, and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.

• Working knowledge of message queuing, pub/sub stream processing, and highly scalable 'big data' data stores.

• Outstanding communication and relationship skills, ability to engage with a broad range of partners, capable of leading by influence.