Location: LPS or HKT Office
Working Hours: Mon – Fri, 0900 - 1800
Annual Leave: 15 days
Summary
- Data Asset team is looking for a Data Engineer with a solid technical background to build and support cloud based data asset platform.
- The platform enables internal engineers and analysts to process SQL or transformations using Apache Spark, Python on compute for batch, streaming and ML processing, executing over schemaed data stored in data lake and data warehouse.
- The team manages data ingestions from various data sources, build and support ETL pipelines, and continuously look for solutions to enhance the robustness and scalability of the data architecture.
Job Duties
You will be responsible for application development and ongoing maintenance support of the Data Analytics area with the following scope of duties:
- Manage cloud resources including EC2, EMR, S3, Redshift, Glue, etc.
- Design data schema and operate internal data warehouses and SQL/NOSQL database systems
- Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using SQL and cloud big data technologies
- Build robust and scalable data integration (ETL) pipelines using SQL, PySpark and cloud data services like EMR, Redshift, etc.
- Implement suitable data model for data warehouse in order to achieve efficient data retrieval
- Gather business and functional requirements and translate these requirements into robust, scalable, operable solutions with a flexible and adaptable data architecture.
- Collaborate with other Engineers and Product Managers to recognize and help adopt best practices in data engineering and analytics: data integrity, test design, analysis, validation, and documentation
Basic Qualifications
- 5+ years working experience in software development, data engineering, business intelligence or related fields
- 2+ years data modeling, ETL development, and data warehousing experience
- 2+ years of experience using big data technologies (Hadoop, Spark, Hive, Impala, etc.)
- Good understanding of cloud infrastructure
- Advanced SQL and query performance tuning skills
- Experience with one of the functional scripting languages (Python, Scala, Java etc.) to process semi-structured or unstructured data inputs.
- Knowledge of professional software engineering practices & best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations.
- Strong customer focus, ownership, urgency, and drive.
- University graduate in Computer Science, Information Technology, Data Science or related disciplines
- Effective verbal and written communication in English and Chinese
- Good team player with strong interpersonal and communication skills
- Forward thinking, embracing changes, fast-learning, energetic, able to perform in challenging environment
