Lead Data Engineer - Spark/Flink/Scala Engineer

Company: Samsung Research Global
Location: Irvine
Posted on: February 22, 2021

Job Description:

Irvine , CA Location: Irvine, CA-- The Visual Display Intelligence Lab at Samsung Research America is building a next-generation data platform to support Smart TV products and services. With offices in Mountain View and Irvine, CA, we are close to a number of our collaboration partners. Our research and development include TV analytics, ads targeting, and personalized services. We ingest and process billions of records daily from millions of TVs in the field, and we are looking for an experienced professional to join our team on the development of an integrated data platform. --At Visual Display Intelligence lab, Samsung Research America, we operate as a start-up while leveraging the broader assets of a larger global enterprise. We do advance research and development to immensely benefit our core business verticals in the space of Ads and Marketing. Our team primarily focuses on processing huge data sets, build robust and scalable big data pipelines, conduct advanced research in implementing state-of-the-art ML and AI techniques, build data services that are both real-time and batch based. All those combined provide the best-in-class experience to our customers across consumer electronics and internet of things. Many of our engineers take pride in publications, filing A-grade patents and build robust products and services. General Description We are looking for lead Scala Engineers with experience with batch and/or streaming jobs. We utilize Spark for batch jobs and Flink for real-time streaming jobs. Experience with Hadoop, Hive, AWS S3 is also an asset. --Responsibilities

• Create new, and maintain existing Spark/Flink jobs written is Scala
• Produce unit and system tests for all code
• Lead architecture / design discussions to improve our existing frameworks
• Define scalable calculation logic for interactive and batch use cases
• Interact with infrastructure and data teams to produce complex analysis across data
• Interact with product managers to understand requirements, come up with estimates and take it to delivery.
• The ability to lead senior engineers. This role has the growth potential of managing a few senior engineers and engineers in distributed locations. --Required Qualifications:
  • Bachelor's degree in Computer Science / MS or equivalent combination of education, training, and experience.
  • 8+ years of experience with Scala and/or Java.
  • Required experience with Hadoop, Spark
  • 2 years of experience in people management is preferred
  • Knowledge and experience with cloud-based technologies
  • Experience in batch or real-time data streaming
  • Ability to dynamically adapt to conventional big-data frameworks and open source tools if project demands
  • Knowledge of design strategies for developing scalable, resilient, always-on data lake
  • Must be very comfortable with reading and writing Scala code
  • An aptitude for analytical problem solving
  • Deep knowledge of troubleshooting and tuning Spark applications and Hive scripts to achieve optimal performance
  • Good understanding/knowledge of HDFS architecture and various components such as Job Tracker, Task Tracker, Name Node, Data Node, HDFS high availability (HA) and Map Reduce programming paradigm.
  • Experienced working with various Hadoop Distributions (Cloudera, Hortonworks, MapR, Amazon EMR) to fully implement and leverage new Hadoop features
  • Experience in developing Spark Applications using Spark RDD, Spark-SQL, Spark -Yarn, Spark Mlib and Data frame APIs
  • Experience with real-time data processing and streaming techniques using Spark streaming and Kafka, moving data in and out HDFS and RDBMS.
  • Experience with ML/distributed ML frameworks like Spark-MLlib, Tensorflow etc.
  • Model training with batch and real-time prediction scenarios : Use machine language and statistical modeling techniques such as decision trees, logistic regression, Bayesian analysis and others to develop and evaluate algorithms to improve product/system performance, quality, data management and accuracy.
  • Familiarity with open source configuration management and development tools
  • Ability to work with cross functional teams from requirements to delivery and drive projects. --Preferred Qualifications: --Hands on experience and production use of Hadoop/Cassandra, Spark, Flink and other distributed technologies would be a plus
    • Other Technologies
    • Scalatest
    • Gradle/Maven
    • Airflow
    • SQL
    • Redshift Samsung is an EEO/Veterans/Disabled/LGBT employer. We welcome and encourage diversity as we strive to create an inclusive workplace. Additional Information for California Residents: --*
      Information about what we are doing to meet the obligations of the California Consumer Privacy Act (CCPA) and other important information for California residents is available at SRA's Job Applicant Privacy Policy . Please review the policy carefully, and if you agree to the collection, processing, use, and transfer of your personal information as described in the policy, please indicate your acceptance by selecting "Accept" from the following drop down menu. Please select U.S. Equal Opportunity Employment Information (Completion is voluntary) Individuals seeking employment at Samsung Research America are considered without regards to race, color, religion, national origin, age, sex, marital status, ancestry, physical or mental disability, veteran status, gender identity, or sexual orientation. You are being given the opportunity to provide the following information in order to help us comply with federal and state Equal Employment Opportunity/Affirmative Action record keeping, reporting, and other legal requirements. Completion of the form is entirely voluntary. Whatever your decision, it will not be considered in the hiring process or thereafter. Any information that you do provide will be recorded and maintained in a confidential file. Gender Please select Are you Hispanic/Latino? Please select Race & Ethnicity Definitions If you believe you belong to any of the categories of protected veterans listed below, please indicate by making the appropriate selection. As a government contractor subject to Vietnam Era Veterans Readjustment Assistance Act (VEVRAA), we request this information in order to measure the effectiveness of the outreach and positive recruitment efforts we undertake pursuant to VEVRAA. Classification of protected categories is as follows: A "disabled veteran" is one of the following: a veteran of the U.S. military, ground, naval or air service who is entitled to compensation (or who but for the receipt of military retired pay would be entitled to compensation) under laws administered by the Secretary of Veterans Affairs; or a person who was discharged or released from active duty because of a service-connected disability. A "recently separated veteran" means any veteran during the three-year period beginning on the date of such veteran's discharge or release from active duty in the U.S. military, ground, naval, or air service. An "active duty wartime or campaign badge veteran" means a veteran who served on active duty in the U.S. military, ground, naval or air service during a war, or in a campaign or expedition for which a campaign badge has been authorized under the laws administered by the Department of Defense. An "Armed forces service medal veteran" means a veteran who, while serving on active duty in the U.S. military, ground, naval or air service, participated in a United States military operation for which an Armed Forces service medal was awarded pursuant to Executive Order 12985. Veteran Status Please select Form CC-305 OMB Control Number 1250-0005 Expires 05/31/2023 Voluntary Self-Identification of DisabilityWhy are you being asked to complete this form? We are a federal contractor or subcontractor required by law to provide equal employment opportunity to qualified people with disabilities. We are also required to measure our progress toward having at least 7% of our workforce be individuals with disabilities. To do this, we must ask applicants and employees if they have a disability or have ever had a disability. Because a person may become disabled at any time, we ask all of our employees to update their information at least every five years. Identifying yourself as an individual with a disability is voluntary, and we hope that you will choose to do so. Your answer will be maintained confidentially and not be seen by selecting officials or anyone else involved in making personnel decisions. Completing the form will not negatively impact you in any way, regardless of whether you have self-identified in the past. For more information about this form or the equal employment obligations of federal contractors under Section 503 of the Rehabilitation Act, visit the U.S. Department of Labor's Office of Federal Contract Compliance Programs (OFCCP) website at www.dol.gov/ofccp . How do you know if you have a disability? You are considered to have a disability if you have a physical or mental impairment or medical condition that substantially limits a major life activity, or if you have a history or record of such an impairment or medical condition. Disabilities include, but are not limited to:
      • Autism
      • Autoimmune disorder, for example, lupus, fibromyalgia, rheumatoid arthritis, or HIV/AIDS
      • Blind or low vision
      • Cancer
      • Cardiovascular or heart disease
      • Celiac disease
      • Cerebral palsy
      • Deaf or hard of hearing
      • Depression or anxiety
      • Diabetes
      • Epilepsy
      • Gastrointestinal disorders, for example, Crohn's Disease, or irritable bowel syndrome
      • Intellectual disability
      • Missing limbs or partially missing limbs
      • Nervous system condition for example, migraine headaches, Parkinson's disease, or Multiple sclerosis (MS)
      • Psychiatric condition, for example, bipolar disorder, schizophrenia, PTSD, or major depression Disability Status Please select 1 Section 503 of the Rehabilitation Act of 1973, as amended. For more information about this form or the equal employment obligations of Federal contractors, visit the U.S. Department of Labor's Office of Federal Contract Compliance Programs (OFCCP) website at www.dol.gov/ofccp . PUBLIC BURDEN STATEMENT: According to the Paperwork Reduction Act of 1995 no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. This survey should take about 5 minutes to complete.

Keywords: Samsung Research Global, Irvine , Lead Data Engineer - Spark/Flink/Scala Engineer, Engineering , Irvine, California