Job Description

COMPANY DESCRIPTION

General Dynamics Information Technology is an IT firm engaged in supporting defense, government, and intelligence projects. The firm offers network system integration, information technology, software, system development, business process outsourcing, security, and spectrum management services. General Dynamics Information Technology provides technology solutions, strategy, and mission services, including technology delivery models, AI, Cloud, Cyber, and data analytic. The company serves intelligence, health, defense, civil, homeland security, state, and local government agencies. John Philip Holland founded General Dynamics Information Technology in 1899, and its headquarters is located in Reston, Virginia, United States. The company helps its customers to advance mission performance, discover opportunities, and transform operations to build a better future. General Dynamics Information Technology provides competitive salary packages, a 401k plan, paid vacations, professional development training to its employees.

JOB DESCRIPTION

GDIT has an opportunity available for a talented and innovative High Performance Computing (HPC) Linux Systems Administrator within our High Performance Computing Center of Excellence to provide continuing on-site support for our NOAA Research and Development High Performance Computing Systems (RDHPCS) customer at the NOAA Global Systems Laboratory in Boulder, Colorado.

The qualified candidate will bring their hands-on technical and system administration expertise on-site to maintain the operational readiness and availability of NOAA's RDHPCS high performance computing systems at the Boulder, Colorado facility; manage and support new technology insertions, as well as provide remote technical support and collaboration with our other supported NOAA sites at Fairmont, West Virginia and Princeton, New Jersey.

Responsibilities and Duties:

* Daily monitoring and management of medium to large HPC cluster environments; working and contributing as a member of a small team for local systems activities and a large team for cross-site support.
* Maintain an overall situational awareness of HPC environment to identify and correct issues before they impact operations.
* Independent problem solving and troubleshooting to quickly advance towards viable resolutions.
* Plan, prepare, and execute required timely security patching and software upgrades during scheduled maintenance periods; includes building of open source software packages and installation of commercial software and license managers.
* Install and/or rebuild HPC cluster nodes (including front ends, compute, and administration nodes) in diskful, disklite, and diskless environments; deliver consistent configuration management within the HPC environment.
* Monitor, troubleshoot, and assist with capacity planning for network fabrics such as InfiniBand and Ethernet, and file systems such as NFS and Lustre.
* Manage, extend, and develop customized scripts to support the HPC user, monitoring, and system administration environments.
* Follow formal change and configuration management practices, ingrained into the daily operations, to ensure changes and configuration control implementations are properly documented and approved.
* Ensure the customer, support team, and HPC users and other stakeholders are properly engaged and informed throughout any system changes, upgrades, or problem mitigation efforts; utilizing verbal and written communication, task-management, and follow-up.
* Support HPC system Users, leveraging the helpdesk ticketing system.
* Develop, improve, and enhance user and system administration online documentation repositories; Utilize existing platforms to support the various documentation efforts; contribute to selection of and migration to new platforms, if needed.
* Perform break/fix support, which may include node, blade, or board-level replacements, replacement of backplanes, failed DIMMs, hard drives, controller boards, failed cables, network switches, and other failed components.
* Troubleshoot and repair in conjunction with OEM vendors to identify root cause of issues, their timely resolution, and restoration of SLA-related services.
* Manage and maintain spare part inventories; perform tracking, shipping, receiving of vendor RMAs.

Qualifications:

Must have:

* 5 or more years of experience in Linux Systems Administration.
* 2 or more years of direct break/fix experience within a high performance computing environment.
* Bachelor's degree, or equivalent, in computer-related field, CS preferred.
* Hands-on experience with computer hardware maintenance, such as replacing processors, DIMMs, disk drives, PCIe cards and other field-replaceable components.
* Proficiency with installing and removing software for Linux, including pre-built packages, and compiling from source.
* Experience with configuration and use of provisioning tools, such as xCAT.
* Experience managing NFS, Lustre, or other NAS and parallel file systems.
* Familiarity with monitoring and metrics gathering tools such as Nagios, Zabbix, collectl, rancid, and PCP;
* Familiarity with basic networking technologies, components, and tools, with a solid understanding of subnet and routing concepts.
* Experience with HPC batch scheduling and queuing systems.
* Programming experience and familiarity with scripting languages in the Linux/Unix environment such as bash, awk, Perl, Python, and expect; proficiency in one or more of the above.
* Proficiency with documentation, collaboration, and task management tools, such as the Microsoft Suite (Project, Visio, Word, and Excel), Google Docs, MediaWiki, and Trac.

REQUIRED SKILLS
* Reading Comprehension- Understanding written sentences and paragraphs in work related documents.
* Active Listening- Giving full attention to what other people are saying, taking time to understand the points being made, asking questions as appropriate, and not interrupting at inappropriate times.
* Speaking- Talking to others to convey information effectively.
* Critical Thinking- Using logic and reasoning to identify the strengths and weaknesses of alternative solutions, conclusions or approaches to problems.
* Complex Problem Solving- Identifying complex problems and reviewing related information to develop and evaluate options and implement solutions.
* Judgment and Decision Making- Considering the relative costs and benefits of potential actions to choose the most appropriate one.

Desired Attributes:

* Team player, with the ability to work in with a diverse team in both local and remote technical support environments.
* Resourceful, with initiative to perform independent technical troubleshooting and identify/recommend solutions and improvements.
* Disciplined troubleshooting skills balanced with creative problem-solving skills, to tackle highly complex large-scale technical problems.
* Attention to detail in the areas such as time management, pre-planning, analytical thinking, observation, and active listening.
* Exceptional verbal and written communication skills using the English language.
* Demonstrate the willingness to continually expand current technical and soft skillsets.

Nice to have:

* 10 or more years of experience in High Performance Computing Systems.
* Daily experience using helpdesk ticketing software; familiarity with OTRS is a plus.
* Extensive experience with RedHat/CentOS server administration.
* Familiarity with developing and maintaining a high-speed network and MPI software stack.
* Experience developing software in C and/or Python; including revision control using tools such as Git.
* Experience with configuration management tools, such as Ansible and Puppet, and a strong interest in automation of common tasks.
* Experience with high-speed/low-latency networks such as InfiniBand; Proficiency with debugging and tuning Mellanox IB and configuration of OpenSM is a strong plus.
* Hands-on experience with network hardware maintenance, including cable management and labeling and the ability to identify common cable and connector types.
* Working knowledge of the Slurm batch scheduling system (or extended knowledge of Torque/Moab).
* Working knowledge of Hitachi HNAS storage appliances.
* Proficiency with Lustre admin tools; experience with the DDN EXAScaler and/or HPE-Cray ClusterStor products is a plus.
* Familiarity with container creation and management, using Docker and Singularity; experience implementing Virtual Machines; ability to manage orchestration of the container and virtualization environments.
* Familiarity with modern messaging, alerting and display frameworks such as Kafka, ElasticSearch, Logstash, Kibana, Alerta, and/or Grafana.

EDUCATION/Experience
* 5 or more years of experience in Linux Systems Administration.
* 2 or more years of direct break/fix experience within a high performance computing environment.
* Bachelor's degree, or equivalent, in computer-related field, CS preferred.
* 10 or more years of experience in High Performance Computing Systems, desired.

The likely salary range for this position is $104,000 - $156,000, this is not, however, a guarantee of compensation or salary; rather, salary will be set based on experience, geographic location and possibly contractual requirements and could fall outside of this range.

Scheduled Weekly Hours:
40

Travel Required:
10-25%

T elecommuting Options:
Some Telecommuting Allowed

Work Location:
USA CO Boulder

Additional Work Locations:

Total Rewards at GDIT:
Our benefits package for all US-based employees includes a variety of medical plan options, some with Health Savings Accounts, dental plan options, a vision plan, and a 401(k) plan offering the ability to contribute both pre and post-tax dollars up to the IRS annual limits and receive a company match. To encourage work/life balance, GDIT offers employees full flex work weeks where possible and a variety of paid time off plans, including vacation, sick and personal time, holidays, paid parental, military, bereavement and jury duty leave. To ensure our employees are able to protect their income, other offerings such as short and long-term disability benefits, life, accidental death and dismemberment, personal accident, critical illness and business travel and accident insurance are provided or available. We regularly review our Total Rewards package to ensure our offerings are competitive and reflect what our employees have told us they value most.

We are GDIT. The people supporting some of the most complex government, defense, and intelligence projects across the country. We deliver. Bringing the expertise needed to understand and advance critical missions. We transform. Shifting the ways clients invest in, integrate, and innovate technology solutions. We ensure today is safe and tomorrow is smarter. We are there. On the ground, beside our clients, in the lab, and everywhere in between. Offering the technology transformations, strategy, and mission services needed to get the job done.nnGDIT is an Equal Opportunity/Affirmative Action employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status, or any other protected class.

PI131033663