Allen Institute, Seattle, Washington
Central IT Infrastructure Senior Manager
January 2023 - present
- Lead a team of nine system, storage, and network engineers supporting scientific and enterprise infrastructure on about 450 Linux and Windows servers
- Manage 30 PB of file and object storage on Isilon, VAST Data, 45Drives (Ceph), and NetApp equipment, and 12 PB in AWS and Azure
- Coordinate on-prem and cloud infrastructure technology strategy for the Institute
- Participate in migrating on-prem workloads to cloud platforms such as Amazon Web Services and Atlassian Cloud
- Created new departmental processes for on-call response and change management
- Worked with the Human Resources team to update titles and pay grades for team members
- Instituted well-defined training and certification paths for team members to enable growth in skills and careers
- Successfully made the case to upper management for substantial Central IT budget and staffing increases
- Helped develop a new org structure for an expanded Central IT department
Central IT Infrastructure Manager
April 2021 - January 2023
- Led a team of seven system, storage, and network engineers supporting scientific and enterprise infrastructure on about 450 Linux and Windows servers
- Administered systems built on HPE, Lenovo, Supermicro, and Pogo Linux hardware, including a 128-node HPC cluster running the Slurm workload manager
- Operated OpenStack, KVM, VMware, and Kubernetes virtualization/containerization platforms
- Managed 25 PB of file and object storage on Isilon, 45Drives (Ceph), and NetApp equipment, and nearly 10 PB in AWS and Azure
- Oversaw management of a network built with Cisco, Fortinet, and F5 gear
- Coordinated on-prem and cloud infrastructure technology strategy for the Institute
- Helped implement a top-to-bottom restructuring of cybersecurity posture
- Established structured project management based on Kanban methodology
- Worked to instill rigor in budgeting and spending
- Successfully restored morale to a historically dispirited team
Big Fish Games, Seattle, Washington
Senior Director of IT Systems
October 2017 - September 2020
- Led four teams of up to 24 IT professionals supporting ecommerce and enterprise infrastructure on more than 3000 Linux, Windows, and macOS servers
- Set technology direction for the company in the areas of systems, storage, databases, applications, and data center facilities
- Drove the investigation and deployment of Docker container technology
- Participated in planning and execution of a company headquarters office relocation
- Led data center consolidation efforts amidst a pivot to public cloud (GCP, Azure, and AWS)
- Integrated four teams into a newly reorganized group
- Guided teams through a second corporate acquisition
- Helped restructure the company's technology group using a DevOps organizational model
Director of Production Systems
December 2015 - October 2017
- Led two teams of up to ten system administrators supporting a global ecommerce environment on more than 2500 Linux servers
- Accelerated provisioning of game deployment environments from months to hours through improvements in process and automation
- Standardized server hardware platforms to improve ordering and troubleshooting processes
- Successfully reorganized teams to optimize workflow and employee engagement
- Redesigned ticket and project management to improve throughput
- Guided teams through the company's first acquisition by a new corporate parent
- Engaged in coaching and development of team managers
System Administration Senior Manager/Infrastructure Architect
June 2011 - December 2015
- Led a project-focused team of up to seven system administrators located in domestic and overseas offices
- Supported web, infrastructure, game, and development services on more than 2000 Linux servers sited in globally distributed data centers
- Administered systems built on HP, IBM, SeaMicro, and Sun hardware, as well as Xen, VMware, KVM, and OpenStack virtualization platforms
- Performed administrative tasks on NetApp file servers, and F5 BIG-IP and Citrix NetScaler load balancers
- Conducted team project planning, forecasting, and portfolio management
- Participated in promoting a healthy culture of change management
- Built a complete and accurate device database with correctness enforced by automation
- Initiated development of a service catalog and a tier-based support structure
- Improved processes, procedures, and documentation to reduce off-hours escalations by over 50% while quintupling the size and change rate of the infrastructure
- Augmented and improved the server configuration management system by:
- Adding process management capabilities and support for automated software package installations
- Refactoring the design and layout of configuration files
- Extending the use of configuration templates to reduce configuration variance
- Writing configuration style guidelines, as well as a tool to automatically check and enforce them
- Adding rigor to the production deployment process
- Accelerated the pace of server provisioning by eliminating or automating manual steps
- Drove deployment of microserver-based systems resulting in lower operational expenses and improved data center space and energy efficiency
- Performed extensive capacity planning and resizing exercises
- Supervised hardware and software lifecycle replacement activities
- Planned budgets and made purchasing decisions
- Rearchitected the centralized Linux account management system, which provided:
- Stronger security through the use of password aging and complexity checks
- Automated pre-expiration notices to users
- User and administrator ease-of-use in combination with security improvements
- Managed Akamai CDN and GTM properties, and developed an automated cache purging mechanism
- Led technical integration efforts for the acquisition of a remote software development office and data center site
- Practiced operations management principles derived from lean manufacturing, the theory of constraints, and new DevOps paradigms
Intelius, Bellevue, Washington
Tools & Release Engineering Manager/Senior System Administrator
November 2007 - June 2011
- Led a team of three people responsible for developing operations tools, releasing code to production, and supporting web applications
- Supported web, database, infrastructure, and development services on more than 400 Linux servers
- Administered systems built on HP and Supermicro hardware, as well as VMware and OpenVZ virtualization platforms
- Performed administrative tasks on F5 BIG-IP load balancers and Juniper NetScreen firewalls
- Served as a backup MySQL database administrator
- Worked with developers to identify operational requirements, characterize software behavior, and drive improvements and bug fixes
- Tracked down errors and bugs in production PHP code
- Conducted post-mortem analyses of service and software failures
- Prompted improved communication and change management processes within the operations department
- Implemented a centralized logging solution performing real-time analysis on log messages from servers, firewalls, load balancers, and switches
- Participated in migrating a production data center with no downtime
- Improved the effectiveness and organization of a configuration management infrastructure consisting of CFEngine, Kickstart, MySQL, and DHCP
- Streamlined and standardized the Linux and Windows DNS architecture for better manageability
- Ensured ubiquitous deployment of strictly configured host-based firewalls
- Developed a tool for continuous real-time recording of performance statistics from hundreds of Linux servers
- Guided development of web-based server management and monitoring applications
Philips Electronics North America, Bothell, Washington
UNIX Security Lead/Senior UNIX System Administrator
January 2004 - November 2007
- Supported corporate infrastructure, engineering, and ERP services on more than 500 Solaris, SunOS, HP-UX, AIX, and Linux systems
- Performed administrative tasks on Network Appliance filers
- Introduced measures to significantly enhance security and meet Sarbanes-Oxley (SOX), ISO, FDA, and other audit requirements, including:
- Establishing administrative control over all UNIX systems at local site
- Eliminating the use of cleartext network communication protocols
- Improving password encryption methods
- Automating password testing, with user notification and account locking
- Restricting privileged access
- Instituting a comprehensive and consistent system configuration process
- Supported the rollout of ITIL processes for incident and change management
- Provided primary support for UNIX directory services, including:
- Streamlining and standardizing the DNS server architecture for better manageability and security
- Deploying and retiring NIS domains
- Devising a method to incorporate UNIX systems into an Active Directory infrastructure using Samba and custom scripts
- Establishing consistent DNS and NIS client configurations
- Developed a centralized logging infrastructure based on syslog-ng, Simple Event Correlator (SEC), and custom tools, which provided:
- Aggregation of system and application logs
- Automated log reduction and analysis
- Real-time alerts and regular reporting of anomalous behavior, leading to greatly improved operational insight
- Secure source of event data for use in forensic analysis
GeoPacket, Seattle, Washington
Software Developer/Project Manager
March 2003 - October 2003
- Developed software to analyze packet characteristics for real-time determination of call quality on voice-over-IP (VoIP) networks
- Formulated a patented call quality assessment method
- Coordinated the efforts of a small team of software and network engineers
- Managed relations with the customer supporting application development
- Served as a founder and a member of the board of directors
Amazon.com, Seattle, Washington
UNIX Systems Engineer
November 2001 - November 2002
- Supported business-critical back-end services on more than 400 Linux, HP-UX, Solaris, and Tru64 servers
- Managed highly available services utilizing Cisco load balancers, HP ServiceGuard clusters, and other HA technologies
- Performed administrative tasks on HP storage arrays and NetApp file servers
- Accepted increasing responsibilities as several sysadmin functions were merged while headcount and capital expenditures were reduced
- Provided primary server support for crucial business functions, which at various times included:
- Inventory and supply chain management
- Corporate finance and human resources
- Electronic payments
- Fraud detection
- Customer service
- Oracle database services
- Delivered escalated response, resolution, and root cause analysis for critical events
- Conducted dependency and performance analysis and optimization
- Completed scaling requirements analysis and server deployment for the holiday season
- Planned and deployed the server architecture for a redundant secure payments site
- Assisted in developing and testing a corporate disaster recovery plan
- Implemented regular network scanning of outsourced customer service sites to monitor compliance with contractual agreements
- Improved the effectiveness and organization of a centralized configuration management system
edge2net, Kirkland, Washington
Network Services Engineer
March 2001 - October 2001
- Managed services for a global telecommunications network on about 20 Solaris, Linux, and AIX servers
- Provided primary support for many network services, including:
- Network monitoring (OpenView NNM, Big Brother)
- Directory services (DNS, LDAP)
- Voice-over-IP calling card IVR service
- Fax-over-IP mail service
- Conducted a transition of directory service from NIS to LDAP
- Secured network communications via a scalable, cross-platform IPsec solution
- Extended the capabilities of host and network monitoring platforms with custom software
- Implemented an automated file distribution mechanism used in providing IVR service and in-device configuration
- Instituted a comprehensive and consistent system configuration process
Bazillion, Seattle, Washington
Applications Group Manager
August 2000 - January 2001
- Led a team of six system administrators supporting Internet and messaging services on about 80 Linux, Solaris, CMC UNIX, and Windows servers
- Coordinated a merger of personnel and responsibilities resulting from a corporate acquisition, and grew the team to its final composition
- Conducted budget planning and made purchasing decisions
- Designed a highly scalable, fault-tolerant distributed service infrastructure
- Supervised testing and deployment of new unified messaging services
- Implemented comprehensive monitoring of hosts and network services
System Administrator
March 2000 - August 2000
- Supported Internet services for a nationwide startup communications provider on about 30 Linux and Solaris servers
- Provided primary support for system administration functions, including:
- Network information and authentication (DNS, DHCP, NTP, RADIUS), email, and Web services
- Service provisioning and usage collection
- Server capacity and budget planning
- Procedural documentation
- Abuse incident response
- Developed software to implement robust and secure provisioning and mediation of distributed Internet and VoIP services
- Automated gathering of metrics to aid in proactive troubleshooting and capacity planning
- Enhanced host and network security
- Instituted a comprehensive and consistent system configuration process
- Specified standard server, storage, and backup systems
Metapath Software International, Bellevue, Washington
Lead Security Administrator/Senior UNIX System Administrator
April 1998 - March 2000
- Supported corporate infrastructure services and software development on about 70 Solaris, HP-UX, Linux, DYNIX/ptx, and AIX servers
- Participated in managing a global network of more than 1000 nodes using Cisco, Ascend, Bay, 3Com, and Livingston network gear
- Managed a distributed perimeter security system composed of firewalls (Check Point, Gauntlet), choke routers, and secure Internet links (VPNs)
- Provided primary support for information services (DNS, NIS, NTP), electronic mail, and network news
- Rearchitected the global network infrastructures for email, information services, logging, and perimeter security
- Instituted a comprehensive and consistent system configuration process
- Implemented a security policy restricting privileged access
- Participated in a long-term program to improve corporate IT effectiveness
- Assisted in planning the relocation of the headquarters office
- Made purchasing recommendations and participated in budget planning
Test Engineer
February 1998 - April 1998
- Led testing of network driver software critical to a customer release
- Documented test results and procedures
- Trained colleagues in software testing techniques used
Customer Support Engineer
September 1997 - February 1998
- Supported operations in production network environments for wireless carriers using proprietary software and hardware
- Performed administrative functions on customer DYNIX/ptx, AIX, and LynxOS systems
- Performed basic Oracle database administration on production systems
- Installed telecommunications equipment at customer sites
- Composed documentation for use by customer support staff
GTE Network Management Organization, Bothell, Washington
Technical Consultant
June 1997 - August 1997
- Performed administrative tasks for a network of more than 150 Solaris and IRIX systems
- Automated network management tasks using shell scripts
- Implemented network performance monitoring
- Conducted performance tests of network file system protocols
- Administered a Network Appliance filer
- Assisted in planning and implementing a transition between
NIS+ and NIS
PSW Technologies, Austin, Texas
Associate Software Engineer
July 1996 - June 1997
- Supported corporate infrastructure services and software development on more than 300 NEXTSTEP, AIX, Solaris, Auspex, IRIX, and Windows systems
- Performed administrative tasks for a nationwide network using Cisco, Ascend, and Telebit gear
- Provided primary support for network security, account management, backups, network news, and resolution of internal support issues
- Maintained directory services (DNS, NIS, NetInfo, NT Domains), file services (NFS, Samba), printing facilities, and electronic mail
- Established a common software repository
- Implemented a security policy restricting privileged access
- Aided in the planning and execution of a successful corporate relocation, including:
- Restructuring the local network for a higher-bandwidth, switched architecture
- Assigning IP addresses in the renumbering of all network hosts
- Writing scripts and checklists used in reconfiguring hosts for the new network
- Provided NEXTSTEP, NetInfo, and general network consulting for external clients
Zilker Internet Park, Austin, Texas
System Administrator
November 1995 - July 1996
- Provided technical support to customers of a respected local Internet service provider
- Managed DNS, Web, email, Usenet, FTP, and other Internet services on SunOS and Solaris servers
- Maintained network connectivity for customers using Livingston, Ascend, and Specialix gear
- Developed Web tools for customers using Perl CGI scripts
- Organized staff meetings and training sessions
University of Texas Computation Center, Austin, Texas
Computer Programmer/Services Assistant
March 1995 - March 1996
- Provided front-line technical support to users working in UNIX, VMS, Macintosh, and Windows environments
- Composed documentation for use in standardized email responses
- Developed software used in the customer support environment
- Performed limited system administration tasks on SunOS, Solaris, AIX, Digital Unix, OSF/1, Ultrix, and VMS servers
- Trained and supervised new hires
Advanced Micro Devices, Austin, Texas
Co-op Associate Engineer
August 1990 - May 1991
- Constructed and documented computer models used to refine microprocessor fabrication processes
- Trained colleagues in the use of an Apollo Domain/OS engineering workstation and wrote a user's manual for its operation
- Performed backups and other routine system administration tasks
