| 2021 - Present |
Open World Tech |
| |
B2B Technology Consulting company, focusing on automation, scalability, serverless using Public Clouds (AWS, GCP, etc) and GitOps based CD/CI, using Open Source Tech and Tools |
| |
Co-Founder |
| |
|
| |
Cloud and Serverless Tech Development and Consulting |
| |
• Developing and Deploying Gitops based automated and scalable solutions for Web, Mobile, API, Dedicated Game Server and Pixel Streaming applications |
| |
• Focusing on Open Source software, tool sets and integrations |
| |
|
| May 2021 - Mar 2023 |
Improbable Worlds |
| |
British multinational company focusing on technology to support large scale games, metaverse, and virtual worlds/events |
| |
Sr LiveOps Engineer |
| |
|
| |
Infrastructure and Application Deployment and Management |
| |
• Work closely with the LiveOps, DevOps and Dev Teams to build, test and deploy scalable application and infrastructure Tech stacks based on Terraform, Dockerized Applications, Helm Charts, Kubernetes Clusters, etc. |
| |
• Plan, Develop, Test and Deploy large-scale system upgrades, with additional HA redundacy and monitoring to reduce the risk of Customer or User impact. |
| |
• Develop and upgrade dashboards and alerts using SLOs, SLAs, KPI metrics and logs. |
| |
|
| 2019 - 2021 |
Sage Intacct |
| |
British software company focusing on financial services and management |
| |
Sr SRE, Sr Site Reliability Engineer |
| |
|
| |
Infrastructure and Application Monitoring and Alerting |
| |
• Work closely with the SRE, Cloud Ops and DBA teams to build, test and deploy scalable application and infrastructure Tech stacks based on custom Python, Bash and Config Management Tools. |
| |
• Complete a deep-dive review of the existing Metrics Collection, Storage and Visualizations infrastructure. |
| |
• Complete full evaluation and scoring for several modern industry standard alternatives against existing requirements and desires for a complete revamp/replacement. |
| |
• Collect feedback from various stake-holder Teams and individuals about scoring values for viable alternatives. |
| |
• Architect a full project plan to deploy and migrate to a newly developed metrics collection and storage solution, while carefully scaling back and retiring the legacy system. |
| |
|
| 2012 - 2019 |
Sony Interactive Entertainment, Playstation |
| |
Playstation Now, a global video game streaming platform |
| |
SRE, Site Reliability Engineer |
| |
|
| |
Infrastructure and Application Monitoring and Alerting |
| |
• Worked directly with Onsite DC and “Remote Hands” Engineers to deploy thousands of new servers and network hardware to dozens of datacenters and POPs in countries around the world |
| |
• Define and update KPIs, SLOs, SLIs, SLAs, metrics and alerting |
| |
• Design and develop solutions to collect, search and visualize metrics and fire alerts and notifications to appropriate Teams based on KPI and SLA breaches. (Prometheus, Grafana, Ansible, Opsgenie, Kubernetes, Gitlab CDCI, Fluentd, Rsyslog, etc.) |
| |
• Design and develop solutions to collect, search and visualize logs and events and fire alerts and notifications to appropriate Teams based on application errors, logs and KPI breaches. (Elasticsearch, Kibana, Prometheus, Grafana, Ansible, Opsgenie, Kubernetes, Gitlab CDCI, Fluentd, Rsyslog, etc.) |
| |
|
| |
Automation and Hands-on Operations |
| |
• Configure and maintain Amazon Web Services (AWS) and Google Cloud Platform (GCP) cloud computing environments |
| |
• Perform operational tasks to mitigate major (business or customer impacting) incidents, or unblock Team members, where automation is not yet in place. |
| |
• Develop operational tooling, for “one off” updates and playbook automation |
| |
• Improve automation for systems inventory updates and configuration management |
| |
• Optimize and improve SDLC/CDCI pipeline, processes and infrastructure |
| |
|
| |
Solutions Architecture |
| |
• Perform requirements gathering and resource planning for new projects |
| |
• Research and evaluate industry standard solutions |
| |
• Evaluate and compare onsite, private, public cloud service options and offerings, including feasibility, compatibility, security, compliance and TCO evaluations |
| |
• Maintain up-to-date understanding of all mission critical infrastructure, service architecture and updates |
| |
• Document, communicate and advocate for SRE best practices throughout the company |
| |
|
| |
Technical Lead and Project Management |
| |
• Manage project timelines, deliverables and resource planning |
| |
• Lead architecture and design sessions for cross team projects |
| |
• Provide cross-team architectural consulting, production readiness review and validation |
| |
|
| |
SRE Team Building |
| |
• Proactively help to build and scale out an effective global SRE Team |
| |
• Review, interview and screen potential SRE candidates |
| |
• Train and mentor team members |
| |
• Develop, maintain and update candidate screening and interview procedures and processes |
| |
• Update and maintain “New SRE” startup and training materials |
| |
|
| |
Incident Management Process and Reporting |
| |
• Participate in Oncall Rotation |
| |
• Develop and Maintain Incident Management and Review Processes |
| |
• Develop and communicate RCA and issue mitigation plans |
| |
• Refine and improve KPIs, SLOs, SLIs, SLAs, metrics and alerting, based on incidents and discovered observability gaps |
| |
• Perform and report RCA and Postmortem findings |
| |
• Troubleshoot/break fix and/or escalate discovered issues to relevant teams or engineers |
| |
|
| 2010 - 2012 |
Digital Domain - Vancouver, BC and Port St Lucie, FL |
| |
Sr. Systems Admin and Sr. Systems Engineer |
| |
On-screen Credits: The Legend of Tembo, Jack the Giant Killer, Transformers 3, Tron Legacy, Thor |
| |
|
| |
Systems and Infrastructure: |
| |
• Worked to duplicate, setup and integrate Linux environments for new 200 seat and then new 500 seat VFX Studios. Which included 200+ HP Workstations, 1000+ HP High Density Blade Servers and 100+TB of Isilon or NetApp Enterprise class Storage, and high performance Brocade switching environment. |
| |
• Setup, configure, and maintain OS and Software installation and configuration management systems ( Redhat Kickstart, Onesis, Puppet, CFEngine, etc). |
| |
• Worked with sister companies in US and Canada to integrate VFX Pipeline and Software synchronization. Including CentOS Linux operating system updates and changes, site specific software package installations and deployments, etc. |
| |
• Worked with Linux Kickstart, Onesis and Puppet to setup fully automated bare metal installs for CentOS Linux Operating systems, custom packages, connections to shared storage, custom CG Pipeline and Toolset, etc. |
| |
• Worked to develop scripts and procedures to bind CentOS Linux and MacOSX workstations and servers to Windows Server 2008 via LDAP with Kerberos encryption. |
| |
• Acted as Lead Support for all Render Queueing and Job Management, including automation and scripting. |
| |
• Handled large scale file system sorting, cleanup, transfers, and digital delivery packaging. |
| |
• Configured Symantec Netbackup to run daily, weekly and monthly backups. As well as final show archiving, removals and restorations. |
| |
• Worked with VMWare ESXi Server to deploy, maintain and balance several key server VMs. |
| |
• Setup and maintain Monitoring and Alerting systems for all Storage, Networking, Servers and Workstations for the Studios. |
| |
• Acted as Level 2 and 3 technical support for all Linux and Unix based issues with all Workstations and Servers. |
| |
• Provided detailed documentation and training for Level 1 and Level 2 Technical Support to handle commonly occurring issues. |
| |
|
| 2007 - 2009 |
Keystone Pictures |
| |
Visual Effects, Lead Technical Director, Technical Supervisor |
| |
Onscreen Credits : The “Buddies” Series (Space Buddies, Santa Buddies, Adventure Buddies, etc. ) |
| |
|
| |
Systems and Infrastructure: |
| |
• Worked with several hardware and software vendors to install and configure a 100 SGI Linux 1U render nodes, 25 MacPro Workstations, and a 40TB SGI Raid Storage Server, connected through a new HP ProCurve Gigabyte network. |
| |
• Developed a clone-able dual-boot MacOSX and Fedora Core Linux system install for the studio’s 25 MacPro Workstations. |
| |
• Developed a clone-able Fedora Core Linux based system install for the Studio’s 100 Render nodes, using Render Management through PipelineFX’s Qube. |
| |
• Manage and Support the Studio’s Render-farm with 100 Linux 1U RenderNodes and 25 MacPro workstations. |
| |
|
| |
Render and Color Pipeline: |
| |
• Worked with the CG Supervisor to help develop an AOV based render work-flow for workstations and renderfarm using Mental Ray 3.6. |
| |
• Developed a LUT to translate between the 10Bit Log Panasonic Genesis Camera format to linear, and back within Shake 4.1. |
| |
|
| |
Character Lighting and Fluid FX: |
| |
• “Finaled” the Lighting and Rendering of 65 animated face replacement shots. |
| |
• “Finaled” all in-house Fluid FX using Maya 2009 and Houdini Master 10. Including dust, smoke, clouds, rocket thrusters, etc. |
| |
|
| 2000 - 2007 |
American Museum of Natural History (https://www.amnh.org/) |
| |
Rose Center Engineering (RCE), Rose Center Productions (RCP) and Science Bulletins (SciBul) Departments |
| |
Technical Director/Unix Systems Administrator |
| |
|
| |
Systems and Infrastructure: |
| |
• Began working with Engineering and Productions, which is a group of about 15 VisualFX Artists, System Administrators, Video Engineers, and Production Staff responsible for developing, maintaining and upgrading all Computers, Video Systems and Video Content for the Digital Dome and Space Shows. This includes two SGI OnyxII Reality Monster Super Computers, several SGI Octanes, O2, Linux and Windows graphics workstations, and 7xHDTV and 4xHDTV Projector Theaters. |
| |
• Worked with Systems Admins and Video Engineers to Design, Create and Test a 7 Node Linux Graphics Cluster for Interactive 3D and Digital Dailies playback in the Hayden Planetarium in preparation for the upcoming show. This system was based on non-proprietary, commodity-based hardware (Dual AMD64, Nvidia Quadro FX 4400, etc) and software (Linux, PiranhaHD). |
| |
|
| |
Full Dome Visual FX and Animation: |
| |
• Worked with the Art Director to design, model and animate “Feather Dream”, which comprises 2 of the 36 Minute Planetarium Music Show entitled “Sonic Vision”. Additionally, worked to create several background elements and transitions between other shots within the show. “Feather Dream” was created using Maya6 and Shake3.5. |
| |
• Created two 2.5 minute quarterly news animation sequences for the Science Bulletins Department at AMNH using Partiview, Uniview, Maya7, Shake3.5 and PiranhaHD, which were recorded to HDCam and then encoded to HDTV Mpeg2. For playback to visitors within AMNH, and distributed to a network of Museums and Educational Institutions around the world via the Internet. |
| |
|
| 1999 - 2000 |
New York Institute of Technology (https://www.nyit.edu) |
| |
Advanced Computer Graphics Department |
| |
SGI/Unix Systems Admin |
| |
|
| |
Systems and Infrastructure: |
| |
• Maintained and supported graphics software and hardware for Computer Graphics Labs in Manhattan, NY. Including Silicon Graphics (Unix) Workstations, Avid Video Editor, Softimage3D, Alias Wavefront, etc. |
