Career Profile

Summary

A Consultant Architect and Software Engineer with extensive experience in delivering robust, distributed, web-scale systems using Cloud infrastructure and proven database technologies.

I have the insight and vision to understand both tactical and strategic business goals and how to deliver the maximum value where it matters. Whether through architecture, software and tools or by coaching and mentoring to help minimise cost.

Key Skills

  • Big Data, Solutions and Enterprise Architecture
  • Pragmatic approach to software design and implementation
  • Exceptional analytical and troubleshooting capabilities
  • Strong mentoring and coaching skills
  • Technology evaluation and due diligence
  • Proficient verbal and written communicator at all levels
  • Mentoring and technical leadership

Key Technical Capabilities

  • Scala
  • Java
  • Flink
  • Kafka - Streams, KSQL, Connect
  • Spark
  • Cassandra
  • Hadoop
  • Linux
  • Docker
  • AWS
  • GCP

Career History

Wayfair Europe, Data Engineering Consultant, July 2020 - Present

Initially working on a discovery phase of a project to design an appropriate target architecture for supplier data where I engaged with major stakeholders to understand the existing mechanisms that supply this data. As part of this work domain models were created to provide a common vocabulary between stakeholders and also to provide insight into where to target re-engineering efforts, adding the most value. Recently I architected and, with the team, engineered a near-realtime data pipeline that provides insight into some of the most valuable supplier metrics within minutes of them being produced. This replaced a legacy, daily batch ETL process that was poorly understood, complex and error prone.

Atom Bank, Consultant, April 2020 - May 2020

Working with the Data Management team I assessed their current architectural approach and assisted them to move to a more component based model which provides flexibility and scalability both in terms of development and also deployment.

  • Analysed existing codebase and architecture.
  • Implemented a componentised template target architecture.
  • Implemented a custom Kafka Connector providing pluggable persistence.
  • Provided implementations of component tests for Kafka Streams considerably reducing test/fix cycle times.
  • Mentoring, pair programming with staff to explain the concepts and benefits around TDD and CI.
  • Report on work carried out and recommendations on a technical roadmap provided to senior technical management.

Royal Sun Alliance, Consultant, January 2019 - June 2020

Implemented an archiving solution that extracts data from hundreds of homogenous Cassandra tables, transforming and masking the data so that it complies with GDPR regulations. Integrity checks are implemented numerous times to ensure that the data written to the output lake (HDFS) is identical to the source and has not been corrupted.

  • Implemented a configurable extraction process using Spark that includes many options to cater for the large variety of data structures.
  • Masking of data to comply with GDPR rules imposed.
  • Provides status of dates of extraction to ensure data is not extracted more than once.
  • Retry mechanisms implemented to cope with failure.
  • Integrity checking of data both during and after the extraction to the data lake has been successfully completed.
  • Automated build and deployment process using SBT, GIT, Jenkins and Ansible

Blis Media, Consultant, April 2018 - December 2018

An Ad-Tech company, Blis process billions of transactions per day producing 100’s of gigabytes of data which needs to be processed and ingested into their core platform. Blis’ data pipelines were poorly implemented and managed after many years of accruing technical debt due to poorly skilled engineers producing tactical solutions in order to deliver functionality quickly. This required constant firefighting in order to keep the core systems available. I worked with a small team to introduce stability, refactoring and re-engineering core data pipelines employing TDD and continuous integration.

  • Reversal of extensive technical debt.
  • Stabilisation of data pipelines to eliminate firefighting.
  • Stabilisation of volatile core Cassandra infrastructure.
  • Performance tuning of Cassandra data models to ensure efficient writing and retrieval of data.
  • Provisioning and tuning of Amazon EMR clusters.
  • Advising the CTO on a technical roadmap to improve the platform as a whole.

Camelot, Consultant, Sep 2017 - April 2018

Camelot’s Cassandra cluster faced major issues with repairs, latency and disk usage. I was tasked with providing a report on the underlying causes, providing recommendations on resolutions and subsequently creating and executing a plan to perform the required remedial action.

  • Performed extensive remodelling of data to ensure both partition sizes were manageable and that latency was reduced
  • Employed Spark to migrate across from the legacy data model and introduced Elassandra to enable multi-faceted searching, removing the need for secondary indexes and reducing the number of materialised views required.
  • Comprehensive performance testing using Gatling was carried out to ensure existing SLAs were maintained and in most cases surpassed.
  • Reduced disk usage to 1.5% of the existing solution (~1TB to 18GB).
  • Planned deployment and migration steps to reduce risk with zero downtime.
  • Provided general architectural advice and governance across the organisation, reducing the risk of overly complex solutions and project failure.
  • Mentored development teams on scalability and best practice.

HMRC, Technical Architect, Jun 2017 - Aug 2017

Worked under the Making Tax Digital programme, providing governance across the organisation.

  • Worked with the team to define the public facing API and provide governance on integration with legacy middleware systems.
  • Met with senior stakeholders to discuss impact of planned implementations and associated timelines.
  • Reviewed design documents providing feedback and highlighting concerns.

Lloyds Bank, Big Data Consultant, Mar 2017 - May 2017

Working with Sapient and Lloyd’s teams consulting on the use and implementation of Cassandra for the Open Banking API project and also as a strategic Big Data platform.

  • Provided detailed guidance to the senior Lloyd’s architecture group on the topology and deployment of their proposed Cassandra infrastructure for strategic use within the organisation.
  • Sized the proposed Cassandra cluster and derived appropriate data models based on business requirements, adopting an algorithmic approach.
  • Highlighting and documenting best practice and associated anti-patterns when deploying Cassandra on virtualised infrastructure and bare metal.
  • Consulted with business teams to understand data volumes, frequency and ingestion patterns in order to provide recommendations on configuration and proposed toplogies.
  • Implemented unit/integration framework using CassandraUnit to eliminate the need for external dependencies during the test lifecycle.
  • Assisted with the day to day running of teams following agile practices and providing guidance on best practice.

Datastax, Principal Solutions Architect, Dec 2014 - Mar 2017

Senior post sales, customer facing consultant operating in the EMEA region autonomously. This role required an in-depth understanding of Cassandra, distributed systems, scalability and associated DataStax technologies in order to provide pragmatic, accurate advice, recommendations and guidance to customers. Performing this role I was exposed to a huge range of diverse problem spaces and associated challenges.

  • Engaging with customers across Europe from a wide variety of verticles, providing architectural reviews and guidance covering the full stack: network, storage, operating system, software and Cassandra.
  • Performance testing at scale using both cassandra-stress and Gatling to provide a clear profile of how systems will behave in production.
  • Tuning of Data models, compaction strategies, JVM, GC and driver options to ensure optimal performance is achieved.
  • Implemented PoCs installing, configuring infrastructure, implementing driver software and providing detailed reports on key metrics.
  • Assisting with the development of a developer focused training course, implementing both UI and server side code.
  • Delivering both public and private training on the DataStax product suite and associated technologies.

Cisco, Cloud Web Security - Technical Leader, Aug 2013 - Dec 2014

As Technical Leader for the Core Services team I was responsible for all aspects of quality, technical innovation and excellence. I mentored and coached within the team, often pairing to lead by example and demonstrate good practice in design and implementation. My scope of influence also extended to the wider Secure Cloud Operations division, where I was frequently consulted on best approaches to scalability, SOA and Cloud infrastructure.

I designed and played a key part in implementing cloud-based event-driven notification and orchestration services. In addition I delivered messaging, logging, monitoring and storage infrastructure, providing a scalable platform which other teams have utilised and extended.

I acted as proxy Product Owner for the team helping to prioritise the backlog and also developing the Roadmap for the medium/long term.

Further career history available on request. Please contact me for more info.