Native query engine
2025 to now · AWSTechnical lead for the performance and stability of a native data analytics accelerator, co-leading the GA performance effort. More once it is public.
Panagiotis (Panos) Garefalakis
Staff Software Engineer, Data Analytics · San Francisco Bay Area
I enjoy building runtimes for large-scale data analytics engines! Today I am the tech lead of the Upper Engine team at Amazon EMR, where I focus on the performance and stability of our native query engine: vectorized execution, adaptive runtime filtering, and query optimization.
Earlier I was part of the core team that launched a managed, serverless stream processing service at Confluent, and improved the query runtimes behind the cloud data warehouse at Cloudera. Before that, a PhD at Imperial College London on resource management for shared compute clusters, supervised by Peter Pietzuch, with two summers at Microsoft's research lab in Silicon Valley. I am a PMC member on Apache ORC and a committer on Apache Hive.
Highlights
-
2025 to nowTech lead of the Upper Engine team at Amazon EMR, AWS.
-
2022 to 2025Part of the core team that took Confluent's Apache Flink service from inception to general availability across all three major clouds: elastic autoscaling with no capacity to size, an always-current runtime, usage-based billing, and a 99.99% uptime SLA.
-
2019 to 2022Query runtime behind the cloud data warehouse at Cloudera, and open source work in Apache ORC and Apache Hive.
-
2016 and 2017Research intern over two summers at Microsoft's Cloud and Information Services Lab, Silicon Valley, on scheduling for long-running containers in production clusters.
-
2015 to 2019PhD on resource management for shared compute clusters. The work shipped as placement constraints in Apache Hadoop YARN 3.1 and appeared at EuroSys and SoCC.
Selected work
Flink exception classification
2022 to 2025 · ConfluentDesigned, implemented and delivered exception classification, the single biggest usability improvement and adoption enabler for the Flink GA. It surfaces actionable, user-facing errors across more than 1M jobs per week, holding unclassified exceptions below 0.1%. Core contributor for Flink Java user defined functions and lead for Python UDFs. Delivered a new user code registry (Artifact Service) that validates and runs customer code across clouds and regions, plus a Validation API reused by three other products. Introduced feature-flagged SHIM versioning that cut UDF hotfix turnaround from weeks to minutes.
Runtime filtering in Hive and ORC
2019 to 2022 · ClouderaDesigned, implemented and delivered several enhancements to Cloudera's scalable query processing engine, such as lazy decoding (row-level filtering) and data-induced query predicates, improving query latency by up to 50%. Lazy decoding is an open source feature that cuts query runtime for production workloads by saving memory allocations and CPU cycles across long-running executors on large-scale clusters.
Open source
- Apache ORC PMC member
- Apache Hive Committer
- Apache Gluten Contributor
- Apache Hadoop YARN 3.1 placement constraints
- Apache Flink FLIP-304
- Apache Spark Coroutine-based task suspension