Performance Optimizations
++
+ From July 10 to 12, we organized the first Flow PHP hackathon together with + Józef (@stloyd) + and Aleksander (@MrHDOLEK). +
++ We had one goal: improve general Flow performance. We created a dedicated GitHub label for the event that + groups together all the work we pushed through those 3 days. Those changes also pushed me to redesign Flow's + serialization/hydration architecture and invent a new binary data format, called Floe. +
++ GitHub Label: hackaton-07-2026 +
+Benchmarks
++ Before I explain what and how we changed, take a look at the numbers. +
+
+ Benchmarking Method
+ 100k rows, same dataset files, same machine, PHP 8.3 + opcache, phpbench (mode of 3 iterations, 1 warmup).
+ Delta = (1.0 - 0.41.1) / 0.41.1; negative is better.
+ 1.0 pipelines read input from Floe format,
+ 0.41.1 from Parquet (Floe did not exist yet); timings include input reading.
+
{% apply escape %}{% include template_folder ~ '/benchmark.php' %}{% endapply %}DataFrame Operations
+| Benchmark | +Rows | +Time 0.41.1 | +Time 1.0 | +Mem 0.41.1 | +Mem 1.0 | +Δ time | +Δ mem | +
|---|---|---|---|---|---|---|---|
| Join inner | +100k | +8.03s | +3.35s | +44.9mb | +30.8mb | +-58.3% | +-31.4% | +
| Join left | +100k | +8.23s | +3.41s | +44.9mb | +30.8mb | +-58.6% | +-31.4% | +
| Join left anti | +100k | +7.74s | +2.53s | +44.9mb | +29.5mb | +-67.3% | +-34.4% | +
| Join right | +100k | +8.27s | +3.33s | +44.9mb | +30.9mb | +-59.7% | +-31.3% | +
| Cross join | +100k | +5.26s | +1.36s | +45.7mb | +31.1mb | +-74.2% | +-32.0% | +
| Join each | +100k | +6.32s | +1.77s | +45.8mb | +30.0mb | +-71.9% | +-34.6% | +
| Sort (external) | +100k | +25.08s | +2.64s | +115.0mb | +93.3mb | +-89.5% | +-18.9% | +
| Group by | +100k | +4.59s | +1.32s | +44.1mb | +23.9mb | +-71.2% | +-45.9% | +
| Group by + aggregates | +100k | +5.85s | +1.85s | +44.1mb | +24.1mb | +-68.4% | +-45.5% | +
| Filter | +100k | +4.96s | +0.87s | +36.9mb | +23.7mb | +-82.4% | +-35.7% | +
| Schema inference | +100k | +6.85s | +1.61s | +44.8mb | +29.0mb | +-76.5% | +-35.2% | +
| Cache write | +100k | +5.91s | +2.10s | +63.6mb | +34.9mb | +-64.5% | +-45.2% | +
| Cache read | +100k | +4.37s | +0.62s | +133.7mb | +34.9mb | +-85.8% | +-73.9% | +
Data Formats & Storage
+| Benchmark | +Rows | +Time 0.41.1 | +Time 1.0 | +Mem 0.41.1 | +Mem 1.0 | +Δ time | +Δ mem | +
|---|---|---|---|---|---|---|---|
| CSV read | +100k | +2.52s | +2.57s | +9.1mb | +9.6mb | ++1.9% | ++5.2% | +
| CSV write | +100k | +5.40s | +1.32s | +22.8mb | +23.4mb | +-75.6% | ++2.5% | +
| JSON read | +100k | +4.29s | +3.91s | +9.1mb | +10.1mb | +-8.9% | ++10.9% | +
| JSON write | +100k | +5.35s | +1.58s | +22.8mb | +24.3mb | +-70.5% | ++6.5% | +
| Parquet read (php) | +100k | +6.60s | +3.97s | +210.1mb | +216.4mb | +-39.9% | ++3.0% | +
| Parquet read (arrow) | +100k | +3.82s | +1.42s | +22.7mb | +23.1mb | +-62.8% | ++1.7% | +
| Parquet write (php) | +100k | +17.82s | +4.22s | +212.3mb | +45.8mb | +-76.3% | +-78.4% | +
| Parquet write (arrow) | +100k | +5.79s | +1.79s | +22.9mb | +23.6mb | +-69.0% | ++3.3% | +
| XML read | +100k | +4.56s | +4.59s | +105.8mb | +105.7mb | ++0.5% | +-0.1% | +
| XML write | +100k | +14.76s | +6.24s | +22.8mb | +23.7mb | +-57.7% | ++4.1% | +
| Excel read | +100k | +14.21s | +13.01s | +9.1mb | +9.7mb | +-8.5% | ++7.1% | +
| Excel write | +100k | +11.66s | +7.40s | +22.9mb | +22.8mb | +-36.5% | +-0.6% | +
| Text read | +100k | +0.50s | +0.21s | +9.1mb | +8.9mb | +-57.5% | +-1.9% | +
| Text write | +100k | +4.88s | +0.84s | +22.8mb | +17.5mb | +-82.7% | +-23.2% | +
| PostgreSQL write | +100k | +8.57s | +5.37s | +45.0mb | +33.3mb | +-37.4% | +-26.0% | +
| PostgreSQL read | +100k | +8.05s | +7.06s | +9.1mb | +10.9mb | +-12.3% | ++19.9% | +
| Doctrine write | +100k | +9.61s | +6.10s | +44.6mb | +33.3mb | +-36.5% | +-25.3% | +
| Doctrine read | +100k | +6.61s | +6.32s | +9.1mb | +10.3mb | +-4.4% | ++13.1% | +
+ + Parquet engines: both versions ship php and arrow engines. 1.0 pins the engine explicitly per benchmark; + 0.41.1 auto-selects (arrow whenever ext-arrow is loaded), so its php rows were measured with ext-arrow + disabled and its arrow rows with ext-arrow loaded. On 0.41.1 the selected engine applies to both the + input + read and the write inside the write benchmark; on 1.0 the write benchmark reads its input from Floe. + +
+Floe
++ A floe (or ice floe) is a large, flat sheet of ice floating on the ocean, a river, or a lake. ++
+ Floe is a dedicated, immutable binary data format created to speed up serialization/hydration of Flow Rows. + It combines ideas from Parquet & Avro, optimized for PHP and Flow Rows. +
++ Floe is a row-based binary format, with metadata stored as JSON in the footer. It is self-descriptive, which + means the format provides a schema (also in the footer) that supports deeply nested data structures like + lists/rows/maps. +
++ Floe also supports schema evolution without breaking backward compatibility. This means we can add new + columns between write batches, as long as those columns are nullable. +
+ +Encoders and Hydrator
+ +
+ Introducing Floe was just the first step to optimize performance. It allowed us to replace the old Serializer
+ based on PHP's serialize & unserialize functions.
+
+ The diagram below shows how Flow used to read and convert data from datasets into Rows, and then back to the + format expected by the destination storage. +
+ }})
+ + As you can see, EntryFactory was responsible for encoding the values coming from a data source and casting them + to a specific schema, and then each loader converted them back to whatever format the destination storage expected. +
++ That was one of our biggest bottlenecks. Because of how generic it was, EntryFactory always cast data into a + specific format, regardless of which data format it was reading from. You might ask: what is wrong with this approach? + Most data formats like CSV, JSON or PostgreSQL return data as strings or simple scalar values anyway. +
++ That is correct, but there are also data formats that are self-descriptive (they come with a schema), like Parquet or Floe. + For those, casting — especially of nested data structures like lists / structures / maps — was redundant. That redundancy + is not so painful when working with a few thousand rows, but once we move beyond 100k, the pain becomes real. +
++ So with our new architecture, each data format provides its own dedicated Encoder/Decoder, which internally decides + how to handle data just after reading it from the source and just before we write it to the destination. +
+{% apply escape %}{% include template_folder ~ '/encoder.php' %}{% endapply %}+ Once our data is encoded into its plain PHP representation, we pass it to the Hydrator, whose only goal is to create the Rows instances + based on the passed RawRowValues. +
+ +{% apply escape %}{% include template_folder ~ '/hydrator.php' %}{% endapply %}
+ This separation also allowed us to create two instances of the Hydrator: one based on PHP that internally uses EntryFactory, and a
+ Rust-based hydrator available when flow-php/flow-ext is installed.
+
+ The Hydrator is not the only place where we managed to leverage the PHP extension. We used a similar pattern (php / native) for Floe's dedicated Encoder, + AdaptiveFloeEncoder, + which let us squeeze out even more performance. +
+ }})
+ Caching / Sorting / Grouping / Joining
++ Since all the mechanisms mentioned above need to dump Rows from memory to disk (or other storage) at some point, Floe became + the default serialization format, which allowed us to significantly improve the performance of those operations. +
++ But we had to slightly change the Serializer contract, which previously attempted to serialize/unserialize any object. + Now it focuses only on Rows, which can really represent any data structure. +
+{% apply escape %}{% include template_folder ~ '/serializer.php' %}{% endapply %}+ The new serialization contract also expects source/destination streams to read from and write to. +
++
Bucketing
++ The final step of this refactoring was improving bucketing (breaking the dataset into smaller buckets to reduce memory consumption). +
+
+ Sort / Join / Group By - all 3 are directly affected by those changes, and can now operate on datasets of any size.
+ I believe there is still room for optimization, especially around Sort, which currently still takes around 100mb of RAM to sort 100k rows.
+
+ This is also not the final performance we are aiming for. All of the benchmarks above are executed in a single process. + True scalability will come with parallel processing, and that's another area where Floe will shine, since we'll be able to + move (shuffle) data between workers faster. +
++ Using PHP's native serializer, 100k rows produced a 1.5GB string; with Floe we managed to drop that to 36MB — and this is still before compression! +
++ The goal is to implement different compression algorithms and let the Floe reader/writer use them, like Snappy, which is pretty much the + industry standard today. +
+The end goal
++ These numbers might not feel very impressive yet, but joining 100k rows in under 3 seconds with only 30MB of RAM might + already be a good reason to move some processes outside the database and work on plain files. +
++ Flow's true performance boost will come with parallel processing, which will make speed depend on another variable — + the number of processes, and maybe even nodes, that can process ETL pipelines in parallel. +
++ That's our final goal, and I hope it will make Flow a real alternative to other data processing frameworks. +
+ +