Is Haskell-Beam Deserialization slow?

[imsaravana369]

I'm using Beam in Haskell to load around 14.5 million rows from a PostgreSQL table (for benchmarking purpose)
Posted this in Stackoverflow(but got rejected🥲): https://stackoverflow.com/questions/79687504/is-haskell-beam-deserialization-slow

Here’s the simplified version of my query:

selectAllImdbNames :: Connection -> IO [ImdbName]
selectAllImdbNames conn =
  runBeamPostgresDebug putStrLn conn $
    runSelectReturningList $
      select $
        all_ (imdbNames imdbDb)
This takes ~110 seconds(network latency we can discount as im running the db locally), whereas an equivalent Rust query (using tokio-postgres) loads the same data in ~18 seconds.

Here's the Table

data ImdbNameT f = ImdbName
  { nconst :: Columnar f BS.ByteString,
    primaryname :: Columnar f (Maybe BS.ByteString),
    birthyear :: Columnar f (Maybe Int32),
    deathyear :: Columnar f (Maybe Int32),
    primaryprofession :: Columnar f (Maybe BS.ByteString),
    knownfortitles :: Columnar f (Maybe BS.ByteString)
  }
  deriving (Generic)

Total rows: 14,519,095

PostgreSQL query execution itself takes only ~4s
When I rewrote the same query using postgres-simple, the load time dropped to ~60s (which is still slow)
because the profiling told something like

COST CENTRE                       MODULE                               SRC                                                            %time %alloc

getvalue'.\                       Database.PostgreSQL.LibPQ            src/Database/PostgreSQL/LibPQ.hs:(1170,30)-(1178,58)            14.3    2.5
runPgRowReader.step               Database.Beam.Postgres.Connection    Database/Beam/Postgres/Connection.hs:(142,5)-(181,21)           13.1   15.0
ftype.\                           Database.PostgreSQL.LibPQ            src/Database/PostgreSQL/LibPQ.hs:1101:57-76                      4.8    0.6
withResult                        Database.PostgreSQL.LibPQ            src/Database/PostgreSQL/LibPQ.hs:1860:1-46                       4.4    4.8

Is the overhead coming from libpq(getValue') which beam and postgres-simple uses or the free monad that beam relies on?

How can I mitigate this problem?

[LaurentRDC]

Hmm this is interesting, thanks for your benchmark!

The beam PostgreSQL backend is based on postgresql-simple. If it easy for you, could you conduct the same experiment using postgresql-simple?

Another potential idea: did you compile the Haskell executable using the maximum amount of optimization? I don't know if the base level of optimization in Rust is different.

[imsaravana369]

I did try with postgres-simple and it was able to resolve in ~60sec. Will run with max optimization and let you know

[LaurentRDC]

Thanks, this is already quite a large difference.
Based on your findings with optimization, it would be a good idea to create an issue. The beam-postgres backend should be around the same speed as postgresql-simple.

[imsaravana369]

Even after -O2 I'm getting the same. I have created a repo.
https://github.com/imsaravana369/beam-benchamark-demo
For some reason: Now the time is 90sec (not 110sec for beam), for simple-postgres its still (60s)

[LaurentRDC]

Awesome, thank you for your analysis. Could you create an issue? This should definitely be addressed

Edit : I created issue #769