13:55 <zao> Once again, I'm experimentative enough to consider using HPX for a problem that TBB doesn't quite cut it for.

14:11 <zao> Essay of questions incoming...

14:12 <zao> I've got some somewhat parallel data processing that I'd like to run a bit faster than the naive approach gets me.

14:12 <zao> These serve as the source data for a decompression and splitting step that produces around 50 gigs of data across 800k virtual files - these are also deduplicated into another datastore. The final output of one cycle of the program is a pair on manifest files that map from original "loose" and "bundled" paths to their content hashes for future serving to clients through another application.

14:12 <zao> At its core it's reading in around 30 gigs of compressed bundles across 40k files on slow HDDs. Each of these "loose files" is hashed and ingested into an LMDB datastore on disk keyed on the content hash for deduplication.

14:12 <zao> I've got some trouble figuring out how I would rate limit the different processing steps to avoid exceeding the system memory as my working set is way larger than the node RAM (80 GiB vs. 16 GiB for the dev box, less in production). With TBB and their pipeline system I could leverage tokens to avoid having too much data in flight.

14:12 <zao> I need some OS thread affinity for the datastore as it leverages regular OS thread TLS which I believe I can achieve by making a separate small thread pool for specific tasks in a partitioner callback? How do I deal with costly I/O with some but low parallelism, another dedicated thread pool?

14:12 <zao> I'm looking at HPX dataflow for the overall program flow but can't quite figure out how I'd limit the amount of loose file data that's in flight, and similarly avoid intermediary read/write steps from bunching up too much on the slowest part of the system. Can I model this kind of credit/limit system in HPX somehow?

14:27 <hkaiser> zao: do you rely on external libraries to do the threading?

14:30 <zao> I have no such thing at the moment, only reached for TBB to see how well their things would work.

14:31 <zao> Library-wise I've got LMDB for data access, a task-safe custom decompression library and a task-safe library of mine for interpreting the index. Standard I/O-streams for the initial file reading, no memory mapping or other amortization there as I want the data resident rather than spreading the I/O cost everywhere.

14:32 <zao> LMDB explicitly warns about green threading and needs real threads in serial, at least for write transactions.

14:32 <hkaiser_> zao: just asking

14:32 <zao> At some point in time I'd like to gather up multiple DB writes into larger write transactions as well, but that's more of wishful thinking.

14:32 <hkaiser_> so you need to define affinities for your threads, mainly

14:34 <hkaiser_> is that what you're looking for?

14:41 <zao> I think the affinity stuff is probably somewhat straightforward.

14:42 <hkaiser_> zao: you can use our counting semaphore to limit the amount of tasks scheduled, there is also some restricted executor doing similar things

14:46 <zao> Ooh, that's the kind of construct I had forgotten existed :D

14:51 <zao> This might be something to start prototyping on, thanks!

19:57 <zao> Bah, left my HPX 1.8.1 compile running overday on the VM and it failed out thanks to not having the fix for PR#6166 in the source tree. vcpkg doesn't package 1.9.0 yet for some reason.

20:42 <gonidelis[m]> https://wandbox.org/permlink/YYXD5Zgu8aEfic66

20:42 <gonidelis[m]> auto&& and decltype(auto). are they equivalent? What’s a case where they give a different deduction result

20:43 <gonidelis[m]> Also hkaiser who’s doing the vcpkg for our releases ? 1.9.0 is not there yet and we plan on releasing 1.9.1 already

21:11 <HHN93> Is there any reason tag_fallback_invoke is generally marked as a friend function?