00:08 <hkaiser> gnikunj[m]: will do

00:08 <gonidelis[m]> gnikunj: what paper ?

00:09 <gnikunj[m]> I had to write one for SC (resiliency stuff) but didn't due to time restrictions :/

00:09 <gnikunj[m]> I'll write down that one now for Europar or IPDPS

00:10 <hkaiser> gnikunj[m]: done, might take a day, however

00:10 <gnikunj[m]> hkaiser: Thanks! I'll start reviewing the work I've done in the meantime.

09:14 <zao> I've got a piece of personal software for which HPX might be the least horrible choice for task infrastructure. I'm not sure if I dare try to use it :D

09:15 <zao> Currently trying to understand and leverage TBB, heh.

09:15 <zao> Biggest concern I have is how well I can interact with HPX tasks to/from native threads, as I've got several parts that need to interact with the real world.

09:17 <zao> I'm gonna have to make costly blocking queries into sqlite3 and leveldb, do long-running downloads with curl, and honoring RPC calls from clients over gRPC, Cap'n Proto or some other fabric like WebSockets, as well as a bit of a management UI with Win32 windowing.

09:17 <zao> Not sure if all this renders HPX pretty much unusable and I'm better off trying to remember how to write Rust again.

09:18 <zao> At least it's not distributed, so HPXlocal would do, if it had been on vcpkg.

11:32 <ms[m]> zao: if all you're looking to do is that kind of blocking io stuff then hpx is probably not the right choice

11:32 <ms[m]> but if you're looking for both that and the rest of the lightweight tasking then it's more interesting

11:33 <ms[m]> there are the service pools (with configuration numbers of threads) that you can use for those blocking calls and keep the main thread pool for regular hpx tasks

11:35 <ms[m]> looking if we have any examples on using them...

11:36 <zao> It's kind of a weird application, as the workloads are pretty much accepting jobs, downloading and ingesting data and JSON, making queries against the data. The biggest "computation" is probably decompressing zstd and processing NDJSON.

11:37 <zao> I'm leaning toward an async-like architecture and everything in the C++ space kind of sucks there. Right now I've got a bunch of synchronous threads that do request-response via TBB concurrent_queue:s.

11:38 <ms[m]> this is I think the only example/test we have for the service pools: https://github.com/STEllAR-GROUP/hpx/blob/1.7.1/libs/parallelism/executors/tests/unit/service_executors.cpp

11:38 <ms[m]> tbb is probably not a bad choice either, but I don't know how much it'll limit you in the future

11:40 <ms[m]> zao: asio?

11:41 <zao> Not using asio at the moment, but usually do for thread pools. Biggest problem there is that I can't really do much in the way of dependencies between posted tasks.

11:41 <zao> Downloads are curl via the multi interface running on a single thread, taking download jobs on a queue, pumping whatever it has, and notifying the results on a queue.

11:42 <zao> RPC infra is kind of blocking too, on its own threads.

11:44 <zao> It's a bit of a learning experience but I can't tell where to go sync and where to full pants-on-head async.

11:49 <ms[m]> zao: how serious is it if you make the wrong choice?

11:49 <zao> Just losing progress on a piece of infrastructure that I want out of the way and working for the rest of my hobby projects.

11:50 <zao> The application is a daemon that runs on an end-user computer, serving as an interface between my remote data store (over HTTP) and local client applications (over some sort of RPC), to stage and read datasets.

11:51 <zao> So a long-running process with a dynamic amount of clients that come and go and upwards of 200 GiB persistent state.

12:07 <zao> The things that normally tend to trip me up with async tooling is how to interact with it from regular threads, a lot of HPX as far as I remember kind of requires you to be a HPX thread to do things.

13:45 <deepak[m]> Is this [Script-language-binding](https://github.com/STEllAR-GROUP/hpx/wiki/Google-Summer-of-Code-%28GSoC%29-2021#script-language-bindings)

13:45 <deepak[m]> project already taken or in progress. If not, I would like to work on this, it would be great if anyone can give any insights about this.

13:45 <deepak[m]> And I was wondering can we use "PyBind11" instead of "Boost.Python", if possible.

13:46 <gonidelis[m]> deepak: apologies for not getting back to you

13:46 <gonidelis[m]> hkaiser: what do you say ^^?

13:46 <deepak[m]> gonidelis[m]: No problem :)

13:47 <hkaiser> deepak[m]: yah, pybind11 is better

13:47 <gonidelis[m]> deepak: afaik this project has not been implemented yet

13:48 <deepak[m]> hkaiser: thanks, I will report any further progress within this week

13:48 <hkaiser> cool~

13:48 <hkaiser> !

14:27 <gonidelis[m]> what's `f1` and `f2` here ? https://github.com/STEllAR-GROUP/hpx-local/blob/da9c37c2ba4b08ce03502d6d22eea754270756f6/libs/core/algorithms/include/hpx/parallel/util/partitioner.hpp#L169

14:31 <hkaiser> gonidelis[m]: look at how it's used

14:32 <hkaiser> gonidelis[m]: for instance: https://github.com/STEllAR-GROUP/hpx-local/blob/master/libs/core/algorithms/include/hpx/parallel/algorithms/minmax.hpp#L496-L525

14:33 <hkaiser> gonidelis[m]: f1 will be invoked for all partitions, f2 will be invoked afterwards to perform some reduction on the results of the invocation of f1

14:34 <gonidelis[m]> f1 is used for partitioning and f2 for reduction

14:34 <hkaiser> f1 is used for each partition, yes

14:35 <gonidelis[m]> ok

14:36 <gonidelis[m]> it is a generalized execution scheme that we apply in various end-user algos

14:36 <gonidelis[m]> mapreduce probably

14:36 <hkaiser> yes, some of the algorithms can be represented by such a scheme

14:37 <gonidelis[m]> nice

14:37 <hkaiser> as shown above, min/max/minmax are some of them

14:38 <gonidelis[m]> but we can also apply it when no reduction function is used

14:38 <gonidelis[m]> https://github.com/STEllAR-GROUP/hpx-local/blob/da9c37c2ba4b08ce03502d6d22eea754270756f6/libs/core/algorithms/include/hpx/parallel/util/partitioner.hpp#L278-L287

14:39 <hkaiser> looks like it

14:40 <gonidelis[m]> looks like u suspected we needed to support that case ;p

14:40 <gonidelis[m]> does it relate to the taskbench forkjoin runs?

14:41 <gonidelis[m]> though i don't see static_partitioner being used

14:41 <gonidelis[m]> from fork-join at all

14:42 <hkaiser> don't remember where it's used, see for yourself

15:49 <mdiers[m]> Hello again. I had asked a long time ago for an example of how I use the compression. In the meantime, however, quite a lot has changed in the structure that I can not find it again. Is there still an example of the use of compression?

15:50 <hkaiser> yep, here: https://github.com/STEllAR-GROUP/hpx/blob/master/libs/full/parcelset/tests/unit/put_parcels_with_compression.cpp

15:51 <hkaiser> the key lines are here: https://github.com/STEllAR-GROUP/hpx/blob/master/libs/full/parcelset/tests/unit/put_parcels_with_compression.cpp#L124-L130

16:41 <zao> gnikunj[m]: Singletons are not great due to several reasons. As they're globally accessible, they may be an invisible dependency in code, making both them and the calling code harder to reason about. Their lifetime is bothersome as well, as you have 0-1 instances, either creating one at program startup (init order problem) or lazily on first access (sensitive to call order). Destruction cannot happen until program end, which may be too late.

16:41 <zao> While cumbersome, it's often healthier to explicitly pass state into your objects as you construct them or have a more dynamic way of obtaining functionality you need via factory methods or other repositories.

16:50 <hkaiser> the biggest issue however is that singletons are prone to initialization sequencing issues

16:50 <hkaiser> if you're not careful, that is

16:56 <zao> In summary, while they're awfully convenient, they're very prone to subtle error or brittleness.