hkaiser changed the topic of #ste||ar to: STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar.cct.lsu.edu | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | Buildbot: http://rostam.cct.lsu.edu/ | Log: http://irclog.cct.lsu.edu/
bita_ has joined #ste||ar
bita_ has quit [Client Quit]
Yorlik has quit [Read error: Connection reset by peer]
Yorlik has joined #ste||ar
bita has quit [Read error: Connection reset by peer]
K-ballo has quit [Quit: K-ballo]
nikunj97 has quit [Read error: Connection reset by peer]
nikunj1997 has joined #ste||ar
eschnett has joined #ste||ar
eschnett has quit [Quit: eschnett]
Yorlik has quit [Ping timeout: 245 seconds]
Yorlik has joined #ste||ar
hkaiser has quit [Quit: bye]
jaafar has joined #ste||ar
nikunj1997 has quit [Read error: Connection reset by peer]
nikunj has joined #ste||ar
<Yorlik>
heller_: yt ?
david_pfander has joined #ste||ar
<heller_>
Yorlik: 5 minutes
<Yorlik>
Super ! :)
<heller_>
Yorlik: appear.in?
<Yorlik>
Ya
<heller_>
I'm in
<Yorlik>
Thanks a ton - very helpful talk!
hkaiser has joined #ste||ar
K-ballo has joined #ste||ar
<Yorlik>
I think there is an unresovable conundrum: I tried to find a way to compose a variant out of different template specializations at compile time, but I think it's probably impossible. I'm composing it now manually in the place, where all the different template instantiations are defined. However: Any objections? Could it be done in a generic way?
eschnett has joined #ste||ar
<hkaiser>
Yorlik: sounds doable to me
<Yorlik>
I'd like to see that. But it's above my head for now.
<Yorlik>
You'd need some sort of collector function which I can't imagine how toi do (which doesn't mean a lot)
<hkaiser>
I would need more information to try helping
<K-ballo>
you can't stitch a literal `variant<T1, T2, T3...>` over specializations with `T1`, `T2`, `T2`, etc.. but you could always do some other form of type erasure
<Yorlik>
hkaiser: You know the problem with the different entities composed of different components. Cou already helped me with the inheritance order check and the bitmask composition. The riddle presented here would be to compose all these types together in a way to create variants from it. I am currently using 2 variants: One has pointers to entities and another a pointer to a vector of entities
<hkaiser>
Yorlik: how would pointers help at compile time?
<Yorlik>
with all entities or vectors of them being specializations from their templates
<hkaiser>
I think you're saying that you don't know what types to compose into a variant at compile time and then you complain that yu can't do it ;-)
<Yorlik>
I decided to use a master object which will hold the id_type and has a variant which points to the respective entity
<Yorlik>
I know the types - but I have to compose the variants manually
<hkaiser>
I'd do type erasure as K-ballo suggested
<Yorlik>
Lazy as I am I'd like the compiler to do it for me
<hkaiser>
define one or more functions that expose the same interface that perform the operations on your specialized Entity as needed
<hkaiser>
then store those functions (std::function<> or hpx::util::function<>) in your registry
<hkaiser>
then expose those operations from the registry which will look up the function and invoke it
<hkaiser>
you can still be lazy and let the compiler generate the functions to be bound etc.
<Yorlik>
For me the question is also a bit about if compile time specialization collection is doable at all. Sortof an academic question since I can code around the problem.
<Yorlik>
I also need to think about where to stop the templates and expose a uniform interface
<K-ballo>
it is not
<Yorlik>
K-ball: You mean not as in not possible?
* Yorlik
throws a missing o
<hkaiser>
Yorlik: you have 3 different things to worry about
<hkaiser>
a) the entity class - I think that is under control
<hkaiser>
b) a type representing a collection of Entities of the same type
<hkaiser>
I'd not use vector<> for that but some custom type that holds a vector, amongst other things
<K-ballo>
compile time is functional, you can't carry state, you can't "collect" types from specializations
<hkaiser>
and c) the registry which should be collecting the types from b)
<hkaiser>
doesn't look too bad
<K-ballo>
I take by "collect" you mean as if by putting in a collection
<Yorlik>
hkaiser: b is indeed a so called "EntityContainer" which is a template
<hkaiser>
yes
<Yorlik>
K-ballo: Collect as in putting them into a typelist to create a variant from
<K-ballo>
exactly, that type list would be the collection
<K-ballo>
10:30:21 AM - K-ballo: you can't stitch a literal `variant<T1, T2, T3...>` over specializations with `T1`, `T2`, `T2`, etc, but [...]
<Yorlik>
hkaiser: I think c is where I need to get rid of the types and that is where i am using a variant with pointers to the different EntityContainer types
<hkaiser>
your EntityContain should expose the functions that will be bound to type erased functions (e.g. std::function) and that will be stored in the registry
<hkaiser>
Yorlik: exactly
<hkaiser>
no need of a variant, just derive all EntityContainers from a common base class
<hkaiser>
gtg
hkaiser has quit [Quit: bye]
<Yorlik>
Such i can just get the variant and -> to the appropriate templated functions in the Container - which is more than just a container - it also creates and destroys
<Yorlik>
Abstact interface should do, indeed.
<Yorlik>
Actually the term abstraction makes sense exactly here.
<Yorlik>
I think for performance the trick is to find the right spot where to abstract dynamically and point to statically polymorphed objects
bibek has joined #ste||ar
aserio has joined #ste||ar
<Yorlik>
K-ballo: One reason why I am trying to templatize and typeify a lot of my code is, because I have the strong impression - after my initial toddler explorations into the lands of pink elephants - that this will exactly help me to shift possible later runtime errors to compile time and in the end have more correct code..
<Yorlik>
So - I'm hitting walls all the time now - but better now than later when the system has reached a much higher level of complexity than now. I'm really laying foundatiuons here.
<Yorlik>
Feels a bit like trying to build the groundwork for a house on top of a minefield.
<K-ballo>
C++ being the minefield? :P
<Yorlik>
No - the complexity of what we want to build
<Yorlik>
C++ is just learning
<Yorlik>
And ofc I am wrestling a lot with it in the moment.
<Yorlik>
But language issues aside - we are writing a complex distributed application with real time constraints - all the initial decisions have deep consequences later.
<Yorlik>
I strongly believe that an attitude of "We'll optimize it later" is not going to work in such a situation.
<K-ballo>
"later" you can optimize implementations, but hardly ever the design
<Yorlik>
The beginning phase, where we lay out the data structures and overall architecture is probably the most dangerous and vulnerable phase
<Yorlik>
Yes - agreed.
<Yorlik>
We are going slow now - to be faster later.
<Yorlik>
Which gives me time to learn C++ along the way :)
akheir has quit [Quit: Konversation terminated!]
akheir has joined #ste||ar
hkaiser has joined #ste||ar
<hkaiser>
Yorlik: so you either type erase using a abstract base class for your EntityContainer or by using type erased function that are bound to one of the EntityContainer instances
<Yorlik>
Yes
<Yorlik>
hkaiser: for me it's still about finding out where to draw the line between static polymorphs and dynamic calls. I absolutely want to avoid virtual calls sneaking into my tight loops.
<hkaiser>
sure
<hkaiser>
the registry seems to be a good place
<Yorlik>
Ya
aserio has quit [Ping timeout: 252 seconds]
aserio has joined #ste||ar
<aserio>
hkaiser: Are you headed down?
jbjnr has joined #ste||ar
Yorlik has quit [Read error: Connection reset by peer]
<jbjnr>
hkaiser: ( also K-ballo ) - I have the libfabric PP now bootstrapping with the sockets provider. It's not production ready, but PoC is working. This would allow us to completely remove the tcp parcelport if all bugs are fixed (working on that). If that were the case, we could potentially remove boost:asio - would it be easy to move the io pool and timer pool over to just simple stf::thread code, or do we rely heavily on boost:asio
<jbjnr>
for those?
bibek has quit [Quit: Konversation terminated!]
<hkaiser>
jbjnr: we'd need some kind of thread-pool + scheduler code to replace those
aserio has quit [Ping timeout: 250 seconds]
<jbjnr>
that's a pity
hkaiser has quit [Read error: Connection reset by peer]
hkaiser has joined #ste||ar
<hkaiser>
jbjnr: sounds like a GSoC project...
<zao>
jbjnr: So a forced external dependency on libfabric on all platforms?
<zao>
Support on Windows seems "fun".
<zao>
(I gather that you'd use some sort of "sockets" backend if you didn't have proper fabrics?)
<jbjnr>
zao: libfabric has 'providers' that are the backends. one is based on sockets. And yes, it would be a forced dependency, but it could be downloaded and built as part of our build process.
<jbjnr>
(similar to apex).
<K-ballo>
remove asio sounds cool, I would have never imagined it possible
<K-ballo>
does libfabric replace tcp, or just make it obsolete?
<zao>
FWIW, the readme on github says "Libfabric currently supports GNU/Linux, Free BSD, and OS X."
<K-ballo>
certainly, but I had gotten excited about dropping boost.asio
<zao>
Replace it with Rust and mio/tokio :D
<zao>
(please don't)
<K-ballo>
boost's system_error is a pain, and it is mainly due to asio (but also filesystem, and also hpx, and then hpx again)
<jbjnr>
I must be confused, because I thought windwows was supported. It was in earlier releases. perhaps they abandoned it. Maybe my memory is going soft.
<jbjnr>
ok. it is going soft, but in this instance ....
<jbjnr>
K-ballo: yes, the sockets provider would replace our tc parcelport. I had not really thought through the whole windows issue.
<jbjnr>
surely nobody uses windows any more do they? it's only android and linux in the real world!
<jbjnr>
and mac.
<jbjnr>
:(
<zao>
I guess it'd be optional for single-node builds, or are the parcelports/bootstrapping deeply entangled?
<K-ballo>
there is some HAVE_NETWORKING effort
<jbjnr>
yes, the have_networking=off does a very good job of disbling all the cruft we don't need. But my initial euphoria that I could remove boost:asio has been crapped on from a reasonable height.
<jbjnr>
if we got rid of asio, then I think removing the rest of boost would be almost doable.
aserio has joined #ste||ar
<K-ballo>
asio, filesystem, program options.. removing those means replacing them with similar facilities
<K-ballo>
we are supposed to no longer depend on boost.thread, not even header only bits
<K-ballo>
anyways, from 673 down to 465 total, right direction
eschnett has quit [Quit: eschnett]
eschnett has joined #ste||ar
<K-ballo>
oh no, I suspect the old measurement includes examples or tests or something like that
jbjnr has quit [Read error: Connection reset by peer]
jbjnr has joined #ste||ar
<jbjnr>
hkaiser: actually, we have two projects here that have their own simple std::thread pools with simple schedulers in. would only be a few day's work to drop them into the io/timer pools if the scheduling is not too complex.
<jbjnr>
zao: spirit is only used for the process binding parsing I think. (that could be replaced with something less sophisticated, nobody uses the command line binding really)
eschnett has quit [Quit: eschnett]
eschnett has joined #ste||ar
eschnett has quit [Quit: eschnett]
<jbjnr>
and performance counters. forgot about them
eschnett has joined #ste||ar
<heller_>
jbjnr: and slurm environment
aserio has quit [Ping timeout: 246 seconds]
aserio has joined #ste||ar
<simbergm>
what do the asio thread pools actually do that our usual thread pools/schedulers can't?
<jbjnr>
not much, but blocking things happening on them won't interfere with operation
hkaiser has quit [Quit: bye]
eschnett has quit [Quit: eschnett]
aserio has quit [Quit: aserio]
hkaiser has joined #ste||ar
jaafar_ has joined #ste||ar
jaafar has quit [Remote host closed the connection]