hkaiser changed the topic of #ste||ar to: STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar-group.org | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | This channel is logged: irclog.cct.lsu.edu
<john98zakaria[m]>
I installed sphinx and everything
hkaiser has joined #ste||ar
<john98zakaria[m]>
Never mind, ubuntu\s old packages had taken precedence over the newer python ones
<john98zakaria[m]>
So i removed them all and now building
K-ballo has joined #ste||ar
Yorlik_ is now known as Yorlik
<Yorlik>
hkaiser: YT?
<hkaiser>
here
<Yorlik>
Is HPX serialization safe enough to deserialize messages from untrusted sources?
<Yorlik>
I am thinking of provoked buffer overflows and other shenanigans.
<hkaiser>
Yorlik: serialization doesn't handle buffers, you provide those
<hkaiser>
if you have specific test cases, I'd be happy to have a look, though
<Yorlik>
hkaiser: Do you consider it safe for receiving messages from a client and deserializing them?
<hkaiser>
well, if you're concerned about this kind of thing you might want to add a checksum or similar to your buffers to verify the buffer's content integrity
<hkaiser>
serialization assumes that the internal structure of the data corresponds to the expected
<hkaiser>
we don't do anything special in that regard
<Yorlik>
The networking we do is encrypted. But I am thinking about clients manipulating packets for malicious reasons like crashing a server.
<Yorlik>
Or worse, like arbitrary codeexecution.
<hkaiser>
if you add a certificate or similar they won't be able to do that
<hkaiser>
Yorlik: gtg, sorry - I'll dig out some old stuff we've had in place a long time ago for this
<Yorlik>
NP
<Yorlik>
See you !
hkaiser has quit [Quit: Bye!]
diehlpk has joined #ste||ar
diehlpk has quit [Quit: Leaving.]
diehlpk has joined #ste||ar
diehlpk has quit [Quit: Leaving.]
diehlpk has joined #ste||ar
diehlpk has quit [Ping timeout: 250 seconds]
diehlpk has joined #ste||ar
<john98zakaria[m]>
What is this trying to tell me ?
<john98zakaria[m]>
<K-ballo> "struct fun { void operator()(..." <- You saved me from going insane
<john98zakaria[m]>
I also tried c++ 20 with lambdas worked perfectly
<hkaiser>
K-ballo: the constraints are that the function object needs to be serializable
<john98zakaria[m]>
hkaiser: Should I add this information to the docs ?
<hkaiser>
sure, if you like?
<hkaiser>
it's not specific to all_reduce (or even the collectives), though
<john98zakaria[m]>
hkaiser: I will save future generations from the pain I suffered xD
<hkaiser>
it's specific to sending function objects over the wire, so in this sense it's implied as otherwise youldn't be able to send it
<john98zakaria[m]>
hkaiser: Is there another way to make a function serializable ?
<john98zakaria[m]>
Other than the empty function struct?
<K-ballo>
yes, a peroper way
<K-ballo>
*proper
<K-ballo>
empty works because you get a default serialization that serializes nothing
<hkaiser>
john98zakaria[m]: things are serializable if a) they are default constructible and are either empty types, have an serialization function, or all members are accessible publicly and they can be used in structured binding contexts
<hkaiser>
or b) if they are not default constructible, they need to have special serialization support
<K-ballo>
hkaiser: there should be a copy-constructible constrain as well, to allow the implementation to operate on copies
<hkaiser>
true
<hkaiser>
move-only types work as well
<john98zakaria[m]>
Is this possible to be applied on a free function?
<hkaiser>
but in this case the user needs to move the argument into async (or similar)
<hkaiser>
john98zakaria[m]: no, serializing pointers doesn't make sense
<hkaiser>
the same entity is not guaranteed to have the same address on two localties
<K-ballo>
move-only is weird, you still end up operating on multiple independent copies (manifested via serialization)
<hkaiser>
K-ballo: well, you 'move' from one locality to the other one (conceptually)
<K-ballo>
multiple ones, potentially
<hkaiser>
K-ballo: not really, de-serialization is always done on one locality only
<hkaiser>
we don't do broad-casting of messages
<K-ballo>
uuuhm, something sounds off
<K-ballo>
only one locality does the reduction, but then even when that locality is the current locality serialization is still required?
<hkaiser>
K-ballo: wel, sure - I was talking about serialization in general
<hkaiser>
for collective operations additional constraints may apply - I agree
<K-ballo>
so all_reduce does operate on multiple copies of the given function object?
<K-ballo>
I'm trying to determine what I should expect if my serializable function object depends on internal mutable state
<hkaiser>
the reduction is done in one place (currently)
<hkaiser>
but if we build a tree of localties, the reduction may end up being done piece-wise
<hkaiser>
K-ballo: I don't think you should rely on that - I'd have to check the code
diehlpk has quit [Quit: Leaving.]
diehlpk has joined #ste||ar
<hkaiser>
K-ballo: btw, do you know if the networking TS say anything about how it inter-operates with the C++ memory model?
<hkaiser>
i.e. does it say anything about this kind of problems as moving objects or what copying means in the context of networking?
diehlpk has quit [Ping timeout: 244 seconds]
<hkaiser>
we have been treating our networking as some kind of 'magic' that makes objects disappear into nowhere and materialize on the other end
<K-ballo>
not of the top of my head
<K-ballo>
it's probably just all raw bytes
<hkaiser>
right
<hkaiser>
or integral types
<hkaiser>
bitwise-copyable things
<Yorlik>
To std::span or not to std::span raw memory, that is the question ...
<hkaiser>
Yorlik: it's always a good idea to use span instead of a ptr and a length
<Yorlik>
I'm thinbking about making polymorphic objects from raw memory blocks to give their data to Lua scripts.
<Yorlik>
But have them all in an array / vector
<Yorlik>
Could give us some runtime freedom for scripters.
<Yorlik>
I'd just pretend the block to be an array of T, U, Vs and access spans [n] and keeping track of which positions I can use.
<Yorlik>
Like a dynamic struct exposed to Lua scripts
<Yorlik>
So objects could have a dynamic data part local to all their data.
<Yorlik>
So the "raw memory" block would be some byte[sizeofblock] block; inside their struct.