hkaiser changed the topic of #ste||ar to: STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar.cct.lsu.edu | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | Buildbot: http://rostam.cct.lsu.edu/ | Log: http://irclog.cct.lsu.edu/
quaz0r has quit [Ping timeout: 255 seconds]
quaz0r has joined #ste||ar
hkaiser has quit [Quit: bye]
K-ballo has quit [Remote host closed the connection]
eschnett_ has quit [Quit: eschnett_]
<Yorlik>
Is there a hash functor for id_types somewhere i could use for an unoprdered map key?
<Yorlik>
I would like to use id_types as kleys in an unordered map.
<heller_>
they should support it out of the box
<heller_>
or do they?
<Yorlik>
I can't find a hash functor anywhere and have issues using them
<jbjnr__>
does anyone know what was changed that broke the guided pool executor stuff? I just had a quick look, but knowing what was changed might be useful
<jbjnr__>
seems to be dataflow related.
pmikolajczyk41 has quit [Ping timeout: 256 seconds]
<heller_>
jbjnr__: could be some missing includes
K-ballo has joined #ste||ar
<jbjnr__>
seems to be a tuple unwrapping issue. I expct KBallo fixed something and my workarounds are now causing a problem
<jbjnr__>
I know where to look
<K-ballo>
nope, I didn't fix anything
<K-ballo>
let me point you to the line that causes it
<jbjnr__>
It seems that my overloads were called with a,b,c, but now I get tuple(a,b,c) when dataflow passes things on to the executor internals
<jbjnr__>
I need to recompile with template backtrace limit unset
<jbjnr__>
probably one of my specializations just foesn't get picked any more and so the tuple stuff is not being handled as it used to
<K-ballo>
my suspicion is some substitution going wrong on an overload that is not intended to be called, but could not figure out which ones are intended
<K-ballo>
although to be fair, that is usually my first suspicion for everything that involves overloads behaving differently for different compilers
<jbjnr__>
K-ballo: do you know which merge triggered the new errors?
<K-ballo>
oh but this is not about arbitrary dataflow calls anyways, is it? just the internal ones related to executors?
<jbjnr__>
(the other test is called async_customization and has more stuff that might be appropriate)
<hkaiser>
jbjnr__: there have been changes to the default parallel executor, not sure how that affects yours
<hkaiser>
shouldn't have any effect, though
<jbjnr__>
K-ballo: yeah. It would only be a problem if dataflow was used with my executor and had some scalrs mixed with futures. I will try it once I fix this
Abhishek09 has joined #ste||ar
<Abhishek09>
hello
Abhishek09 has quit [Ping timeout: 256 seconds]
<jbjnr__>
hkaiser: Just out of curiosity ... "there have been changes to the default parallel executor, not sure how that affects yours" - aren't we supposed to not merge PRs that break things? (Or has policy changed)
<jbjnr__>
(and please note the heavy use of sarcasm - I'm just surpised that it wasn't considered)
Abhishek09 has joined #ste||ar
daissgr has joined #ste||ar
<hkaiser>
jbjnr__: I don't think this PR has broken anything - you were asking what has changed and I tried to be helpful in telling you
<jbjnr__>
(the guided pool stuff broke). I don't really care much, but am just surprised by the nonchalence
<hkaiser>
jbjnr__: I doubt the guided pool stuff broke because of this particular PR
<jbjnr__>
No it was the other one.
<hkaiser>
but if it did, I apologize, the tests I looked at were all green - so it couldn't have broken everywhere
<hkaiser>
ok, if it was the other one - why do you accuse me of nonchalance?
<jbjnr__>
Sorry. I didn't mean to accuse you of and wrongdoing.
aserio has joined #ste||ar
<jbjnr__>
I just thought things seem very casual round here at the moment
<jbjnr__>
^of any^
<jbjnr__>
not ^of and^
hkaiser has quit [Quit: bye]
<Abhishek09>
parsa:how does circleci works in phylanx?
eschnett_ has joined #ste||ar
Abhishek09 has quit [Ping timeout: 256 seconds]
Abhishek09 has joined #ste||ar
<Abhishek09>
hey parsa: are you here?
<zao>
Abhishek09: A lot of CI systems are controlled by files in the repositories for the software to test.
<zao>
How so?
<zao>
If you're curious, you might want to look up some quick guides describing how to get started with a CI environment and check that against what's configured in the repos.
<Abhishek09>
i think it is a server which run cibuild.
<zao>
Could you clarify what your question is actually about?
<zao>
I interpreted it as you asking how Phylanx is tested with CircleCI and other CI environments.
<zao>
Is the actual question about something else, like what CircleCI is at all?
<Abhishek09>
zao:you mean to say it runs phylanx for testing test.But I thought it can be used for building:
<zao>
Abhishek09: The purposes of CI systems are many.
<zao>
They can be used to test if software compiles. They can be used to run test suites. They can be used to prepare release packages.
<zao>
Anything that falls under "continuous integration".
<zao>
You can read the configuration files in the phylanx repository to see what that particular project uses CircleCI and Appveyor for.
<Abhishek09>
what about travis CI?
<zao>
What about it?
<Abhishek09>
phylanx doesn't uses.
<zao>
There are many different CI services. HPX used to use Travis before moving to CircleCI.
<Abhishek09>
is circleci is better than travis ci?
<zao>
Don't know.
<zao>
Travis recently fired a lot of senior staff, so they might be getting worse now :D
<K-ballo>
I think we switched out of travis because our builds take just too long and there's a time limit.. but that was years ago
bita_ has joined #ste||ar
Abhishek09 has quit [Ping timeout: 256 seconds]
bita_ has quit [Quit: Leaving]
<heller_>
would have been interesting what the actual question was
<zao>
So say we all ;)
<zao>
heller_: I tried at least :)
Abhishek09 has joined #ste||ar
aserio1 has joined #ste||ar
aserio1 has quit [Client Quit]
aserio has quit [Ping timeout: 268 seconds]
<diehlpk_work>
Abhishek09, diehlpk_works:I have found blaze on two places -> Use the one from bitbucket
<diehlpk_work>
As far as I know this is the main repo and all work goes there
<diehlpk_work>
Abhishek09, diehlpk_work:is it necessary to install hpx or i can use lib files -> I do not understand this question
daissgr has quit [Ping timeout: 258 seconds]
hkaiser has joined #ste||ar
daissgr has joined #ste||ar
<parsa_>
Abhishek09: what do you mean by how does phylanx circle work? it builds and tests phylanx. the commands it runs are in .circleci/config.yml and you can see the builds in https://circleci.com/gh/STEllAR-GROUP/phylanx
parsa_ is now known as parsa
aserio has joined #ste||ar
david_pfander has quit [Ping timeout: 258 seconds]
<Abhishek09>
deihlpk_works: i consider the package for fedora platform.
<Abhishek09>
Most of us are on this community using fedora.
<Abhishek09>
parsa:can i use cibuildwheel on circleci?
diehlpk_work has quit [Remote host closed the connection]
<parsa>
Abhishek09: what's the point? circleci doesn't support windows, we don't pay circleci to get the mac option, and we only use ubuntu to test on circleci...
<parsa>
don't worry about circleci... build one wheel for phylanx on any platform you can and demonstrate that it works
<parsa>
locally, on your own machine
<Abhishek09>
parsa:it currently works on Travis CI and CircleCI to build Linux and Mac wheels(PAID), and Appveyor to build Windows wheels.
<parsa>
Abhishek09: do you have a wheel file you have made for phylanx?
<Abhishek09>
i reuse that file config.yml to build the wheel
<parsa>
look, travis, circle ci, azure pipelines, gitlab, etc only run commands on their machines. you should be able to build on your own machine
<parsa>
circleci and the like are used for continuous integrations (i.e. making sure everything still works for every change). you don't need to worry about that now
<Abhishek09>
But it is easy to create .whl on fedora.
<Abhishek09>
i use cibuildwheel on circleci
<parsa>
then give us that wheel file and let us see what you've done
<Abhishek09>
No, i have created it.It is part of Gsoc.Now only have think how would it implemented and how it works
<zao>
*haven't ?
<Abhishek09>
we would create when coding begins.
<zao>
Feels like something you might want to test out beforehand maybe.
<zao>
I would, just to be more certain of my proposal if I did one.
<zao>
When are they due, by the way?
diehlpk_work has joined #ste||ar
<hkaiser>
heller_: yt?
<parsa>
Abhishek09: do you even have a proposal?
<Abhishek09>
zao:you also applying for Gsoc:you working on which task?
<parsa>
:)))))
<zao>
Abhishek09: No, I'm not a student and not part of the HPX/Phylanx GSoC.
<zao>
I hang around here giving good/bad advice and like trying to build HPX.
<zao>
(I'm a systems engineer at a HPC site)
<zao>
My job is literally installing silly scientific Python software for a living :D
<diehlpk_work>
Abhishek09, I think parsa is right, it would help to write down what you want to do
<Abhishek09>
parsa: Yes i roughly created proposal.I will upload when GSoC it start to accept So we people can review my proposal .
<diehlpk_work>
Abhishek09, Normally, you share the proposal with your mentors before uploading
<diehlpk_work>
We can see your proposal only after submission
<diehlpk_work>
At least in the last years, students shared a google doc with their mentors and they made remarks
<Abhishek09>
GSoc offer review option for mentor for proposals.So you people can suggest changes.
aserio has quit [Ping timeout: 264 seconds]
<Abhishek09>
deehlpk_works:Is sharing of proposal is safe,confidential.
<heller_>
As safe and confidential as you want it to be
<heller_>
The organizations haven't even been announced yet
daissgr1 has joined #ste||ar
<hkaiser>
just received an email that we're not part of GSoC this year
aserio has joined #ste||ar
<Abhishek09>
this will be announced on 27th
<Abhishek09>
How would get that email now?
<zao>
Oh dear.
<K-ballo>
our cool down year?
<Abhishek09>
diehlpk_work,parsa:is this msg is true?
<hkaiser>
Abhishek09: it is official
<zao>
More time to work on HPX then, yay!
<zao>
parsa: Heh, tried making my own little Python package with a C extension. Turns out that there doesn't seem to be any naming scheme for distro-specific wheels, just some nondescript "linux_x86_64" platform tag or the manylinuxes.
<zao>
I guess that nothing would stop you from having your own wheelhouse with some sort of tagging, I guess.
bibek has joined #ste||ar
<zao>
I think that Compute Canada builds all their software as wheels and put them on CVMFS for their sites.
<parsa>
at least say for sure at least in case of pandas, it downloads a prebuilt binary for debian and builds from source for alpine
<parsa>
***i can say at least in case of pandas
<zao>
Those seem to be manylinux1, sadly.
<zao>
I have not managed to find anything more distro-specific than manylinux1 and manylinux2010,.
<parsa>
i don't know, it could be that people have something elaborate going on their setup.py
<zao>
And it doesn't seem like pypi allows uploads of the naked linux_x86_64/i686 tags at all, only manylinux:en.
<zao>
I would reckon that alpine doesn't conform to manylinux:es at all, considering the lack of glibc.
<zao>
(Alpine runs musl libc, as you probably know)_
<parsa>
yeah, you can get glibc to work on it but its hacky
<zao>
Ah, it's even mentioned in PEP 513, probably explicitly declares itself to be incompatible.
<Abhishek09>
i not able to digest this sad news that ste||ar has not selected for Gsoc.
<parsa>
zao: i don't know if we can provide an hpx build independent of the linux distro. we're pickier than most packages on our dependencies. we probably have to always build everything we need then
eschnett_ has quit [Quit: eschnett_]
<zao>
Abhishek09: A bit unfortunate. I hope that you can find some other project that suits you.
<Abhishek09>
But i have almost done this project
<zao>
parsa: At least our deps are fairly simple to build privately.
eschnett_ has joined #ste||ar
<Abhishek09>
parsa,deihlpk_works:they have never confirmed me
<Abhishek09>
i want to listen from them.
<zao>
Abhishek09: You'll find out in public on February 26 12:00 UTC then as the list is published.
<K-ballo>
Abhishek09: if it's any consolation, if you truly were almost done with the project, we wouldn't have accepted you without at least increasing the scope of the project first
<zao>
In any way, the various attempts to make a manylinux2010 image seem to mostly aim for devtoolset-7, which is GCC 7.3.1.
<zao>
They're a bit stuck on that RH doesn't have any newer toolsets for i686, so manylinux2010 would have either no i686 flavour or 4.8.2 as advertised in the PEP.
<zao>
For future work, manylinux2010 or a newer equivalent would be excellent if the aim is to get packages onto pypi.
nikunj has joined #ste||ar
<zao>
I'm not sure if you could have custom platform tags if you ran your own wheelhouse and if they would be honored by installers. If you could, that'd be an alternative I guess if you had the resources to build on Popular Distros.
<zao>
All in all, situation hazy, hopefully I haven't confused it further with this research.
<parsa>
zao: no no no, it is helpful. thanks for investigating!
<diehlpk_work>
Abhishek09, Yes, this message is true
<diehlpk_work>
We got not accepted this year
Abhishek09 has quit [Quit: Page closed]
<zao>
Ah, ComputeCanada uses separate wheelhouses and generic names like -cp27-cp27mu-linux_x86_64.whl
nikunj has quit [Quit: Leaving]
hkaiser has quit [Quit: bye]
<heller_>
lol
<heller_>
what happened, why is migrate_component failing again :(?
<K-ballo>
isn't that the one that was failing intermitently?
<heller_>
yeah, and now it is failing consistently for the MPI parcelport
<heller_>
we fixed it for good one week ago
<heller_>
at least that was the idea ;)
<heller_>
and all of a sudden, after adding asserts, it seems to work again...
<heller_>
because I am running the TCP parcelport...
<K-ballo>
consistent is good
<K-ballo>
time sensitive is so bad
<heller_>
sure
<heller_>
just wondering that noone noticed
<heller_>
simbergm: we need a more versatile docker setup... cramming everything into one image isn't good
daissgr1 has quit [Read error: Connection reset by peer]
<Yorlik>
Should this compile? I thought ele would be invalid at the second push_back, but it compiled: buffers[0]->push_back(std_move(ele));
<Yorlik>
buffers[1]->push_back(std_move(ele));
<K-ballo>
the type of `ele` won't change after a push_back
<heller_>
just because it is invalid, doesn't mean it doesn't compile
<K-ballo>
there is no way whatsoever in which those two separate expressions are distinguishable at compile time
<K-ballo>
they either both compile or neither does
<heller_>
the code above looks perfectly fine (without further knowledge of the code)
<heller_>
there's nothing inherently wrong with it
<K-ballo>
that said, fluid typing is sweet!
<Yorlik>
I thought ele would be sortof invalid after being moved away.
<heller_>
BUT, use after move is seldomly something you want to do do (unless you assign a new value to it or somesuch)
<heller_>
sort of, probably, but in a usable state
<K-ballo>
ah, the destructive moves paradigm.. we don't have that in C++
<Yorlik>
Std_move to a vector should create a copy anyways, right? After all internally a vector is based on an aligned array.
<heller_>
depends on the object, really
<Yorlik>
Its a simple struct
<heller_>
a move, in C++, primarly is not about optimization, but about semantics
<Yorlik>
I thought it was about object ownership
<heller_>
sure, which is semantics
<Yorlik>
Since I gave ownership to the first vector, the second shouldn't have a right to receive the object as i see it. that's why I'm so puzzled
<K-ballo>
that assumes a destructive move model, which we don't have
<heller_>
well, the second line moves a different object
<K-ballo>
you transfered ownership of the resources within the object, not the object itself
<K-ballo>
the object itself remains in a valid (but often unspecified) state
<Yorlik>
So - in the case of a pd (simple struct) it just copies?
<heller_>
probably, yes
<Yorlik>
s/pd/pod/g
<Yorlik>
Could I create the object directly in the slot of the vector, to avoid copying the data?
<heller_>
yes, there is emplace_back
<heller_>
which constructs the object in the place it is supposed to end up in
<Yorlik>
Nice
<Yorlik>
I shall check that out
<heller_>
(could also call the move constructor)
<Yorlik>
Basically I have a vecore of elements, which are indexed in an addition unorderd_map by their associated id_type
<Yorlik>
The id type belongs to the mopther object, the entity, not the component which is stored in the vector
<heller_>
Yorlik: FWIW, most cases of undefined behavior in C++ do not require diagnostics from the compiler
<Yorlik>
I see: I'm in a minefield and nead to tread carefully ;)
<heller_>
mostly you'll be fine
<Yorlik>
BTW
<heller_>
remember though: premature optimization is the root of all evil :P
<Yorlik>
How would I migrate an object, that has custom dependencies which also need migration
<Yorlik>
After that we talk about premature yadda yaddas ;)
<Yorlik>
The structure I am building here is the tightest loop in the entire system I'm building - the heart of the entity component system design
<heller_>
Yorlik: I know this talk, and my point is still valid :P
<Yorlik>
This loop is where I really should optimize where possible
<Yorlik>
I agree for other situations
<K-ballo>
lol, that talk
<K-ballo>
that was fun
<Yorlik>
But these component vectors will have to deal with tens of thousands of updates per frame
<Yorlik>
I love that talk
<Yorlik>
My fav quote was " If you don't understand the data you don't understand the problem"
<Yorlik>
And: "Everyone who thinks premature optimization can leave the room now" (When talking about L2 cache misses)
<K-ballo>
yeah, the content was fine, but the speaker was completely ineffective
<Yorlik>
Let's say he was pretty excited about the matter.
<Yorlik>
The content though was really good, I think.
<Yorlik>
Back to the question before I kinda lost:
<Yorlik>
My entities are hpx components
<Yorlik>
they hold an int65 which i use as a bitfield to indicate which components the entity has
<Yorlik>
int 64 ofc :D
<Yorlik>
The entities live in vectors specialized for their type
<Yorlik>
And they are additionally indexed in an unordered mnap by id_type of the mother entity
<Yorlik>
If I want to migrate that entity later, the system has no way to know, that the components also need to migrate
<Yorlik>
How wouzld you approach that ?
<Yorlik>
A custom serializer?
<Yorlik>
I was thinking about an entity parcel package strucxture
<Yorlik>
it would hold the actual data and not the references. But thats ineffective ofc
<Yorlik>
Because ti implies copying
<Yorlik>
Essentially it's about migration of a group of related objects
<Yorlik>
And not all of them are components, some are just POD
<heller_>
you'll always need a custom serializer
<heller_>
not all copies are bad
<heller_>
K-ballo: FWIW, the migrate component test failure seems to be a problem with the MPI implementation, I can reproduce it with the same version, and it goes away with a newer one. The error itself is strange as well, without a clear origin (that is, the data received never gets sent anywhere)
<heller_>
not sure why it only triggers with that particular test though
hkaiser has joined #ste||ar
* heller_
hates MPI
<zao>
Yorlik: If you want a world where objects that have been moved are invalid, you want to program in Rust :D
<Yorlik>
We tried Rust and deliberately decided against it ;)
<zao>
Chickens! :)
<heller_>
zao: how long did it take you to get your first non trivial program compiled?
<zao>
It was way easier this time around compared to the first time I tried to pick it up a few years ago.
<heller_>
had quite long conversations with rust heads last week...
<zao>
I still trip myself up, of course.
<heller_>
talking about tokyo and rayon
<hkaiser>
heller_: short question
<heller_>
hkaiser: 42
<zao>
The state of tokio and futures-preview for async/await is a right mess.
<hkaiser>
that's an answer to a long question
<zao>
I'm going to let that solve itself :)
<heller_>
hkaiser: worth a shot...
<hkaiser>
heller_: can't find it :/
eschnett_ has quit [Quit: eschnett_]
<heller_>
hkaiser: the question?
<hkaiser>
the link
<heller_>
he
<heller_>
btw: I have a undefined behavior sanitizer clean build now!
<hkaiser>
heller_: saw that, yay!
<hkaiser>
merge it!
<heller_>
someone just broke migrate component
<heller_>
I need to fix that first ;)
<heller_>
and of course the guided pool mess that bubbled up
<hkaiser>
on the gitlab builder - there was a strange MPI error while executing the migration test
<heller_>
*nod*
<hkaiser>
did you see that?
<heller_>
yes
<hkaiser>
ok
<heller_>
goes away when updating the MPI version
<hkaiser>
ok, good
<heller_>
tried to hunt it down today
<heller_>
totally unclear where it comes from. The gist of the story: Someone receives a message header, but the actual size of the MPI message is zero
<heller_>
thus, the entire header is filled with a -1 pattern, leading to that failure
<heller_>
if I just ignore that case, I get a hang
<heller_>
the release build seems to be not affected by that glitch, the debugger however, clearly shows that no data was actually received
<zao>
Rather nice, I pulled your docker container and it also blows up on the `tests.unit.components.distributed.mpi.migrate_component` test for the commit that hkaiser linked there.
<zao>
Figured I'd try on my Ryzen build machine, see if it happens here too.
<zao>
A bummer I don't have any build automation yet.
aserio has quit [Quit: aserio]
<heller_>
zao: yes, the docker image has the problematic MPI version
<zao>
Very curious, should try with my EB tree later outside the container, got a ton of MPI versions there in different toolchains.
<zao>
But now, sleep :)
<zao>
Worth noting, we have had some OpenMPI/impi versions at work that have just not worked right with some codes. Can’t recall which now but we build with other toolchains every now and then.