hkaiser changed the topic of #ste||ar to: The topic is 'STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar.cct.lsu.edu | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | Buildbot: http://rostam.cct.lsu.edu/ | Log: http://irclog.cct.lsu.edu/
primef has joined #ste||ar
primef has quit [Ping timeout: 246 seconds]
Yorlik has quit [Ping timeout: 268 seconds]
adi_ahj has joined #ste||ar
adi_ahj has left #ste||ar [#ste||ar]
hkaiser has quit [Quit: bye]
K-ballo has quit [Ping timeout: 240 seconds]
K-ballo1 has joined #ste||ar
K-ballo1 is now known as K-ballo
mdiers_ has quit [Ping timeout: 260 seconds]
mdiers_ has joined #ste||ar
mdiers_ has quit [Remote host closed the connection]
<Yorlik>
Not sure if this is a directed acyclic graph ;)
mdiers_ has quit [Remote host closed the connection]
mdiers_ has joined #ste||ar
primef has joined #ste||ar
<simbergm>
Yorlik: yep, beautiful, no? ;) it is acyclic, but unfortunately a bit too messy
<Yorlik>
I wish Doxygen / dot would accept parameters for the graph creation
<simbergm>
I'd like to show something like that in the docs, one option would be to group them into a few coarser groups
<simbergm>
they're easy enough to generate without doxygen as well
<simbergm>
directly with cmake
<Yorlik>
Is there a sort of architectural concept for HPX? Like components/packages - just something along the ideas John Lakos suggests?
<Yorlik>
I mean - somhow you obviously manage dependencies not to get out of hands.
<simbergm>
haven't watched the video yet
<simbergm>
but there's nothing formal yet, it would be something to put together for the next big (BIG) release
<simbergm>
I'm not sure we want to expose all the small modules we have right now (there's a lot of them), having a few bigger groups (components is way too overloaded) would be good to have
<Yorlik>
I think it would make sense to think of a concept or ruleset while modularizing HPX because that's probably a pain anyways.
<Yorlik>
Lakos' talk essentially is about ideas mentioned in his 1996 book about large scale C++ development. I think HPX absolutely is in the ballpark for this kind of thinking
<heller2>
well, we usually use minor versions for bugfix releases
<Yorlik>
Erm ... hwat soes that mean?
<heller2>
we already fail at point 1
<Yorlik>
Like a minor change might break the API? Or a bugfix might introduce a new feature?
<heller2>
"Software using Semantic Versioning MUST declare a public API. This API could be declared in the code itself or exist strictly in documentation. However it is done, it SHOULD be precise and comprehensive."
<Yorlik>
You mean you don't manage to break the API with major releases ;)
<heller2>
no, we try to include include bugfixes in minor releases
<heller2>
and no new features
<Yorlik>
What would the third number (fixes) then mean in the context of HPX?
<Yorlik>
Seems you are aggregating 2nd and 3rd number then
<heller2>
yes, we don't have patch release
<heller2>
and break API between minor releases
<heller2>
so yeah, we don't follow semver at all
<Yorlik>
I mean - as long as the API is stable it's not that much of a problem, ofc.
<Yorlik>
But as a user I'd like to use versions that get more and more stable.
<Yorlik>
If patches and new features are mixed I can never know which version to use in production.
<heller2>
sure
<Yorlik>
In other software I sa solutions, where they flipped the minor versions like unequal is a fix and equal a new feature.
<Yorlik>
And major was API breaking stuff
<heller2>
we try to never break API between version 1.x and version 1.x+1, but we deprecate APIs, which means you'll get breakage when updating from version 1.x to 1.x+3, for example. of course, this process isn't perfect yet
<Yorlik>
Is semver so problematic to maintain? I mean - you'd have to have bugfix branches,
<Yorlik>
Maybe you could have something like applying fixes to lets say the last 3 minor releases
<Yorlik>
Such that if you fix something in 1.5, the fix would be applied to 1.4 and 1.3 too
<Yorlik>
Not sure how much additional work that would be
<heller2>
that creates overhead, indeed
<Yorlik>
I wonder if it could be automated
<Yorlik>
Like if a fix is accepted the system tried to apply and test it
<Yorlik>
Which would possibly imply to write a test for the fix
<heller2>
what if the application of the patch fails?
<Yorlik>
Then the patch version number would not be applied
<Yorlik>
Lets say you have branches for 1.5, 1.4 and 1.3 which are the "maintenance pool" - they receive new fixes
<Yorlik>
So a patch would maybe create 1.5.1, 1.4.1 and 1.3.12 (like 1.3 already having a bunch of fixes)
<heller2>
we currently don't have the manpower to support such a scheme
<Yorlik>
If it fails, the new patch number is not assigned, but the commit stays at head
<Yorlik>
Thats why i wondered about automation.
<heller2>
well, if we maintain version 1.2 and 1.3
<heller2>
and there's a bug in 1.2, which is already fixed in 1.3, for example because there was a new feature
<heller2>
you can't automatically backport that to 1.2, so manual intervention is required
<Yorlik>
The idea is not to buy into the risk of new bugs when an old bug gets fixed, because if new features coming along with it.
<Yorlik>
IC
<heller2>
if you have a bug in 1.3, how do you determine if that bug is also in 1.2? Was it caused by the new feature? Does 1.2 require a totally different fix?
<Yorlik>
A bug usually get rported for a specific version I'd guess, so you'd apply the fix not before that version reported.
<Yorlik>
So - if I report a bug for 1.4.0 it would not autmatically be applied to 1.3.9
hkaiser has joined #ste||ar
<heller2>
Well, we officially only support the latest version
<Yorlik>
Latest patch version? Minor Version?
<Yorlik>
As long as this is the case stability will always be an issue
<heller2>
The version that came out the latest :p
<heller2>
Sure
<heller2>
As said, there's not enough manpower
<Yorlik>
... which autmatically disables HPX for a bunch of production uses (like outside research and bleeding edge scenarios).
<heller2>
Would you apply the same reasoning to Boost?
<Yorlik>
Even when not supporting backwards maintenance because of lack of manpower - using semver correctly would atr least make it explicit
<Yorlik>
I don't know how Boost is doing it.
<heller2>
In the very same way
<Yorlik>
Doh
<Yorlik>
At least every bugfix should come along with some sort of test :)
<heller2>
We try
<Yorlik>
If I were Jeff Bezos I'd sponsor you - unfortunately I'm not .... :(
<heller2>
Do you know of any open source project that strictly follows semver?
<Yorlik>
Just making it explicit in the version, without supporting it would already help.
<Yorlik>
Anyone providing a fix for a needed version could make a pr to add a patrch
<simbergm>
exactly, since we already do a kind of pseudo-semver we should just do it properly
<Yorlik>
You'd just open the door to make it possible.
<Yorlik>
You'd just have to clearly separate adding features from fixes in your commits
<hkaiser>
simbergm: doesn't semver imply to maintain more than the latest version?
<hkaiser>
primef: are you a windows person? or do you use linux?
<primef>
hkaiser: I use linux. Though I have a windows machine but don't develop on it usually.
<Yorlik>
hkaiser: You'd just make sure that bugfixes only raise the patch number(third) and new features add to the minor number.
<hkaiser>
Yorlik: don't we do that already?
<hkaiser>
primef: ok
<primef>
hkaiser: Why do you ask?
<Yorlik>
hkaiser: Not sure - I was asking about semver and heller2 meant not really. But he was referring to the public API point in the paper. I guess there is a subset of semver. Just wondering if you have it defined at all.
<Yorlik>
My main concern was, that raising release numbers do not guarantee more stability
<Yorlik>
If patch numbers reflect strictly JUST fixes, at least you could assume 1.4.0 to 1.4.1 does probably not introduce new bugs
<Yorlik>
But 1.4 to 1.5 would add features and thus put you at risk of new bugs.
<Yorlik>
If you or someone would need to stick with a minor release for whatever reason, a fix could be backported, e.g. from 1.5.1 to 1.4.1
<Yorlik>
In that case 1.4.1 would be released after 1.5
<simbergm>
hkaiser: no, it'd be pretty much what we have now, just our minor version moved up to major
<Yorlik>
Major numbers should reflect API breaking, minor new features or marking API as deprecated.
<zao>
semver with HPX? Heh.
<Yorlik>
For me as a user it would mean, I am using 1.4.0, seeing 1.4.1 and would know using 1.4.1 is a no.brainer.
<primef>
hkaiser: Sorry, have to leave. You can write and I will read your reply on the logs.
nikunj has joined #ste||ar
primef has quit [Ping timeout: 268 seconds]
nikunj97 has joined #ste||ar
<hkaiser>
Yorlik: so it _is_ about maintaining more than one version after all
<hkaiser>
Yorlik: sure, the idea of a 1.4.1 would be to fix problems with 1.4.0
<Yorlik>
That would be ideal from a user perspective. But having the versioning precise (not sure if this is the case or not with HPX) would at least make the situation explicit.
<hkaiser>
Yorlik: also, raising (minor) version numbers never increases stability
<Yorlik>
So - even if you do NOT maintain multiple versions, consequent numbering would at least clearly communicate what's going on.
<Yorlik>
That should be doable, imo. But maybe you do it already anyways.
<hkaiser>
Yorlik: well, as I said - I think that's what we're trying to do
<hkaiser>
perhaps not in a very formalized way, however
<Yorlik>
Do you have something like written release and versioning guidelines people can read?
<hkaiser>
primef: I thought that if you were a windows person you could look into implementing the IO performance counters we currently have for linux only
nikunj has quit [Ping timeout: 268 seconds]
nikunj97 has quit [Ping timeout: 260 seconds]
hkaiser has quit [Ping timeout: 265 seconds]
<Yorlik>
If I create a static thread local object pool singleton and I want to guarantee that object is never used outside the thread by another user - is there a way to guarantee a hard error if someone tries to use it outside the thread where is was created ? Would I have to check against the initial thread ID or how would that be done best?
hkaiser has joined #ste||ar
primef has joined #ste||ar
<primef>
hkaiser: I already saw that issue and discarded it, because I'm not usually working on my Windows machine. Sorry for that!
<hkaiser>
primef: no worries
weilewei has joined #ste||ar
<hkaiser>
primef: would you rather implement new things or work on existing code (modifying it)?
<primef>
But as I'm here and we are talking about it. I talked with hkaiser yesterday about issues I could helping with. If someone has some suggestions for issues (HPX beginner issues) I can work on in my spare time, please let me know!
<K-ballo>
hi primef, new student?
<primef>
hkaiser: I'm open to both, my only thought is that as I'm a newbie in HPX it could be harder to me to implement new stuff. But if you have another opinion let me know. Basically I'm open to anything.
<hkaiser>
primef: existing code might be better
<primef>
K-ballo: Hi, quite :-) at least for what concerns HPX
<hkaiser>
primef: how about #3646
<K-ballo>
welcome!
<primef>
hkaiser: Give me a sec. I'm going to check it out!
<primef>
K-ballo: thanks!
<hkaiser>
or #3829 which is to address one or two algorithms from #3646, but is somehow broken
<hkaiser>
weilewei: please try again, I think I have addressed the include path issue
<weilewei>
hkaiser sec
<primef>
hkaiser: #3646 sounds interesting! Something I would like to work on. Can I just pick a couple of algs to work on? Where is the prio?
hkaiser has quit [Quit: bye]
<diehlpk_work>
heller2, When was the first time we joined GSoC?
RostamLog has joined #ste||ar
weilewei has joined #ste||ar
hkaiser has joined #ste||ar
diehlpk_work has joined #ste||ar
weilewei has quit [Remote host closed the connection]
weilewei has joined #ste||ar
<weilewei>
hkaiser so command `make tests` pass no problem, when I do `ctest -V`, most of tests failed due to hpxrun.py not exists in build/bin/ folder
<hkaiser>
ahh, it's trying to use the script from the build directory, should use the one that was installed
<weilewei>
do you pass any special flag to second build?
<diehlpk_work>
hkaiser, We have access to the xpress allocation again :)
<weilewei>
oh ok
<hkaiser>
diehlpk_work: nice!
<diehlpk_work>
jbjnr, Do you need access to Cori as well?
<diehlpk_work>
hkaiser, ?
primef has quit [Ping timeout: 252 seconds]
<weilewei>
hkaiser no meeting w/ Ronnie today, he texted he will reschedule it
<hkaiser>
weilewei: ok
<hkaiser>
weilewei: pls try again
<weilewei>
hkaiser ok
<heller2>
hkaiser: I don't like this complication of our buildsystem for such a corner case. Did you investigate to just create a seperate CMakeLists.txt and leave our main CMakeLists.txt alone?
<hkaiser>
heller2: I simply copied our existing CMakeLists.txt and include it
<hkaiser>
heller2: but the same can be done with a separate one
<heller2>
right, that's what I thought
<hkaiser>
cmake doesn't allow to specify the name of the file to use...
<heller2>
no, but a directory
<hkaiser>
hmmm
<hkaiser>
then the PROJECT_DIR is different and would require to touch on all our helper scripts that rely on it
<heller2>
It might be worth opening a seperate github repository, call it HPX integrations, or so, where we can collect things like that... might be nice to run before releasing, or so
<heller2>
that would be unfortunate
<hkaiser>
that's why I went for the current scheme
<heller2>
fair enough
<heller2>
does return() work from the main CMakeLists.txt?
<hkaiser>
shrug
<diehlpk_work>
heller2, hkaiser, jbnr Please have a look at the GSoC application. simbergm and I completed the application, but you might want to edit things.
<hkaiser>
ok
<heller2>
ok
<heller2>
hkaiser: might make the whole thing cleaner and the diff shorter ;)
<hkaiser>
heller2: not sure I understand
<hkaiser>
you mean include just the new stuff and leave everything else mostly in place?
<diehlpk_work>
Only missing thing for GSoC is the cleaned list of projects
<heller2>
if (HPX_EXTERNAL_TESTING) include(hpx_external_test) return() endif() #remainder of what we had before
<heller2>
hkaiser: right
<hkaiser>
nod
<hkaiser>
I can try that
<heller2>
thanks!
<heller2>
diehlpk_work: when is the deadline?
<hkaiser>
heller2: yep, that works
<diehlpk_work>
heller2, Feb 4th
<weilewei>
hkaiser the /bin folder is not generated for the second time build, so I got the same error not founding the hpxrun.py
<diehlpk_work>
hkaiser, Build failed with some linking error
<diehlpk_work>
undefined reference to `bool hpx::detail::apply_impl
<hkaiser>
diehlpk: SAME ERROR AS BEFORE - IT BUILT FOR ME HOWEVER :/
<hkaiser>
sorry for the caps
<diehlpk_work>
Ok, so you used the Fedora Docker image?
<hkaiser>
yes
<hkaiser>
rebuilt the image from scratch
<diehlpk_work>
Ok, I will try ot again tomorrow. They might have an older image on the build system
K-ballo has quit [Quit: K-ballo]
K-ballo has joined #ste||ar
primef has joined #ste||ar
diehlpk has quit [Ping timeout: 248 seconds]
primef has quit [Ping timeout: 248 seconds]
<weilewei>
hkaiser I got this error after re-install and re-build: /usr/bin/python3.7: can't open file '/usr/local/bin/hpxrun.py': [Errno 2] No such file or directory
<hkaiser>
weilewei: ahh ok
<hkaiser>
could you try the second build with -DCMAKE_INSTALL_PREFIX=... set to where HPX was installed?
<hkaiser>
as a workaround, I know what to change
<weilewei>
I open my CMakeCache.txt for the second build, CMAKE_INSTALL_PREFIX:PATH=/usr/local
<hkaiser>
yes
<hkaiser>
that's the default
<hkaiser>
I need to fix this
<weilewei>
I tried this before, passing CMAKE_INSTALL_PREFIX to where hpx installed and got another error, however, I did not copy the log
<hkaiser>
weilewei: I think I just fixed this
<weilewei>
let me try again and let you know later
<hkaiser>
pls pull again
<weilewei>
hkaiser ok
weilewei has quit [Remote host closed the connection]