aserio changed the topic of #ste||ar to: STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar.cct.lsu.edu | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | Buildbot: http://rostam.cct.lsu.edu/ | Log: http://irclog.cct.lsu.edu/
eschnett has quit [Quit: eschnett]
ABresting[m] has joined #ste||ar
EverYoun_ has joined #ste||ar
EverYoung has quit [Ping timeout: 260 seconds]
hkaiser has joined #ste||ar
ABresting[m] has quit [Ping timeout: 272 seconds]
<github>
[hpx] hkaiser pushed 3 new commits to fixing_msvc_win32: https://git.io/vHex4
<github>
hpx/fixing_msvc_win32 afcd1fa Hartmut Kaiser: Changing some uses of #ifdef HPX_MSVC to #ifdef WIN32
<github>
hpx/fixing_msvc_win32 1db987d Agustin K-ballo Berge: Add and use HPX_MSVC_WARNING_PRAGMA for #pragma warning
<hkaiser>
careful, certain three-letter-acronyms invite unwanted lurkers
<Smasher>
:D
<github>
[hpx] K-ballo deleted algorithm-overloads-msvc12-fix at bbb29a3: https://git.io/vHepE
<K-ballo>
that stacktrace :'(
<github>
[hpx] hkaiser pushed 1 new commit to fixing_2439_2: https://git.io/vHepS
<github>
hpx/fixing_2439_2 e39cd05 Hartmut Kaiser: Merge branch 'master' into fixing_2439_2
vamatya_ has quit [Ping timeout: 255 seconds]
EverYoun_ has quit [Ping timeout: 260 seconds]
K-ballo has quit [Quit: K-ballo]
hkaiser has quit [Quit: bye]
mcopik has joined #ste||ar
ABresting[m] has joined #ste||ar
mcopik has quit [Ping timeout: 260 seconds]
mcopik has joined #ste||ar
ABresting[m] has quit [Ping timeout: 255 seconds]
mcopik has quit [Ping timeout: 246 seconds]
mcopik has joined #ste||ar
ajaivgeorge has joined #ste||ar
ajaivgeorge has quit [Client Quit]
<heller__>
Smasher: come in
<heller__>
Come on...
<heller__>
Try both localities with the same build type...
<heller__>
Anything else is doomed to fail. No wonder you are getting that error....
vamatya_ has joined #ste||ar
mcopik has quit [Ping timeout: 268 seconds]
jaafar has joined #ste||ar
EverYoung has joined #ste||ar
pree has joined #ste||ar
EverYoung has quit [Remote host closed the connection]
vamatya_ has quit [Ping timeout: 268 seconds]
Remko has joined #ste||ar
Remko has quit [Remote host closed the connection]
Remko has joined #ste||ar
jaafar has quit [Ping timeout: 240 seconds]
shoshijak has joined #ste||ar
david_pfander has joined #ste||ar
pree has quit []
bikineev has joined #ste||ar
bikineev has quit [Ping timeout: 246 seconds]
Remko has quit [Remote host closed the connection]
thundergroudon has joined #ste||ar
thunderGroudon|2 has joined #ste||ar
thundergroudon has quit [Read error: Connection reset by peer]
<Smasher>
ive noticed if i write to a stream from for_loop_n it is synchronized
<Smasher>
magic?
<github>
[hpx] taeguk opened pull request #2639: Fix a tiny typo in tutorial. (master...taeguk-patch-typo-2) https://git.io/vHvRK
<taeguk>
I'm not a narrow-minded person :)
<jbjnr>
synchronized in what sense?
<jbjnr>
Smasher: ^
bikineev has joined #ste||ar
thundergroudon has joined #ste||ar
<jbjnr>
heller__: yt?
<heller__>
jbjnr: hey
<jbjnr>
good morning - Raffaele has a new problem ...
<heller__>
tell me
thunderGroudon|2 has quit [Ping timeout: 246 seconds]
<Smasher>
jbjnr well if you have a concurrency and write to the same stream from different threads its usually messed up
<jbjnr>
he is running out of memory, and it appears that when he creates many many tasks, the stack allocation code in hpx, is creating stacks for tasks, when those tasks complete, the stacks get placed in a 'used stacks' list, but instead of them being recycled, mostly new stacks are created when tasks become ready to run - instead of recycling ones that are done with. Consequently, stack usage...
<jbjnr>
...consumes all the memory and he gets memunmap errors etc etc
<jbjnr>
some are recycled, but new ones keep getting created even when old stacks area vailable
<jbjnr>
are there any knobs and settings I can twiddle to change the stack use behaviour
thundergroudon has quit [Read error: Connection reset by peer]
<heller__>
jbjnr: ugh. that sounds like a severe bug
<heller__>
the old ones should really get recycled
<heller__>
he is using a lot of mixed high priority and regular priority tasks, right? what about different stack sizes?
<jbjnr>
who knows most about thsat code - you or hk?
<github>
[hpx] taeguk closed pull request #2639: Fix a tiny typo in tutorial. (master...taeguk-patch-typo-2) https://git.io/vHvRK
<heller__>
does he turn of guard pages?
<jbjnr>
yes, I think so usually
<heller__>
both :P
<heller__>
i have to dig a little though
<heller__>
a small test program reproducing the issue would help though
<jbjnr>
I'm wondering if completed tasks are going onto the 'completed' queue, but have not been 'deleted' yet, so the stacks are not considered re-usable ...
<heller__>
that sounds about right, yes
<heller__>
let me check...
<jbjnr>
so is there a setting I can tweak that incresases the fequency of clean of dead tasks
<heller__>
nothing exposed, you'd have to change the code
<heller__>
I'll have a quick look
<jbjnr>
no prob
<heller__>
what's the ration of high vs. low priority threads at the moment?
<heller__>
jbjnr: there is a define, HPX_MAX_TERMINATED_THREADS which controls the frequency
<heller__>
which is of course not exposed :P
<heller__>
it is set to 1000 right now
<jbjnr>
I see it. set to 1000 bt default. I'll lower it and see what gives ...
<jbjnr>
thanks
Matombo has joined #ste||ar
<heller__>
jbjnr: don't hesitate changing it to a runtime controllable variable
<jbjnr>
ooh. this is per worker thread too I think
<heller__>
yes, each time a thread terminates, we put it on the terminated list
<jbjnr>
so on daint, we could have 32k dead stacks before cleanup is guaranteed
<heller__>
once the count exceeds the maximum, we clean them up
<heller__>
indeed
<heller__>
well, take that times two
<heller__>
since you are using high priority tasks extensively
<jbjnr>
ouch!
<jbjnr>
related question : if a task suspends, and then resumes on a differnt worker thread - the stack will move from one to the other - are the stacks originally pinned (first touch) to the worker they were first created on
<jbjnr>
(or are they memcopied across from one to another)
<jbjnr>
heller__: indeed. I saw some article a while back that intel/ibm/someone was working on a user level api to allow better memory pinning from use code. I guess the hwloc people are in the best place to work on it
<heller__>
yeah
<heller__>
intel developed memkind
<heller__>
horrible code, mostly to support legacy applications without modification to take advantage of hbm
<jbjnr>
do we have any idea when hwloc2 will be uable?
<jbjnr>
^usable
<jbjnr>
I'm subscribed to the hwloc ml, but I only ever read it when I have problems (which is never now)
bikineev has quit [Read error: No route to host]
bikineev has joined #ste||ar
<heller__>
jbjnr: no idea
bikineev has quit [Ping timeout: 245 seconds]
shoshijak has quit [Quit: Ex-Chat]
shoshijak has joined #ste||ar
pree has quit [Ping timeout: 260 seconds]
shoshijak has quit [Quit: Ex-Chat]
shoshijak has joined #ste||ar
<Smasher>
my own command line arguent which i define with boost::program_options::options_description often collide with hpx's ones
<Smasher>
that's pretty annoying
<shoshijak>
Smasher : how so?
<Smasher>
desc_commandline.add_options()("probedir,pd", value<std::string>()->default_value("probe"), "The directory containing the probe fingerprints. Default: \"probe\"");
<Smasher>
now i run it with my_exe -pd=myfolder
<Smasher>
and it throws an exception
<shoshijak>
shouldn't it be --pd=myfolder (with 2 '-'s)
<shoshijak>
ah no sorry, forget what I said
<Smasher>
my_exe -h says it must be one '-'
<shoshijak>
what does the exception say?
<Smasher>
cant figure it out yet
<Smasher>
without debug the application just crashes
<Smasher>
and with debug "what" is an empty string
<Smasher>
msg = "Cannot open file: robedir=probe_different"
<Smasher>
it thinks -p is --hpx:app-config
<jbjnr>
Smasher: I think the program options allows --first or -second, but the second one is a single char, "pd" might not be allowed as the second version (the short version)
<Smasher>
ah wait, i have another argument this time
<jbjnr>
but I agree that we should be able to disable or override the default hpx ones as they use most of the alphabet already and user code might need -t or op or os etc etc
thundergroudon has joined #ste||ar
<Smasher>
--probedir=.. works
<jbjnr>
-t or -p or -s I meant to write
<Smasher>
but i would like to have my own shortcut
<Smasher>
we should be able to disable or override the default hpx ones
<Smasher>
^
<jbjnr>
file an issue requesting the ability to disable the default hpx shortcuts.
<jbjnr>
it might already be possible, but I don't know how
<jbjnr>
" It is possible to define your own shortcut options. In fact, all of the shortcuts listed above are pre-defined using the technique described here. Also, it is possible to redefine any of the pre-defined shortcuts to expand differently as well."
<Smasher>
ah no, that is something different
<Smasher>
ah
<Smasher>
well i dont have an .ini
hkaiser has joined #ste||ar
<Smasher>
and not planning to add one
<jbjnr>
quite right. we should add a compile time option in cmake to disable all that cruft
<shoshijak>
Hi hkaiser! I'm working on allowing several thread-pools to run in HPX. With the changes I made, the threadmanager now holds several pointers to threadpools. I now run into a bunch of difficulties because of functions like threadmanager_impl::register_work(...) which calls thread_pool::create_thread(...). There will have to besome sort of mechanism for the user to specify which threadpool the task should be created on
<shoshijak>
and I'm not sure how to go about this
<hkaiser>
shoshijak: for now I'd suggest that the thread-manager API uses one of the thread-pools as before
<shoshijak>
[note that with my changes, runtime and threadmanager are not templated on the scheduler type anymore, but thread_pool still, and uses type-erasure so that threadmanager can hold a pointer to thread_pool_base]
<shoshijak>
hkaiser: you mean just schedules everything on a default thread pool?
<hkaiser>
any extensions should go into executors which can go through a new API for scheduling on a separate thread-pool
<hkaiser>
yes
<shoshijak>
oh I see
<hkaiser>
shoshijak: wrt type erasing runtime and thread_manager - good! that's something which was in preparation for a while
<hkaiser>
shoshijak: in the end I think we will get rid of the register_work API and replace it with executors
<shoshijak>
hkaiser: that's great :)
pree has joined #ste||ar
<shoshijak>
the threadmanager does very little now ... mostly just forwards stuff to the pool
<shoshijak>
that class doesn't really have a good raison d'être
<shoshijak>
probably should be deleted at some point, right?
<shoshijak>
and another question: should I just merge the now-untemplated runtime with runtime_impl, or do you want to keep them separate in case?
<shoshijak>
*just in case
<hkaiser>
jbjnr: you can disable the shortcuts by add --hox:ini=hpx.commandline.aliasing!=0
<jbjnr>
ooh. Smasher ^^^ see that ^^^
<jbjnr>
s/hox/hpx/
<hkaiser>
shoshijak: so have you moved the perf-counters to the thread-poll as well?
thundergroudon has quit [Ping timeout: 246 seconds]
<hkaiser>
jbjnr: right, thanks - typo
<jbjnr>
thanks hkaiser been wondering about that for a while - pity it isn't mentioned on the command line options docs page ;)
<zao>
Hah, wrote a horrible program to stable sort the build logs from a devenv.exe invocation into Build Order.
<hkaiser>
sorry
<zao>
Now I can reliably get HPX build logs without VS crashing :D
bikineev has joined #ste||ar
<jbjnr>
hkaiser: the perf counbters is temporarily commented out, but I think(hope) shoshijak has a plan
<shoshijak>
hkaiser : no, since I know very little about the perf counter code, I commented the whole part about perf counter in threadmanager.cpp out. But this will have to be moved to thread-pool, so that the user can get perf info about each threadpool separately
<jbjnr>
we were thinking that the default pool will always be pool#0 and then each user pool gets an Id 1,2,3 ... so that perf counters can be requested by pool using an index - or if the user has named them, using the name
<hkaiser>
shoshijak: the actual perf-counters live in the thread-pools and below (scheduler, queues, etc.) but their definition is in the thread-manager and I think we should keep it that way (YMMV)
<hkaiser>
jbjnr: nod, sounds about right
<hkaiser>
having a class which does nothing more than encapsulate the whole things is not a bad thing, so I'd be hesitant to completely remove it - but you have a better understanding of where you'd like to go
<shoshijak>
hkaiser : I'd rather get to a working solution first, and then once it's more clear to me which role each class has taken on, think of whether this needs cleaning up
<hkaiser>
right, +1
<jbjnr>
hkaiser: when you're looking for PhD students ... I know where you can find one
<shoshijak>
about the perf counters: I haven't looked at that part of the code at all yet... I'd rather leave them commented out for the moment and get back to that also once I have a working solution
<hkaiser>
ohh
<hkaiser>
jbjnr: tell me ;)
<hkaiser>
shoshijak: ok, fine with me
<jbjnr>
I can't - she's reading this!
<hkaiser>
I wouldn't like to lose those, though
<hkaiser>
lol
<hkaiser>
by all means, let's talk about this at som epoint
<pree>
Hi all
<pree>
When applying hpx::apply on global plain action should we care about return value from the action?
bikineev has quit [Ping timeout: 268 seconds]
<jbjnr>
no. apply doesn't return anything, so you can just drop any return value
<pree>
In docs states that return value is ignored
<pree>
solved thank you
K-ballo has joined #ste||ar
<jbjnr>
c++ wizards : I have a new question regarding allocators. If I use a std:::vector<T, rma::allocator<T>> and the vector is resized to a from a large length M to a smaller length N. then the chunk of memory I've got pinned will be length M, and how can I make the vector tell the allocator that only N bytes are in use?
shoshijak has quit [Ping timeout: 246 seconds]
<hkaiser>
jbjnr: why do you have to?
<hkaiser>
pree: apply() invokes a fire&forget operation, it will ignore the result
Matombo has quit [Remote host closed the connection]
<jbjnr>
hkaiser: during serialization/deserialization, the rma chunks have 'size' and 'used space' vars that are used to control the rma operations, now when I serialize the vector I can set the used space flag accordingly, but it occrs to me that there is scope for problems when these two don't agree. everything should work fine, but it'd be nice to make sure by setting the used space in the region...
<jbjnr>
...to be correct
<hkaiser>
pree: apply returns a bool which is true if the operation was executed locally, false otherwise
<pree>
That's the catch. I thought it
<pree>
Thanks !
<hkaiser>
jbjnr: the allocator does not care how much of the allocated memory is actually 'used'
<hkaiser>
it may care about how much memory was allocated for the block, though
<hkaiser>
so I think it shouldn't be problematic if those two numbers diverge as long as used <= allocated
<jbjnr>
the used space is used to limit the rma length when rma put/get operations happen and I need to make sure that the vector sets it correctly during serialization. it should be fine.
<hkaiser>
sure, but that's nothing concerning the allocator
<jbjnr>
I used that flag in some places in the code and I need to be careful
<hkaiser>
your rma_object<> should know how much memory is 'used', so does vector<>
<jbjnr>
yes
bikineev has joined #ste||ar
Matombo has joined #ste||ar
Matombo has quit [Remote host closed the connection]
jaafar has joined #ste||ar
taeguk[m] has joined #ste||ar
EverYoung has joined #ste||ar
EverYoung has quit [Remote host closed the connection]
EverYoung has joined #ste||ar
EverYoung has quit [Remote host closed the connection]
EverYoung has joined #ste||ar
hkaiser has quit [Quit: bye]
jaafar has quit [Ping timeout: 260 seconds]
jaafar has joined #ste||ar
ABresting has joined #ste||ar
aserio has joined #ste||ar
jaafar has quit [Read error: Connection reset by peer]
denis_blank has joined #ste||ar
EverYoung has quit [Ping timeout: 255 seconds]
hkaiser has joined #ste||ar
EverYoung has joined #ste||ar
<github>
[hpx] KhalidHasanov opened pull request #2640: A new throttling policy with public APIs to suspend/resume (master...master) https://git.io/vHvpL
EverYoung has quit [Remote host closed the connection]
<hkaiser>
heller__: care to create a ticket for removing queue?
<github>
[hpx] hkaiser pushed 1 new commit to Resource_manager_fixes: https://git.io/vHffY
<github>
hpx/Resource_manager_fixes 03cb228 Hartmut Kaiser: Using new executor API
<heller__>
hkaiser: done.
<hkaiser>
tks
bikineev has joined #ste||ar
pree has quit []
jaafar has joined #ste||ar
ABresting_ has joined #ste||ar
ABresting has quit [Quit: Page closed]
ABresting_ is now known as ABresting
ABresting_ has joined #ste||ar
ABresting_ has quit [Client Quit]
jaafar has quit [Ping timeout: 258 seconds]
jaafar has joined #ste||ar
david_pfander has quit [Ping timeout: 268 seconds]
bikineev has quit [Ping timeout: 240 seconds]
EverYoung has quit [Remote host closed the connection]
EverYoung has joined #ste||ar
EverYoung has quit [Ping timeout: 260 seconds]
vamatya_ has joined #ste||ar
ajaivgeorge has joined #ste||ar
<jbjnr>
someone asked the question "if an action is invoked on the local locality (find_here()), is the serialization bypassed - and I said, yes it is. Was that correct - or does it still get serialized and then deserialized
<hkaiser>
jbjnr: yes, correct
<jbjnr>
ok. thanks. just wanted to be sure - I started doubting myself
<K-ballo>
it would still require serialization to exist, though, right? it's a runtime decision
<K-ballo>
it's basically serialization elision :)
<hkaiser>
yes
pree has joined #ste||ar
<heller__>
hkaiser: thanks for reacting on khalids PR so quickly
bikineev has joined #ste||ar
<jbjnr>
what does the new scheduler do that differentiates it from others?
<jbjnr>
(I didn't look at the code yet)
<heller__>
jbjnr: it allows use to shut of cores and enable cores again
<heller__>
wasn't doable with the old policy, IIRC
<jbjnr>
ok. hopefully the new resource manager stuff will allow that too
pree has quit [Ping timeout: 260 seconds]
<jbjnr>
(we'll just borrow the code from the old RM and throttling scheduler if/when we need it)
<hkaiser>
heller__: that's something shoshiak will want to be involved so we can do that through the resource manager and not through special magic in the scheduler
<heller__>
hkaiser: yeah
pree has joined #ste||ar
<heller__>
the use case is to have N executors, where N is the number of NUMA domains
<heller__>
each of those executors should get 0 to M cores to run on
<jbjnr>
our RM will allow N pools, each with an executor
<heller__>
and a runtime adaptive measure to adapt
<heller__>
which should not be tied into HPX, imho
<jbjnr>
version 1 - will be static, no resizing of pools, or moving of threads from one to another, but V2 will expose that - and that functionality will need what you've got in the throttler to drain a core of tasks and shut it down before moving it, etc etc)
<heller__>
*nod*
<heller__>
I'll make Khalid aware of the development and let him make contact with shoshiak
<jbjnr>
It's a real shame. shoshana only has a month left, and we're now starting to make proper progress - we have so much still to do
<heller__>
yeah
<heller__>
what can we do keep her (assuming she wants to continue)?
<Smasher>
but the point is... the encryption provider has an encryption modulus too, which is derived from public key on the server side.. i.e. computated after the transmission
<Smasher>
and maybe it differs from the deserialized one
<Smasher>
which it shouldnt
<hkaiser>
just print the values in the serialize function
jaafar has joined #ste||ar
<zao>
I had to try to parse that pointer as a string. Old habits die hard.
<Smasher>
gmp's limbs are unsigned long long, which is 64 bit on x64
<Smasher>
on pi its 32 bit
<zao>
If your input value is constructed from bogus internals, the result of pow_ui is most likely bogus as well.
<Smasher>
hence the problem
<Smasher>
nein
<Smasher>
the input value is corerct
<Smasher>
correct
<Smasher>
the calculation result is wrong because its done 64 bit
<Smasher>
i would need to force gmp to work with 32 bit somehow ...
<zao>
What I'm saying is that you seem to be blitting internal representations of mismatching shape.
<zao>
Either figure out how to reinterpret your internals into the desired shape, or convert to a format that can be cheaply parsed on the other end.
pree has quit [Ping timeout: 260 seconds]
<Smasher>
i dont think i can reinterpret 32bit bigintegers as 64bit on pi
<Smasher>
so i need a solution on the other end
<hkaiser>
Smasher: how does gmp solve this on the pi?
<Smasher>
pff.. no idea
<zao>
hkaiser: Solve what?
<hkaiser>
zao: he is using gmp on the pi, right?
<Smasher>
yes
<zao>
I'm of the impression that GMP itself works perfectly fine on 32-bit pi, except that you may be slightly limited in using the integer constructor.
<hkaiser>
so if gmp uses 64bits to represent something on x64, how does it represent the same thing on the pi?
<Smasher>
in fact this i quite strange... that value has _mp_alloc = 64, _mp_size = 64 on pi... and _mp_alloc = 64, _mp_size = 63 on x64 machine
<zao>
hkaiser: The innards of a bigint is a few book-keeping figures and an allocated array of limbs.
<Smasher>
it needs one limb less
<hkaiser>
god knows what tricks they play internally
<zao>
A limb is a 32/64-bit integer, which together make up the bigint.
<Smasher>
maybe i just have to rebuild gmp as 32 bit somehow
<zao>
Naively blitting limbs together is likely to result in horror.
<Smasher>
gimme that ./configure --help
<hkaiser>
Smasher: so you can't blindly serialize a gmp type
<hkaiser>
you have to reinterpret the data on the other end to construct an equivalent gmp instance
<hkaiser>
serialize to a string a reconstruct or something
<Smasher>
the problem are the calculations
<hkaiser>
and*
<Smasher>
it starts with the encryption modulus with is derived wrong
<zao>
Doing the same operation on any implementation should result in the same result.
<hkaiser>
zao: yes
<zao>
If not, you've found a bug in GMP or are using specific functionality.
<Smasher>
but actually it should represent the same value
<hkaiser>
but the internals may differ significantly between 32 and 64 bit gmp
<zao>
I would recommend altering your serialization to go the slow and safe way by rendering the number to a string, and constructing from the string on the other end in deserialization.
<zao>
Or run some tests with string literals on both ends to verify that your operations result in the same results.
<hkaiser>
right, that's what I said ^^
<Smasher>
i dont think it helps
<Smasher>
the data which is received on the other end looks okay
<Smasher>
but who tells us that gmp on x64 interprets it in the same way
<zao>
As components or as a fully reconstructed GMP value?
<Smasher>
i receive a public key on the other side which consists of two big integers, one of them is a modulus n
<Smasher>
then the server constructs an encryption modulus from that n by powering it to 2
<Smasher>
and that pow operations gives a seemingly different result
<zao>
(ah, there's an gmp_init_set_str too, sneaky docs)
<Smasher>
mhm maybe i made that conclusion too fast...
<Smasher>
[21:43:55] <Smasher> but actually it should represent the same value
pree has joined #ste||ar
<Smasher>
the limbs look different, but they are the same bigint, arent they
<Smasher>
maybe that's a good question for so :)
<zao>
Those limbs represent exactly the same value.
<zao>
mpz_get_str on the corresponding platform would render the same string as the input used.
<Smasher>
but
<Smasher>
what if i get a bigint created on 32 bit machine and would render it on 64 bit?
<Smasher>
lets try it
<zao>
I would recommend investigating mpz_export/mpz_import as a way to render a mpz_t to a portable representation that can be understood by both ends.
<zao>
The meaning of the limbs is very dependent on the way GMP is built on a particular platform, which somehow encodes the sign and the value bits of a bigint in a bunch of native words.
<zao>
Not all bits in a native word may be used, and there may be differences in encoding as it uses them.
<zao>
The algorithms are written to leverage the particular encoding on a particular platform.
<Smasher>
so the serialization just wont work like that between several architechtures
<zao>
A hypothetical representation could be that you have the sign in the lowest bit of the first limb, and value bits in the remaining bits of the first limb, and filling out additional limbs completely.
bikineev has joined #ste||ar
<zao>
Or using say 29 bits out of 32, as algorithms need a bit of headroom anyway to operate.
<zao>
Or something completely different.
<Smasher>
so i seem to be on the right track now
<zao>
The key takeaway is that unless documented to be transmittable, the innards of a mpz_t is mostly opaque.
<Smasher>
im gonna call ToString on both sides now and see what happens
<zao>
Please do.
<Smasher>
(gdb) p paillier.GetPublicKey().n.ToString(10).c_str()
<Smasher>
Thread 2 received signal SIGSEGV, Segmentation fault.
<Smasher>
:D
<zao>
I'm getting fond memories of when I wrote my bignum for ICPC competition practice.
<Smasher>
gdb cant print c++ strings?
<zao>
Figured it was a moot thing to do when I got to division and used MPFR instead.
<zao>
Smasher: I would say that it's more likely that the ToString implementation ran into inconsistent internal state of your value and blew up.
<Smasher>
the first call looked fine though
<Smasher>
(gdb) p paillier.GetPublicKey().n.ToString(10)
<Smasher>
since there are such thick parametrized mpz_import / mpz_export functions, i think the gmp's representation really differs on different architectures
<Smasher>
^ zao
<zao>
Smasher: They're also designed to encompass the case where you have bit bits of a bigint lying around so you can try to reconstruct one.
<Smasher>
so im gonna mpz_export my bigint to an array and send it then via the wire
<Smasher>
not sure what you mean with "bit bits of a bigint lying "
<Smasher>
around*
<zao>
"lying around" as in pre-existing.
jaafar has joined #ste||ar
<zao>
You've got a bigint in some particular format from another library or device.
<Smasher>
i see
<zao>
In this case, we'd use them to emit words of a known format, so we can ingest them again.
pree has joined #ste||ar
<pree>
There are some typos in docs (compiling hpx components with pkg-config section). Can someone correct it? I don't know how to correct it
<pree>
Thanks :)
jaafar has quit [Ping timeout: 246 seconds]
<zao>
Note that mpz_export/mpz_import doesn't do anything with the sign, you have to communicate that separately.
<zao>
pree: The documentation is generated from files somewhere in the source tree, so you could edit the sources and make pull requests.
<zao>
Smasher: There's two modes you can use mpz_export in.
<zao>
You either figure out how much storage you need pessimistically and pass storage in and a pointer to the number of "words" for that storage as the first two args.
<zao>
Or you pass in nullptr for the storage and a pointer to a size_t that is filled with the count of words the function allocates for you.
<Smasher>
oh
<zao>
In either way, it returns a pointer to the storage, which you get to free if you use the lazy allocating form.
<zao>
There's also a special case for a mpz_t that holds a zero, for which it returns nullptr and a count of 0.
<zao>
You should also note that it ignores the sign, so you have to persist/copy that yourself with mpz_sgn and mpz_neg if you care about negative numbers.
<Smasher>
but what i understand is that the format doesnt really matter
<zao>
You can indeed pick almost any representation, as long as it's not machine dependent.
pree has quit [Quit: :)]
<zao>
Passing endian=0 would be dumb, for example.
<Smasher>
as long as i read it with the same format as the one what i save in
<zao>
(0 endian means whatever the code runs on, which will break when going between endianesses)
<Smasher>
mhmm
<Smasher>
im not sure about those "words"
<Smasher>
we are talking about word as 32 bit right?
<Smasher>
or even 16
<Smasher>
32 is dword
<zao>
"words" because the size we talk about depends on the value you pass in as "size" parameter.
<zao>
If you say size=8, a "word" in the function description is a byte.
<Smasher>
okay... so its kinda chunk if we call it more general
<zao>
Indeed, chunk is a much better word.
<Smasher>
what mode would you suggest to go for
<Smasher>
lazy or pessimistic?
<zao>
You also have the related parameters "nails" if you want to skip some of the top bits in each output chunk.
<Smasher>
why in heaven i would want to skip bits?
<zao>
Smasher: If you know something up-front about the storage and can acquire storage cheaper than GMP's allocator, acquire memory up-front and use the one where you pass in a data pointer.
<zao>
If you don't know or don't care, use the allocating form and remember to properly free the storage.
<zao>
Smasher: As the documentation for mpz_import says, some implementations have stupid hardware implementations that creep into user space.
<Smasher>
i dont know and i dont care
<zao>
Where not all bits of a value contribute to the value :)
<Smasher>
:)
<Smasher>
great
<zao>
That's straying quite a bit away from standards C++ and what we care about :)
<Smasher>
so the explicit strorage allocation is safer
<zao>
Smasher: Oh, those were about the "skip bits" part.
<Smasher>
ah, nails
<Smasher>
that would be the beginning, right? char* to_send = mpz_export (nullptr,
<zao>
Aye.
<Smasher>
then you say "a pointer to a size_t that is filled with the count of words the function allocates for you."
<Smasher>
doesnt make sense yet
<zao>
Word salad!
<Smasher>
ah, that just a pointer
<zao>
In a C API, you tend to have input parameters as values, and output or inout parameters as pointerse.
<Smasher>
uallocated
<zao>
So size_t x; f(&x); to get a value out into x from f()
<zao>
My timeline is very tainted by how things arrived in MSVC++>
<zao>
Which is rather distorted.
<Smasher>
delete buf would only delete the first element, right?
<Smasher>
hence delete[] function
<zao>
Smasher: Even worse, it'd burn your house down.
<Smasher>
lol :D
<zao>
Mismatching new[]/delete[] and new/delete is a mortal sin.
<Smasher>
it has been a while i visited c/c++ course
<Smasher>
but i really visited one :)
<Smasher>
hey, lets run that serialization test i wrote >(
<Smasher>
:D
<Smasher>
wew
<Smasher>
[ PASSED ] 4 tests.
<Smasher>
lets hope it works via the network as well
<zao>
Smasher: Also beware the case of value 0 in the mpz_t value.
<zao>
The function will return nullptr then, and you'll have a possibly bad day.
<zao>
Probably safe.
<Smasher>
//todo: take care of that 0 case
<zao>
Scary thing about writing a s11n function is that it may be sufficient for your particular use case, but someone down the line may be using it for zero or negative values :D
<Smasher>
and then burn me :)
<Smasher>
in curses
<Smasher>
welp, a simple if (bla = nullptr) would be enough i guess
<K-ballo>
yey assignment!
<Smasher>
==`
<Smasher>
==
<Smasher>
^
<K-ballo>
sorry, matter of habit
<Smasher>
np :3
<Smasher>
i use sometimes assignments in if statements though :P
<Smasher>
lol i remember a case when a student once said "an if loop"
<Smasher>
his ass got kicked by the docent
<zao>
I still write some of my hobby code with yoda conditionals.
<zao>
if (3 == n)
* K-ballo
shakes his head in disapproval
<zao>
K-ballo: At work I'm compiling the competition now, StarPU.
<K-ballo>
fun! cmake based?
<zao>
Very configure.
<zao>
At least 1.2.x
<zao>
trunk may be different.
jaafar has joined #ste||ar
<github>
[hpx] K-ballo force-pushed drop-vs2013 from 5f9f0e2 to e0af8c3: https://git.io/vHJsA
<github>
hpx/drop-vs2013 98e2b41 Agustin K-ballo Berge: Remove support for VS2013
<github>
hpx/drop-vs2013 8f56767 Agustin K-ballo Berge: Mark C++11 defaulted functions as required
<github>
hpx/drop-vs2013 7352fed Agustin K-ballo Berge: Mark C++11 noexcept as required
<Smasher>
zao ready to learn if it worked?
<Smasher>
finger triple crossed here
<Smasher>
teh gdb still crashes on the server side eheh
<zao>
Smasher: Oh, have you ensured that the mpz_t is mpz_init (or equivalent) on the outside of the deserialization call?
<zao>
(using the C++ wrapper or whatnot)
<Smasher>
ohm.. nope
<Smasher>
i dont really have access, becaue i dont send bare mpz_z
<Smasher>
number of limbs is half because the size of one limb is doubled
<zao>
Seems legit.
bikineev has quit [Ping timeout: 260 seconds]
bikineev has joined #ste||ar
<Smasher>
seems working
<Smasher>
fingers still crossed
<Smasher>
the performance is a joke but it doesnt matter
<Smasher>
running since 7 minutes now which is a good sign
<Smasher>
100% cpu load thoughput
bikineev has quit [Remote host closed the connection]
EverYoung has joined #ste||ar
<Smasher>
[00:02:03] <zao> Better not throw exceptions inbetween and leak :D
<Smasher>
zao so remove std::throw_exception?
bikineev has joined #ste||ar
<zao>
Smasher: If you're dealing with allocated storage not held by smart pointer or other RAII functionality, you get the pleasure of finding out if anything inbetween the point of allocation and the point of release, if there's anything that might throw and unwind there.
<zao>
If so, you get to intercept the exceptions and clean up, or better, wrap your allocations in something that destructs properly on unwind.
<Smasher>
zao i think i can skip this part and just leave it as is
<Smasher>
there is no allocation before the line where the exception can be thrown
EverYoung has quit [Ping timeout: 272 seconds]
<zao>
Don't you allocate, attempt to serialize, and then free?
<Smasher>
Exception e = null; try {...} catch (Exception ex) { e = ex; } if (e != null) throw e;
<Smasher>
just easy as that :)
<zao>
Smasher: You get the joy of cleaning up in the catch handler and then rethrowing.
<zao>
Unless you feel like you can somehow recover from the s11n exception (if there's actually any).
<zao>
(I haven't investigated if our s11n can throw)
<zao>
I'd just dig around for a RAII buffer type, or handroll one.
<Smasher>
ho ho ho ho,... that protocol runs since 40 now.... for one single comparison of one single fingerprint with another single fingerprint
<Smasher>
but it's SAFE AS HELL
<Smasher>
consuming 100% cpu all the time
<Smasher>
im just laughng
<zao>
Smasher: I know of ancient DNS servers out in the wild that are secure simply because a login attempt takes too long time to validate thanks to crypto strength.
<zao>
A bit surprised to see an exponentiation without builtin modulo there, but I guess that your crypto may be different from what I'm used to see :)