32-Bit vs. 64-Bit Interfaces
IRC, freenode, #hurd, 2011-10-16
<braunr> i guess it wouldn't be too hard to have a special mach kernel for
64 bits processors, but 32 bits userland only
<youpi> well, it means tinkering with mig
<braunr> like old sparc systems :p
<youpi> to build the 32bit interface, not the 64bit one
<braunr> ah yes
<braunr> hm
<braunr> i'm not sure
<braunr> mig would assume a 32 bits kernel, like now
<youpi> and you'll have all kinds of discrepancies in vm_size_t & such
<braunr> yes
<braunr> the 64 bits type should be completely internal
<braunr> types*
<braunr> but it would be far less work than changing all the userspace bits
for 64 bit (ofc we'll do that some day but in the meanwhile ..)
<youpi> yes
<youpi> and it'd boost userland addrespace to 4GiB
<braunr> yes
<youpi> leaving time for a 64bit userland :)
IRC, freenode, #hurd, 2011-11-14
<braunr> also, what's the best way to deal with types such as
<braunr> type cache_info_t = struct[23] of integer_t;
<braunr> whereas cache_info_t contains longs, which are obviously not
integer-wide on 64-bits processors
<braunr> ?
<youpi> you mean, to port mach to 64bit?
<braunr> no, to make the RPC declaration portable
<braunr> just in case :)
<youpi> refine integer_t into something more precise
<youpi> such as size_t, off_t, etc.
<braunr> i can't use a single line then
<braunr> struct cache_info contains ints, vm_size_t, longs
<braunr> should i just use the maximum size it can get ?
<braunr> or declare two sizes depending on the word size ?
<youpi> well, I'd say three
<braunr> youpi: three ?
<youpi> the ints, the vm_size_ts, and the longs
<braunr> youpi: i don't get it
<braunr> youpi: how would i write it in mig language ?
<youpi> I don't know the mig language
<braunr> me neither :)
<youpi> but I'd say don't lie
<braunr> i just see struct[23] of smething
<braunr> the original zone_info struct includes both integer_t and
vm_size_t, and declares it as
<braunr> type zone_info_t = struct[9] of integer_t;
<braunr> in its mig defs file
<braunr> i don't have a good example to reuse
<youpi> which is lying
<braunr> yes
<braunr> which is why i was wondering if mach architects themselves
actually solved that problem :)
<braunr> "There is no way to specify the fields of a
<braunr> C structure to MIG. The size and type-desc are just used to
give the size of
<braunr> the structure.
<braunr> "
<braunr> well, this sucks :/
<braunr> well, i'll do what the rest of the code seems to do, and let it
rot until a viable solution is available
<antrik> braunr: we discussed the problem of expressing structs with MIG in
the libburn thread
<antrik> (which I still need to follow up on... [sigh])
IRC, freenode, #hurd, 2012-12-12
In context of memory management.
<tschwinge> Or with a 64-bit one? ;-P
<braunr> tschwinge: i think we all had that idea in mind :)
<pinotree> tschwinge: patches welcome :P
<youpi> tschwinge: sure, please help us settle down with the mig stuff
<youpi> what was blocking me was just deciding how to do it
<braunr> hum, what's blocking x86_64, except time to work on it ?
<youpi> deciding the mig types & such things
<youpi> i.e. the RPC ABI
<braunr> ok
<braunr> easy answer: keep it the same
<youpi> sorry, let me rephrase
<youpi> decide what ABI is supposed to be on a 64bit system, so as to know
which way to rewrite the types of the kernel MIG part to support 64/32
conversion
<braunr> can't this be done in two steps ?
<youpi> well, it'd mean revamping the whole kernel twice
<youpi> as the types at stake are referenced in the whole RPC code
<braunr> the first step i imagine would simply imply having an x86_64
kernel for 32-bits userspace, without any type change (unless restricting
to 32-bits when a type is automatically enlarged on 64-bits)
<youpi> it's not so simple
<youpi> the RPC code is tricky
<youpi> and there are alignments things that RPC code uses
<youpi> which become different when build with a 64bit compiler
<pinotree> there are also things like int[N] for io_stat_struct and so on
<braunr> i see
<youpi> making the code wrong for 32
<youpi> thus having to change the types
<youpi> pinotree: yes
<pinotree> (doesn't mig support structs, or it is too clumsy to be used in
practice?)
<braunr> pinotree: what's the problem with that (i explcitely said changing
int to e.g. int32_t)
<youpi> that won't fly for some of the calls
<youpi> e.g. getting a thread state
<braunr> pinotree: no it doesn't support struct
<pinotree> braunr: that some types in struct stat are long, for instance
<braunr> pinotree: same thing with longs
<braunr> youpi: why wouldn't it ?
<youpi> that wouldn't work on a 64bit system
<youpi> so we can't make it int32_t in the interface definition
<braunr> i understand the alignment issues and that the mig code adjusts
the generated code, but not the content of what is transfered
<braunr> well of course
<braunr> i'm talking about the first step here
<braunr> which targets a 32-bits userspace only
<youpi> ok, so we agree
<youpi> the second step would have to revamp the whole RPC code again
<braunr> i imagine the first to be less costly
<braunr> well, actually no
<braunr> you're right, the mig stuff would be easy on the application side,
but more complicated on the kernel side, since it would really mean
dealing with 64-bits values there
<braunr> (unless we keep a 3/1 split instead of giving the full 4g to
applications)
See also memory management.
<youpi> (I don't see what that changes)
<braunr> if the kernel still runs with 32-bits addresses, everything it
recevies from or sends through mig can be stored with the user side
32-bits types
<youpi> err, ok, but what's the point of the 64bit kernel then ? :)
<braunr> and it simply uses 64-bits addresses to deal with physical memory
<youpi> ok
<youpi> that could even be a 3.5/0.5 split then
<braunr> but the memory model forces us to run either at the low 2g or the
highest ones
<youpi> but linux has 3/1, so we don't need that
<braunr> otherwise we need an mcmodel=medium
<braunr> we could do with mcmodel=medium though, for a time
<braunr> hm actually no, it would require mcmodel=large
<braunr> hum, that's stupid, we can make the kernel run at -2g, and use 3g
up to the sign extension hole for the kernel map
IRC, freenode, #hurd, 2013-12-03
<azeem> I believe the main issue is redoing the RPCs in 64bit, i.e. the
Mach/Hurd interface
<braunr> mach has always been 64-bits capable
<braunr> the problem is both mach and the hurd
<braunr> it's at the system interface (the .defs of the RPCs)
<braunr> azeem: ah, actually that's why you also say
<braunr> but i consider it to be a hurd problem
<braunr> the hurd itself is defined as being a set of interfaces and
servers implementing them, i wouldn't exclude the interfaces
<braunr> that's what*
Structured Data
IRC, freenode, #hurd, 2013-06-25
<teythoon> is there a nice way to get structured data through mig that I
haven't found yet?
<teythoon> say an array of string triples
<braunr> no
<teythoon> :/
<braunr> but you shouldn't need that
<teythoon> my use case is getting info about fs translators from init to
procfs
<teythoon> should I go for an iterator like interface instead?
<braunr> depends
<braunr> how many do you need ?
<braunr> you could go for a variable sized array too
<braunr> have a look at what already exists
<teythoon> records, maybe 10-15, depends on many fs translators are running
<braunr> a variable sized array is ok if the size isn't too big (and when i
say too big, i mean hundreds of MiB)
<braunr> an iterator is ok too if there aren't too many items
<braunr> you may want to combine both (i think that's what proc does)
<braunr> be aware that the maximum size of a message is limited to 512 MiB
<teythoon> yeah I saw the array[] of stuff stuff, but array[] of string_t
does not work, I guess b/c string_t is also an array
<teythoon> how would I send an array of variable length strings?
<braunr> i'm not sure you can
<braunr> or maybe out of line
<teythoon> somehow I expected mig to serialize arbitrary data structures,
maybe it's to old for that?
<teythoon> yeah, I read about uot of line, but that seems overkill
<braunr> it is old yes
<braunr> and not very user friendly in the end
<braunr> let me check
<teythoon> we could stuff json into mig...
<braunr> see proc_getallpids for example
<braunr> we could get rid of low level serialization altogether :p
<teythoon> hah, exactly what I was looking at
<braunr> (which is what i'll do in x15)
<braunr> type pidarray_t = array[] of pid_t;
<teythoon> but that is trivial b/c its array[] of pid_t
<braunr> and always have the server writing guide near you
<teythoon> yes
<braunr> well, make one big string and an array of lengths :p
<teythoon> thought about that and said to myself, there must be a better
way that I haven't found yet
<braunr> or one big string filled with real null-terminated c strings that
you keep parsing until you ate all input bytes
<braunr> i'm almost certain there isn't
<braunr> type string_t = c_string[1024]; /* XXX */
<teythoon> yes
<braunr> even that isn't really variable sized
<teythoon> you think anyone would object to me putting a json encoder in
/hurd/init? it is probably better than me at serializing stuff...
<braunr> try with mig anyway
<braunr> the less dependencies we have for core stuff, the simpler it is
<braunr> but i agree, mig is painful
<teythoon> would it be too hacky if I abused the argz functions? they do
exactly what I'd need
IRC, freenode, #hurd, 2013-06-26
<teythoon> there is https://code.google.com/p/protobuf-c/ and it has a rpc
mechanism and I believe one could plug arbitrary transports easily
<braunr> please don't think about it
<braunr> we really don't want to add another layer of serialization
<braunr> it's better to completely redesign mach ipc anyway
<braunr> and there is a project for that :p
<teythoon> ive seen x15
<teythoon> just food for thought
<braunr> i've studied google protocol buffers
<braunr> and fyi, no, it wouldn't be easy to plug arbitrary transports on
top of mach
<braunr> there is a lot of knowledge about mach ports in mig
<teythoon> but again I face the challenge of serializing a arbitrary sized
list of arbitrary sized strings
<braunr> yes
<teythoon> list of ports is easier ;) but I think its worthwile
<teythoon> so what about abusing argz* for this? you think it's too bad a
hack?
<braunr> no since it's in glibc
<teythoon> awesome :)
<braunr> but i don't remember the details well and i'm not sure the way you
use it is safe
<teythoon> yeah, I might have got the details wrong, I hadn't had the
chance to test it ;)
<braunr> about this dynamic size problem
<braunr> a "simple" varying size array should do
<braunr> you can easily put all your strings in there
<teythoon> seperated by 0?
<braunr> yes
<teythoon> that's exactly what the argz stuff does
<braunr> you'll get the size of the array anyway, and consume it until
there is no byte left
<braunr> good
<braunr> but be careful with this too
<braunr> since translators can be run by users, they somtimes can't be
trusted
<braunr> and even a translator running as root may behave badly
<braunr> so careful with parsing
<teythoon> noted