S1E5: RISC-V runtime on the Web

RISC-V web runtime, Emscripten/WASM to a pure JS decoder for selfie.c and having a C compiler on a webpage. A case study on dodging issues with wasm64, codegen, and writing a new disassembler

Transcript:

all right everybody how is it going how
are all of you this has been quite a
journey so this show was supposed to go
on last week there was a lot that I
didn't expect that ended up happening
and well you know some tools didn't do
exactly what I thought they were gonna
do and the demo didn't didn't quite work
so I ended up pushing last week and Owen
hello hikers hello Stupak and hello
Brandon how are you how are all of you
so I went on kind of an insane journey
and to this this sessions gonna be a
little bit different than most of the
other sessions that I've done we're
pretty much we start from code from
scratch and you know run from there but
today unfortunately there's kind of too
much and I'm not exactly sure how much
we're going to get through so instead of
like just trying to like code everything
like here I think we're gonna break some
of those into some of the Wednesday you
know wine sessions and where we're gonna
use a few things that I built ahead of
time today and after we get that he
learned him I learned new what did I
learn him it's to pack so anyway right
well this has more to do with a case
study and the entire thing sort of began
with a problem that was unexpected when
I was hanging out with the enceladus
Oris okay I got the accent right place
and we were talking about learning live
learning see and I knew that at some
point in the future I wanted to do some
stuff with FPGAs cuz I think bunchy are
gonna be interested in that and one of
the the core that I'm going to use on
the FPGA is based on risk 5 and for
those of you that don't know risk 5 is
it's sort of like it's a it's a
pee-yew instruction set and it's open
which is kind of interesting because a
lot of the instruction sets that we
typically use like on this computer with
x86 64 or on my phone which is running
arm those are those are licensed
instruction sets and what's kind of cool
about you know you have to pay patents
to make chips and things like that you
know they're pretty reasonable but a
bunch of the educators at Cal said you
know this is kind of getting a little
tricky to like share a lot of this code
and share cores and share designs and
you know as teaching tools so why don't
we go with something that we just make
up which is you know based on our
experience with designing instruction
sets and we're gonna make it open source
you know open source hardware and you
know you can kind of do what you want
with it so they did a fantastic job so
I'm gonna be leaning heavily on some of
their work and there's a yeah there is
definitely a crazy journey yes to pack
this is relatively new it's it's been I
mean it's been sort of bubbling up
through the university for you know I'd
say five years or so maybe ten but you
know they they call it risk five because
it's their fifth iteration of the risk
architecture and they've you know kind
of been at this for like you know thirty
forty years who knows how long since
they they debuted their first versions
of risk but risk five is kind of neat
because it follows a lot of modern you
know multiprocessor stuff it allows for
all the modern enhancements that have
happened to CPUs but it it's it's very
minimal and it's designed partially for
education so it's it's not trying to be
like super overly complicated but still
being fairly performant if you wanted to
use it in like real applications and I
mean I think that there's I could go on
about risk five for quite a while we're
gonna get to some of that later but the
the net-net is that I mean they're using
it in they're building a supercomputer
now I think that's in the EU down to you
know open source hardware that you can
run on smartphones a year on design to
the stuff that we're going to be doing
later on with FPGA is we can actually
just synthesize this CPU core like it's
software and run applications on it
which really is software so
it's starting to blur that line so we're
gonna be covering some of that and hey
what's up nightshade dude and I hope you
get your presentation done and I'm gonna
try to get my presentation done which
hopefully all of you are gonna enjoy
today so without further ado let's let's
get this show going because we have a
lot of stuff to kind of get through so I
am going to have a little fun with that
he's just some guy he makes things fast
okay well this is so we are in episode 5
I can't believe it because this has been
quite a journey
that's so far and that today we are
going to be you know let's see o night J
didn't realize you were going mobile
today so good luck I don't know how much
they should be able to see on your
screen but the rough plan for today is
let's let's get our rough plan up so
yeah like I said we're gonna be doing
we're gonna be talking about CPUs
alright so what what's going on in a CPU
well there's there's a couple areas that
were we're gonna need for what we're
going through today and in particular
I'm interested in risk v later on in the
show mainly because this can be easily
synthesized on an FPGA
so if you want to make one of these
things you know to sort of run in real
Hardware as opposed to software
demonstrations that we're going to be
doing today you can do that and what's
kind of also cool about I mean anybody
could just make up their own CPU right
but you know this has been sort of
debugged you know so it it has been
through the paces and most importantly
it has a tool chain so this has been up
streamed into GCC and you know I think
there's an LLVM version of it as well so
you can kind of build like full-blown
applications including the Linux kernel
including all sorts other stuff and
what's cool about it as opposed to like
you know more traditional architectures
like you know motorola 68 km probably
dating myself or x86 64 64 bit version
which is really AMD's creation or you
know what's more typical in lower
powered devices such as the Raspberry Pi
or smartphone is you know the ARM
architecture risk v doesn't require any
licensing
these are all licensed and they're
they're really good they're really
performant we use them every day you're
definitely using them in this stream
just to see what I'm writing but this is
kind of a new new world because we have
the ability to effectively you know
treat this more like software and open
source which is kind of cool
all right so what kind of stuff do you
have an instruction set I mean you know
you you it really varies depending on
the application but in general you know
we're talking about very very so we're
not talking about high level stuff like
you know you might see in JavaScript
like console.log which is actually doing
like an insane amount of work to be able
to handle all sorts of different things
that you can throw at it but we're
talking about much lower level things so
these are things like you know like load
word from memory or store a word to
memory or you know add to this number
which is like immediately right here as
part of my operand or jump which we're
gonna have a little fun talking about
jump and Link a little bit later cuz I
found an error in the spec while working
on this but you know so this is like
very very basic programming in the sense
that like very little is happening you
know from instruction to instruction so
typically it takes lots of instructions
to do anything kind of interesting so
you know we don't actually run these
things directly like this this gets sort
of translated into binary so it you'll
end up with these like really long
binary strings you know which might go
on in our case we're going to be using a
32-bit instruction word today so every
single instruction that runs on this CPU
is 32 bits wide so that's cool because
it's a fixed size now there are variants
of the RISC 5 instruction set it allows
for 64-bit it yeah I can spell 2 so it
allows for 64-bit you know there's 128
bit and there's compressed which is neat
because you know there's a lot of
redundancy when you're when you're
spelling out this entire instruction
word depending on what
in there but we're not gonna be talking
about those today
no sir those are those are much better
understood on the web we're looking at
sort of smaller applications the sort of
thing that you know like pi can't think
like raspberry pi kind of space as
opposed to like full blown computer and
we're gonna be talking about single core
so you know the terminology that the
risk 5 community use is a heart so we're
gonna be talking about single heart
architecture and that's going to
simplify a lot of the complexity that
I'm gonna have to deal with so this is
this is the space that we're gonna be
operating in all right great well what
what I was originally planning on doing
with risk 5 was I was thinking like
alright this sort of came up because I
was I was talking to NC and she was like
yeah hey I know Python but you know I
really want to learn C you know and and
I said well that that's kind of cool
like you know I'll teach you that you
know let's let's just build a tool where
you know you and me can be distributed
different parts of the world and we can
go connect on some learning web site and
you know we'll have a shared IDE and
we'll be able to you know just program C
you know so I thought this was
relatively easy it turns out that this
is actually really really difficult for
those of you who saw the learn streams
and hey what's up that model awesome
yeah that's Oh what would up is here huh
it was his voice actually in the in the
intro stuff so you can't really see that
link right now Stupak so I'm gonna have
to just oh oh is this some library boot
firmware as he opens for that so I
haven't reviewed that but we're not
gonna be talking too much about like
regular firmware we're actually talking
about like the core guts of the computer
so we're talking about the CPU which has
a whole bunch of pins on it and
this literally does things like you know
this is access to memory you know and
this is access to you know some IO
devices and and this stuff maybe there's
some power things in here and you know
there's you know another bus maybe for
connecting to memory or you know like
maybe we're gonna wire USB in here
whatever we feel like doing with our
particular application I mean this is
usually an SOC architecture so I'm not
going to go too much into like Hardware
designs there's there's better streams
where you can go check that out but you
know and then we also need to have some
address pins so you know typically we're
saying things like okay this is the
address and this is the data and you
know this this is super oversimplified
version of the CPU why are we doing this
well it would be kind of cool if we
could take this entire thing and sort of
ship it over to web land because what I
noticed when I was trying to do these
learning sessions is that there isn't a
simple way to compile see applications
if you're on the web so yes well
overdubbed of this is um I thought it
was good that risk five actually fit
into episode five so this you know
what's neat about this is we have our
general web environment which doesn't
have things like you know Agnew C
compiler you know so like say say we
wanted to actually like compile
applications this thing is massive all
right and using some stuff like in
scripting you know we can we can
actually take this whole thing and
compile it into laz 'm which is web
assembly for those of you web developers
out there which is basically kind of
like a really stripped-down version of
just JavaScript and there's some there's
some optimizations available for that so
this was kind of like the path I was
originally thinking of going was you
know taking off-the-shelf compiler go
through this pipeline and end up getting
something that'll like compile down to
JavaScript so that I can just embed it
in a web page and on that web page we
can have like you know here's a
whiteboard and you know here's some
output and and here's a shared editor
you know and whatever we want to compile
in here could just be compiled by this
javascript peek
so what I was trying to avoid and it's
funny you bring that up Stupak because
just about every solution I've seen ends
up using it is I wanted to avoid like
having a server in the mix you know so
like it's very easy if you take this
code that's over here on the shared
editor send it on up to you know a
container or some sort of like server
that's that's divvied up and do some
functioning you know some processing
that you would do that doesn't belong in
a web environment normally and you could
just send the results back and you know
connect that up to the output well
that's what I ended up doing to get the
the course started and that's pretty
much what most of the the the tools that
I saw aside from like I think you
mentioned piyo died in a different week
and that that's that's more on the like
you know try and actually run the thing
in the browser so the like the holy
grail here is we figure out a way to run
a C compiler in a browser and not use
server resources this this is a
well-known pattern and you know it it's
understood how to do those things but it
has scaling issues because you're
consuming server resources so the idea
was like could we take this C and you
know write our a little like hello world
and you know take that thing and compile
it to some sort of object file and then
take that object file and run it on
JavaScript and then send that result out
and then you know we could get back to
like the business of learning C well
yeah that's what I thought naively last
week and then I realized like for
reasons that we're about to get into
that that was a little bit of a pipe
dream
so instead that's kind of where I ended
up dragging out risk 5 so I said alright
what I could do is if I'm limited in my
web environment I could figure out how
to get risk 5 to run in JavaScript and
if I can get that happening then we can
start figuring out how to take some sort
of C compiler and get it to run because
there's a lot of stuff that can run on
top of risk 5 which is not available for
JavaScript so this would effectively
like open up the holy grail
so that's kind of ya Stupak I was trying
to get something that does not involve
installing anything and is definitely
not tied to any sort of licensing
restrictions and by the way if you think
that Visual Studio is free it's not it
is free for certain limited use but it
is not something that is it has its
caveats and it its limitations and I did
not want to go down the path of like
dealing with like Microsoft not being
interested in you know the the sharing
screen you know scenarios so instead I
said all right why don't we stick with
stuff that's actually open instead of
things that are closed and I don't want
things that involve like you know some
long registration process and you know I
mean at the moment we're using discord
for authentication and that seems to be
about the right level because people who
are using discord or you know able to
connect on voice on stream ok so yeah
that was kind of the the general idea so
we're not gonna get that far today but
what we are going to do in order to kind
of go down this path
I found a off-the-shelf C compiler which
is called selfie and selfie is designed
to not only run on risk 5 and emit risk
5 instructions so it can output like an
object file which is risk 5 but it also
is self hosting so you can take that
thing and run it on its own little
hypervisor so you know this is I think
they call it hipster and they also have
this mips Terr piece and you know
there's there's a bunch of stuff here so
I thought ok well great let's just get
going with selfie so if we can just get
selfie to run on top of this then we're
good to go because it can comply it can
compile all those things well yeah
because life is hard there's trouble
when you try to take this thing and get
it to run straight in wasm and that's
that's kind of where we're gonna begin
our journey so yeah and that that turns
out that selfie
what I kind of missed in the intro Docs
is that selfie really is a 64-bit only
thing and lazon 64 is not available in
LLVM so you know strike two
guy so this is kind of just just kept
getting more and more fun when I was
trying to get last week's show coming so
I said okay well if we can't do that
maybe we can just extract the C part of
this and we can omit J s directly well
that's that's a big project cuz this
thing is you know not designed to do
exactly that so so there's a bit of work
there and yeah that's that's kind of
where we're gonna start so I thought
like well let's begin this let's begin
this journey okay so that's our setup
that's that's where things started to
get interesting so yeah I'm gonna cut
back over here and what I'm showing you
here this I'm kind of getting a little
bit ahead of myself but this is uh let
me skip over this is actually the
learning environment that I created and
this is currently borrowing server
resources so this is doing something
similar to what a bunch of the other
environments are doing and we actually I
mean the goal the first goal was really
to just get like off the ground and
start teaching C as soon as possible so
this this did that okay we managed to
build a version of this that you know it
takes C this is a shared editor so as
I'm typing and streaming the other
person is able to see it we're both in
the same voice channel eventually we'll
figure out how to put some video up here
and we have a shared whiteboard and the
output is shared so this is this was
kind of like you know where I got
started with all of that fun so then I
said well alright let's let's go pull
out risk five so risk five you know I
thought well alright before we get to
building our own stuff let's use
something kind of off the shelf you know
because that would be really nice
so what did i do I started with you know
well here we're we're just in our normal
learning and I already apologize to
those of you missed it in the intro but
you know I am going to be borrowing
heavily from pre-written code today so
everything here will not be live and let
me show you what because some of this
stuff takes a while to run so if we
wanted to start with scripting the first
thing we do is we clone it
alright great that works pretty fast
while we're working with and scripting
we can you know then pull it into our
environment so if we do that then we're
gonna end up with this EMCC wonder stuff
but before we can do that we have to
first install the particular tool chain
so this is the EMS DK and I'm going to
need the latest version of this that's
compiled now keep in mind we're running
em scripting on x86 64 so my machine is
you know normal Intel you know 64-bit
machine and this is pulling down a cross
tool chain which is designed which sits
on top of LLVM which basically is going
to take you know whatever programs I
write run on x86 64 entry and output
code that's going to be wiser so that I
can run it over here on the browser so
that's kind of Wick oh and Stupak thank
you for the sub I appreciate it
so that that was you know kind of where
things got started and as you can see
like most commands that are gonna be
involved today like this this stuff
doesn't happen instantly so yeah that's
that's one of our challenges so once we
have this latest thing we're gonna just
activate it for oh sorry I gave it two
conflicting commands there so I want to
activate the latest tool chain that I
just installed okay great
now I have a version of this that
actually runs which can produce wasum
which is which is cool I can I can go
and I can run their examples but I need
to pull in there
this is yeah I need to pull in their
environment
so we need to override so that if we go
running this thing like it actually kind
of runs correctly so we pull that guy in
and we're good to go
all right wonderful now we're ready to
start compiling stuff so if we go here
and you know we write a very simple C
program like you know I'm just gonna and
we'll use this for a bunch of our
samples so this is not gonna be
particularly interesting we're just
gonna take stdio and we're gonna print
like a very simple life hello Stupak for
our stream so alright not a very
complicated C program but you know it
enough to like kind of get us off and
run all right so if we run EMCC which is
now in my path and I compile this C
thing then I end up also realizing that
they need a bunch of support libraries
so you know it lazy loads and it starts
pulling all this stuff down or creating
it alright we're good to go we look and
we end up with you know our test file
our a dot out so this they literally
stick to the convention so you know if
you compile something you don't specify
what the output file is then it's eight
it's the assembler output so there's my
little JavaScript stub so I can go and
look at it and you know this is just
enough of JavaScript to load my wasum
stuff which is binary ish so that's cool
I can actually go and you know if I
wanted to get a little bit fancy I guess
I could I could turn this in scripting
is kind of nice because it'll actually
host your application in a web page so
if I go and I look at this test HTML
then this will actually build just
enough of a web page that I can see my
application running so that I don't have
to worry too much about building the UI
that's going to be here so I can
actually you know
do this so yeah we're going to need let
me just shut this real quick and let's
do yeah all right so yeah all right so
if we go here and we take a look at one
created then here's all my little I'm
scripting files and I've got this test
HTML which gives me hello stupid all
right
great we know how to use em scriptum now
why is this cool because here I'm
actually I wrote a printf right I'm
using C library C calling convention
it's compiling it and it's turning it
into web assembly that's great that's
webassembly 32-bit which you know is
where we're gonna hit the problem
momentarily okay so there I was
Thursday thinking wow and scriptum works
really well so let's just go and pull
down selphie you know and let's let's
get a copy of selfie which is gonna be
our self hosting C compiler which can
output its own just five instructions so
we take this thing and we try building
it so this basically has just a normal
make file and it uses just whatever your
C compiler is set to so you can set your
C compiler to whatever you want
so I'll set it to and scriptum which is
currently in my path and I'll make this
thing now it's it's just a single file
but you're gonna notice a bug report
which is where things started to get fun
so and it really wants me it says please
attach the following files alright we're
not filing the bug report right now and
if we go back through this whole error
thing what we're gonna realize is this
line over here so 64 bit web assembly
no that's problematic because selphie
requires 64 bit alignment because it's
it's by default emitting RV 64 so that's
risk v 64 bit so we can we can go
mucking around with a compiler
little bit you know we can we can we can
edit this thing and yeah this is this is
where we start the journey Stupak so you
know let's let's take our C flags and
let's you know let's kind of bump this
around a little bit let's let's get rid
of the optimizations that are here let's
let's get rid of this extra junk let's
let's let's get rid of the 64 bit and
instead of forcing you in 6040
to a 64-bit data type let's let's make
it a 32 bit data type
alright so sounds good compiler should
do some magic and maybe it'll all fit in
and you know we're good to go
well no because already we're
overflowing things but you know it
generates you know so now I have this
version of selfie which is webassembly
I got it to compile but yeah it's
requiring Fergie's memory so we can go
down the path of trying to modify selfie
and that's I'm not gonna bore you with
the details I spent a lot of time trying
to do that Thursday night and it you
know after stubbing out a bunch of
system calls because it's counting on a
little minimal kernel like it wants to
have open you know if you look in the
selfie code you'll see that there's you
know there's an open call open with a
parenthesis
okay so there's an open call there's a
Mal'akh call there's a reading right so
first you'd have to go and stub all
these out because we don't we're not
running with a kernel if we're running
on the web yeah
so I went down that path and then you
know then you have to change all these
types because they all require 64 and
then you know bad things happen when it
tries to hit a function that's not there
and anyway there went my Thursday into
Friday
so yeah Stupak I didn't either is the
moral of the story there okay so great
that didn't work so now we need to get
ourselves before I went to any further I
said well why don't we try to figure out
how to run risk 5 on the Internet ok and
and this is kind of where you know
things started to get very interest
things I'm just gonna throw this glove
back on because you know attempt one
here ended up not working too well so
all right well let's clear our minds so
this sorry this is you know now that
we're here I want to I want to switch
we're gonna go to another color space
because this is just this is a totally
different world all right so what do we
learn
well I'm script in with you know lazon
64 no dice and the goal there was to get
the selfie C compiler you know somehow
running you know in a web app alright
well we tried that we've learned that
doesn't work we can spend a bit of time
trying to relax this constraint we can
spend a bunch of time trying to get this
to work without a kernel we can do all
sorts other things I ended up not going
down that path I said well what's option
B all right well option B what is action
B you asked well let's not count on this
okay so let's look let's stay away from
lazon 64 let's let let's switch gears
for a minute and get back to what we
were you know originally trying to focus
on which was you know the C compiler
because that's what we needed for class
and of course
I wasn't gonna miss an opportunity to
completely over engineer something so I
said well let's take step 3 and find an
off-the-shelf risk 5 core which you know
will will host
selphie because I can get selphie to
compile to risk 5 not wasm 64 but I can
get it to run our V 64 now at this point
I didn't realize this bit constraint but
you might you the observant viewer might
already know so well I went and I found
like you know there's a couple
off-the-shelf things there's there's the
angel emulator there's some other
berkeley projects you know there's you
know that berkeley not berkeley the
music school there's
and these things and then there's kind
of like full-blown like J s Linux and
you know this this seemed like a bit
extreme just to get a C compiler but you
know it'll work but it'll be very slow
so I didn't want to start there if we
could avoid it and I knew that later on
I was gonna be spending time with risk 5
anyway for the FPGA stuff so I said well
why don't we invest a little bit so what
are we gonna invest in well how hard is
it to get this thing to actually work on
the internet well and that's kind of
where we went down the path so so this
this this like off-the-shelf path ends
up not working how hard is it to get
this to run on the internet well all you
got to do is build a CPU so you know we
have our wires and you know we just take
that and we put it on top of JavaScript
and because the magic of diagrams we're
done well this doesn't work too well so
in order to actually do this I picked a
minimal version of risk 5 so like I said
risk 5 has all sorts of variants I'm
gonna take a 32-bit version of it which
has only the integer extension which is
just I so none of the other stuff if I
can avoid it so the Atomics the
multiplier you know divider the floating
point the vector stuff you know know
okay we're gonna do Oh for like just let
the C compiler worry about how it's
going to assemble it what what minimum
part of risk 5 do I need to get this
working and we're gonna get rid of
compression so we don't want any
compressed op codes so every op code
should be a full 32 bits all right so if
I come here and I get my bits right you
know so this is happening at bit 7 and
this is bit zero and you know this is
bit 15 and this is bit 23 and by the way
I was off all sorts of ones as I was
moving through this over the weekend but
so no cam I haven't actually seen that
so
this if I can get every one of these
opcodes to be exactly the same size then
I don't need to worry too much it won't
be too hard to take every instruction so
this is a full instruction is going to
be 32 bits alright so it you know looks
like a normal you know 1 2 3 4 5 6 7 8
kind of number so this corresponds is
the hex representation of these and we
can actually go and you know look and
decode this instruction and you know see
if we can you know disassemble it if we
can disassemble it eventually we could
get around to interpreting it so this is
you know exactly where I move to next so
whenever I get an instruction coming in
on the CPU there's a couple steps that
the CPU does the first is it decodes the
instruction so it says all right what
what instruction is this you know it's
like okay is this a load so that's the
job of the decoder so I said let's let's
build a decoder first then it goes
alright what are my operands you know so
this is like what what arguments are
being passed along for this particular
instruction that's also part of the
decoder so I should really write this as
like you know we'll call this one point
two we'll call this one point one and
then we actually have the job of you
know running the instruction so this is
like you know execute so this this is
you know sometimes called oops execute
and they call this sometimes retiring
the instructions so you know whatever
needs to happen to actually finish this
thing like you know so like take this
and add it to this or you know whatever
the instruction does like actually go
and run it so this is kind of emulation
so we're not gonna worry about this
right now we're really gonna worry about
this segment for now okay great and hey
what's up then the like how are you
doing
pro-gamer great so sounds easy right
well I went and pulled up this by the
way I ended up and I don't generally
print things out but this is a printout
of all of the instructions that are
available and it's nice that it fits
onto one page and you can all see it
right here because I have it here for
you so these are all of the instructions
that we're going to need in order to be
able to run risk 5 so it's it's this
stuff over here I think they say there's
like 40 something but in reality a bunch
of these instructions you don't need to
emulate for simple purposes because like
you know fence for examples related to
like you know coordinating multiple
processor hearts and we're only gonna go
with one processor you know and there's
a whole bunch of CSR instructions we're
not going to need because you know I
don't we don't we don't really care
about getting our metrics right and all
this other stuff
yeah Denzil this is that where this is
actually about learning building a
learning tool to do si compiling so this
is you know this is kind of like the
gold here it would be you know be able
to do see on the web and run it
somewhere and so that's where I sort of
ended up breaking out into risk 5 so I
said this is a minimal computer we can
get to work in time so anyway yeah
that's definitely annoying Stupak so I
ended up opening up this manual and this
thing is the most recent version that
seemed to be standardized this is
version 2.2 and it's long so we're gonna
cover all of it on this string no no no
we're not gonna do that but what that's
where like you know if we jump down to a
few you know pages that I sort of
extract it just so that we could quickly
look at them this is the basics of every
one of those instructions fits one of
these formats this R is UJ B stuff and
that basically just says okay the opcode
like this is the format for that
particular instruction so for a j-type
which is really simple we have an opcode
which says that this is a jump kind of
thing
and we have Rd and immediate which are
the to feel all of them have this Rd
which is you typically the destination
so in this case I think it's where we're
gonna go next and this which is used as
an argument so this this is actually the
address here and that that's I was just
describing the JL instruction which
actually is kind of where I started to
have a little bit of fun because they
said when I read in the doc they said
well don't worry because in the latest
version of JL and J type instructions we
managed to get rid of them because there
used to be this jump instruction and now
we're using you type formatting so there
is no more J type form this removes the
J type instruction format from the base
eisah I thought that was great until I
went and I looked at the particular
stuff related to understanding this JL
instruction which is yeah this is this
is actually the piece that I just kind
of clicked for you
so this JL instruction which gets talked
about a little bit further down and this
manual is pretty big well it turns out
this still uses J type form there were a
few gotchas as I was implementing this
and it's a work in progress I don't
think anybody's been doing the 32-bit
version recently I don't I don't know
how much focus there's been on that I
think most of it's on the 64 bit but
yeah managed to catch our first book so
well do you want to see it because
that's kind of what comes next
so I ended up saying well let's look at
everything Thank You Stupak let's look
at everything that is going to be coming
and I thought this was going to be
relatively easy it turned out it was way
harder than I thought it was gonna be so
what I'm gonna do here is I'm just gonna
pull in
the version of it that I ended up
building so this is the decoder all
right so if I go and drop this thing in
here oh look magically I typed up this
whole massive file that's 420 lines and
I got it right the first time I did it
so this is this is it and just to give
you a sense of the kind of things that
was going on I was not trying to do
optimizations you know I was just trying
to get something good enough for the
Friday show originally and then I
realized this was way bigger than what I
thought and these masks turns out you
know in JavaScript you can work directly
with binary numbers you just put a 0 B
in front of it and shifts work except
this triple greater than because I
wasn't sure what was happening in
JavaScript when it was making my numbers
negative and well it turns out unsigned
and types and JavaScript I knew that I
was getting myself in for trouble so as
I went through this you know basically
the idea was here's the actual decoder
okay so if I take this decoder and you
know I feed it a place where it can
stick its output and I give it a binary
and I'm gonna get a little bit more into
that later but and we feed it these are
V 32 integer instructions yeah so just
that minimal instruction set that I sort
of showed you before so that's that's
this guy
then everything there should be four hex
digits at a time so you know I'd better
get an even multiple of four otherwise
you know you're sending me an incomplete
program so we go we start taking the
thing apart and basically if we pull the
last seven bits off the number and I did
that you know basically I just said all
right here's the last seven bits and if
I end it with that then I end up pulling
it off the instruction that just came in
by the way
this is the full instruction all I did
was I just put it in the right byte
order here so you know that the first
you know the the first text digits are
you know all the way on the left and
then the last ones are over on the right
so the thing on the right is the least
significant and the thing you know so
you know Indianness and all that fun I
wanted to be operating in JavaScript so
I put it into the right format up here
and we also get out the hex of what
we're gonna you know what it's gonna
what was passed to it so that's
instruction by instruction what I'm
getting all right well let me take this
thing and we want to pull the last seven
bits off it because that's the opcode so
if I look on this column I thought well
great I can just look up exactly what
the code is and then I'll know which of
these instructions you know is being run
and it should be that simple I'll be
done in like five minutes yeah
everything was as simple as I thought
for this episode so this is yeah it
turns out a bunch of these instructions
they have exactly the same opcode so
like if you look down here at like you
call an e-brake you know you'll notice
they're exactly the same how do you tell
them apart well it turns out you have to
look over here so the instruction set is
not like completely one-to-one with
these op codes it turns out part of the
opcode is actually hiding in the rest of
register and they did that to save space
and managed to get a pretty big
instruction set into and leave room for
expansion but boy that was hard yes yes
Denzil they sorry about that it's a
rough episode to come into first yeah so
you know I started off like oh yeah look
here I got the lui instruction we're all
good to go it just looks like that all
right and then if it looks like this
then I've got the au IPC instructions
alright so that was kind of the
beginning of my decoder then I realize
like a whole bunch of these shared the
same opcode you know like these these
branches here and they use these other
bits over here which are referred to as
funk 3 funked 3 and that's how you can
sort of just you know disambiguate those
well alright fine so I started getting
these groups and then things got a
little bit crazier and then I did the
same thing with these guys except you
know some of the turns out there's holes
you know like this bad here you know you
don't want to see bad because that's not
a valid instruction this oh one one it
would be I left notes about what was
what because I kept being off by one and
when you're on binary and you're off by
one you got a lot of trouble so yeah so
here's all the loads they use the L type
well it's not called L type these types
and then there's the the stores and then
here's the ads and then there's ads and
math math stuff ad and then there's the
person of that if it has a parameter in
memory and then anyway long story short
you take all that you package it all up
and you put it into a web app and well I
actually left my test harness down here
at the bottom so if you actually compile
a very basic program and base64 it then
you get your static program over here
and you can go read this binary file and
you can tell it to decode and it'll in
the case of node it'll drop it on
console so that's neat
i uncomment this and that'll work in
theory so yeah let's just make sure I
save that okay I did so we can actually
just run that and it's gonna go looking
because I left it looking for Linux
so let's you know let's turn that off
and see you are gonna get some code
today and what I did with this you know
I'm just gonna pull one of these static
programs I'm gonna take it from buck
base64 I'm gonna turn it back into
normal binary I'm gonna get my compiled
bin and then I'm gonna decode it okay on
it and voila so this is kind of what you
get through all of that suffering I
thought was going to be just a really
quick thing ready for a show so
and here I'm showing you kind of what I
was showing you before on the whiteboard
or as knightshade dude likes to say the
blackboard and that has this is the
actual instruction that's passed in so
this is 32 bits wide the opcode is down
here and then the rest of the opcode and
the arguments are over here and here
I've turned it back into like a normal
mnemonic you know something that we can
read as opposed to something the Machine
reads so here I've got an ad I and it's
it's an eye type instruction so if I go
to my chart
I'll see okay that's this line so in
order to decode that you need to look
for our D and our s1 and immediate now
ad I actually adds whatever is in
immediate to whichever register you
select and it stores the result in Rd so
that's kind of the semantics but what
does it end up doing well it stores it
in register two and this just says that
it's an ad because you know a bunch of
them use the same opcode and it's taking
register two and it's adding negative 32
and that's where my next set of bugs
were because negative numbers are very
large you have to do is complement and
it doesn't work with just the JavaScript
syntax so yes you're welcome nightshade
dude I'm not gonna go too much into the
detail of that because this this ended
up being like crazy but the moral of the
story is that you take all of this code
that I just showed you and I already had
it in it so let's go back to write
terminal and you end up with something
that you know and this is the reason why
I injected console.log because I wanted
to be able to use it at the command
prompt or I want to be able to use it in
a web application so that's why I added
a little bit of thin stuff I figured
like if I could upload a binary and let
the web application decode it that would
be pretty good because that then I could
test that this is working then we can
move on to the next stage of actually
you know virtualizing a whole cpu okay
so yeah this hardware is kind of
interesting I mean it's like it it's it
was a different mindset so I took a
little bit to kind of switch over here
but this is AK
the web app that I rolled so exactly the
same type of output that you saw before
except now I've gone and uploaded an
entire Linux kernel okay so that's a
pretty good torture test for this entire
thing and it ended up working nicely
which kind of surprised me but let's
let's actually go through some steps to
actually get there now we want this just
put this back up so this is the
disassembler and let's just reset no
let's reset this thing okay so it's
running here it's waiting for you to
upload a file all right great
well through the magic of tool chains
which ended up being more interesting
because what you really want to do if
you go and you compile this thing like
let's let's let's go back to our simple
C example again so you know actually oh
sorry I have it in my scripting
directory so let's just copy it from M
script in we had our test at C and we're
gonna put that in this directory alright
so not much going on in this program now
I definitely don't have this right this
this print def so let's just reduce the
complexity of this a little bit so I'm
not going to include stdio.h up than
simple like let's set variable I equal
to let's see who has a number in their
username Stupak does 62
okay so let's set I to 62 and then let's
do something like I is gonna get I plus
17 or actually let's pick something
smaller let's do I plus 7
okay simple C program I go and I compile
it yeah compiles fine right there's my a
out I run it
yeah runs fine okay there's no output on
it but what did I just do that's wrong
this is x86 64 okay that's not my risk
five that I am you know I didn't emulate
x86 64 I emulated risk 5 so I need to
compile this thing for risk 5 now with a
lot of work you can actually go and
create your own tool chain and I'm not
going to show you how to do that right
now but it is something that I'll
actually I'll post it in discord for
anybody who wants to but it turns out
like I originally tried to create this
as an RV 32i tool chain it turns out G
Lib C requires Atomics which is really
funny because Atomics so atomic
operators are basically like if you have
like four processors running something
you want to be able to say like this
process wait wait for all these you know
before you like change anything like
you're not allowed to change this
between when I pull it from memory and
when I'm like calculating it and putting
it back into memory like so there's sort
of like blocks so that you you don't
trip over yourselves with multiple cores
but I'm only emulating a single core so
I didn't need Atomics but G Lib C
requires them so first I built the tool
chain then I built the tool chain again
and then they built the tool chain for
today and then I tried the off-the-shelf
one which created risk 64 and then
anyway lots of fun later which and hey
what's up Panos how you doing
No Denzil you to call it with a switch
so like you know there is an M mark
switch GCC has to have the ability to
generate that instruction set so if i if
i this version of GCC only has x86 64
it typically GCC only has whatever
platform you're running on so this GCC
if i'm running on you know sorry if I'm
running on a x86 64 and I'm compiling on
it I'm compiling for x86 64 I'm sort of
one to one you know I'm omitting
instructions for the for the platform
that I'm running
but I want to go cross more like being
script in case I want to run the
compiler on x86 64 but I want to
generate for risk 5 so that's cross
compiling and the cross compiler is not
built into GCC I mean I guess you could
build a mega GCC that has every target
in it but that thing might be kind of
slow and crazy but so that's what these
are over here so if you if you look in
these directories you'll actually see
like this is basically just GCC that's
created for a particular target so in
this case it's I didn't want elf headers
because you know I'd have to parse it
off into some other stuff so this is
this is actually a 32-bit version so
this is unlike the tools that I
downloaded which all wanted to be 64 but
it will run on x86 64 so if I go and I
look at this thing this program itself
is x86 64 but it's going to emit risk 5
which is what I want
no it cross-compiling no it's not
built-in for targets your Netflix is
showing because we almost always have
arm targets built into our GCC yeah sure
Panos why don't you send it to the
discourse unless it's really short and
you think you could fit it into the
stream all right so I can run this thing
and run it against my little test
program which is could be the poem yes
thank you all right yeah go ahead pass
so
no dude no no bad come on man yes let's
let's keep things PG here so I don't
know can you do that I don't want to ban
you you're alright can I just get rid of
that message no that's gonna mod you
instead
that's all right let me time that out
all right so anyway okay so we go and we
come back here and before the poem and
we do this this cross-compiler that we
just created painstakingly so except
this is the wrong record I want the RB
32 this one was an off-the-shelf one but
it was it had it was generating
instructions for like multiply and other
things that I didn't want
so that been this thing and you know
we'll do some GCC sound good okay that
compiles fine and if I look at the
output
I'll see that this thing was 32-bit and
it was it's packed in the right format
for risk 5 and this should be following
kind of the convention that I want now
the downside with this thing if I just
try to use this file directly which
it's over here somewhere oh yeah it
doesn't like to see my app directory for
reasons so I'm just gonna this hacky
workaround is I'll just put this in the
download directory for us so yeah
there's there's some security thing
about that so if I if I do this thing
then I'm gonna realize that it has an
elf header okay so that's not part of
the risk 5 stuff elf headers are about
like putting the thing into the right
location in memory linking you know
intermediate results we don't want that
we're talking about raw code I want to
see the raw code which is from this
particular file all right well that's
fine because there's another tool that I
can use from the tool chain which is
called object copy and what I can do is
I can say give me the just the binary
from this file disregard the elf headers
and all the other stuff and dump that
thing in this binary file now because I
have to work around that path issue I
can go and load this just binary version
of it and there we go so if I look GCC
has gone it so we're getting a whole lot
more code here then and this well I mean
this is actually just a whole bunch of 0
so this is padding this has to do with
how you link the particular thing but
it's also adding in like some start and
some like typical C stuff for main which
I don't really want so if I want to get
rid of all that I have to tell GCC like
ok don't compile using that stuff so and
this kind of gets into some of the
architecture flags that
Denzil mentioned earlier so in this case
we just need to yeah basically take this
thing and we need to we need to spell
out how to link it because right now GCC
is adding
whole bunch of memory which would be
mapped into RAM onload and you know I we
have to specify like you know a template
for how to do this and I'm going to take
I'm gonna take a premade template that I
did over here which is bare metal okay
so we're gonna take our bare metal
layout and we're gonna put it back into
this app directory so yeah this bare
metal thing this is a linker template so
this basically is telling GCC like okay
I want you to put the code starting at
address 200 I want you to like text is
the actual program text you know with
the old terminology for it's not text
like you can read you know it's just the
the program itself any variables and
data stuff we're gonna put after it and
then we're gonna specify that that's the
end of the thing so it's going to link
it up and it's gonna put an elf header
that matches this format so great all we
got to do is compile that so I'm going
to give you a pretty big gnarly command
so let's take our not this let's take
our GCC command and let's walk through a
few things so first we're gonna spell
out that we only want instructions that
are coming from you know the 32 I set so
I don't want any other Atomics I don't
want any like other vectors or
multiplies or anything else then I want
to take that and I want to specify the
calling format so this is my ABI and I
want to use the standard LP 32 and I
don't want any stark file so this is
like kind of like normally when a
program is starting up but it it
initializes a bunch of stuff about the
environment so we don't want any of that
now I'm not gonna walk you through these
other parameters but they basically just
say like don't include anything okay so
use this linker layout that i just
copied over so this bare metal
out get rid of the debugging stuff and
you know put the actual resulting map
over here in this file
you know the references and make this
thing so that it can just stand on its
own it's not a program that needs to be
run by an operating system it's
something that can just be run by itself
like its own kernel directly on the
hardware now we have to add in all the
functionality to make that work but I
mean that basically tells GCC like don't
add all that other stuff and don't
include the standard library great I get
all that done which I get a new version
of this a out now this guy will match if
I look at that firmware map this will
show me what it actually did and it
should follow what I just told it to do
which actually this did it's going to
you know put the text you know it's
gonna jump ahead like you know to
address 200 it's gonna put the text over
there then the data segment and this is
all pretty much matches the bare metal
layout so that's you know GCC works so
great now I want to strip all of these
other sections out you know so like you
know this BSS and this this data and you
know all these other segments and that's
basically the object copy that I was
just running so let's just do that again
and we'll get rid of the elf header and
now if I copy this thing over to my
downloads so that I can upload it to
this oh and this tool has a little gosh
if that if the file name is called the
same thing then yeah bad stuff happens
okay so let's just reload this and now
we get the program we're expecting it
okay so here these instructions are
about setting up the stack that's just
GCC initializing some memory so that we
have a place to store the variables in
this case I think the variable was I so
we're gonna leave some room and this by
the way threw me off I was like why is
it going negative on the first
instruction because I kept linking it so
that would start at zero and it tried to
go back below zero and bad
so that's why you want to move it to
like 200 and leave yourself a little bit
of room to operate
great well then I'm gonna see like you
know know this stuff we'll have so
here's the 62 that I was just
initializing it to so it's actually
gonna add there is no like move operator
like normally he would just like in an
Intel world you would just move the 62
into the right register they got rid of
that to keep the implementation minimal
on risk 5 so they're just going to take
this 62 and they're gonna stick it into
register 15 then what they're gonna do
is they're gonna store register 15 into
memory location minus 20 from where we
are so that's gonna be up above in that
initialized space and then we're gonna
do our little ad that we just said so I
said add seven to it so I is I plus
seven well that maps to this seven is
going to be added to register 15 and
we're gonna stick the result in register
15 which is fine because we're over
writing I and then we're gonna store
that into memory so there it is so for a
little bit of extra credit fun so I hope
that part made some sense basically this
is the disassembly in pure JavaScript of
this program that we just compiled and
stripped all the other stuff out which
you normally get on an operating system
I could leave the hello Stupak this
would actually sit in a part of memory
which I was just stripping out that's
down below but you know look we'll keep
things simple because we have one last
thing we need to go through which is
just before we got going I said hey
wouldn't it be cool in addition to you
know having the specs so that we can
fully implement this thing and we know
we learned that selfie wasn't going to
kind of get us there what if we took
Linux all of Linux and compiled it with
this tool chain to just be able to fit
into these op codes so this
theoretically is a pretty good torture
test and that's actually what
were kind of sneaking a peek at before
so you basically this is just a recent
Linux I just downloaded and I configured
it for risk five and you know that I'm
gonna here let me just copy out this
kernel because we don't want to watch
the whole thing compiled but yeah we
could actually take this is the minimum
config that you could use to build a
risk 5 kernel so this is basically just
saying all right
architecture is risk 5 and you know
clean up your configs and all that stuff
and we're gonna go with the tiny like
minimal sort of config of Linux so this
has just as little as possible but you
know we'll run Linux so we do that and
this this part actually is not too bad
this this goes pretty quick but there's
a problem
normally I would just go into the menu
config and set like whatever you know
parameters I want they don't work I'm
not exactly sure why but I ended up just
manually this you shouldn't do this at
home but despite what I said it will try
to build a 64-bit version of this kernel
which is not what we want so we're gonna
want this config Arch our V 32 eyes yes
and of course that invalidates a bunch
of other parameters because you know
they they depend on each other and you
know we also are gonna need like you
know a bunch of other stuff which I'm
not going to show you it's basically
these are if you're if you're curious
just for completeness this is what else
I messed around with which was that you
know don't disable compressed
instructions because we want every
instruction to be four bytes and no FPU
so I don't want any of the math
instructions other than just the ads you
know the stuff that's in the base
instruction set so we do all that and
then we go and we actually you know make
this thing and we remember that we have
to specify the architecture I killed
myself on this a bunch of times when I
was doing it but and here I'm just
saying you know do some parallelization
while you do it
and it's gonna ask you a few questions
for things that are now kind of crazy
and I basically just took defaults for
everything else we're just gonna go
default that's where you saw the FPU
support which just shot by sorry about
that and the compressed stuff so you
know do a whole bunch of things here and
it starts compiling except you know of
course you can't do that its current
state so this is like a much longer show
than I really want to get into but
basically after you spend a lot of time
cleaning all that up you end up with
this thing this this version of the
Linux kernel which is actually created
for 32-bit architecture does actually
work on risk 5 and again has an elf
header so we can take this thing and
using that same technique so I'm just
gonna copy that kernel back over here so
let's copy VM Linux over to apps so this
was this is episode 5 because risk 5 and
so now we have it over here and we want
to just pull out you know that one part
and we want to make a binary you know so
to take the VM Linux thing strip it all
out and just end up with this VM Linux
dot bin ok so if I look at this if I
look at VM Linux it recognizes it as an
elf header and it sees all this stuff
but if I look at the binary it doesn't
know what it is because it just shows up
as data there's no there's no file
headers there's nothing it's just raw
code that's running there alright so I
use do my little hack workaround for the
browser copy that in and i downloads
directory pull this up there's my vin
decoder Stupak it was way more time than
I was hoping for for a weekly show I can
tell you that much and you know there's
some zero padding there's some data
stuff down here at the bottom we're not
really worried about that because this
actually works so this decoder is as far
as I got the decoder some of these
parameters are I didn't
decoding all the parameters for the show
like you know I think this one but you
know and then I trapped a few that
weren't showing up so it does take quite
a while to get to one that I think yeah
that unknown those are those are mostly
no offs when you see the unknown but
this this particular trap so that's
that's something that it wants to
implement so this is emitting some code
which is not in that base set that I got
okay so yes do pack the long story short
is the the plan is to open source this
when it's kind of farther you know like
like this where it's at right now we've
got a good chunk of a disassembler
written i can't say how much of it
because I haven't finished and I don't I
won't know until I'm done but the goal
is that when this is at least able to
kind of get through a full Linux kernel
kind of program then I will push this up
so that'll be something that we're going
to come back to in future editions so
that's yeah and we're going to use it by
the way this is going to be the same
host tools that we're going to use to
run on the FPGA later so that'll be fun
later on in the season so anyway sorry
it doesn't fit the normal set of shows
where we get to just write something but
you know this ended up being
ridiculously more complicated that I
thought it was going to pay but I felt
pretty good I mean just case study wise
I think it's I think it's interesting if
you're ever getting into computer
architecture stuff I mean the whole
stream that we're doing is about C you
know and learning C that's so I've got
two students now running on Mondays
there's one that's living Python from
scratch and one who's a Python program
was learning C and she was commenting
that yes he's really giving her a better
sense of what it's like in terms of the
metal but I mean this is really and and
that's true
coming from Python C is basically the
metal but going from there from C this
is literally the metal like I mean these
instructions are what are being written
so it's funny that the tool that's going
to help
teach you know for teaching for her is
written down to that level so I think if
we get a little bit further I'll
probably you know I'll finish the
disassembler first maybe we'll add an
assembler I don't know if that's
necessary because I think it would be
more fun to build an interpreter you
know maybe we'll see if we can actually
run programs on the web and basically
without Azzam and without anything I
don't know how far away it is from here
so I don't wanna make any promises but
this is just what I had in the schedule
at this point yeah this is a different
level I do apologize for that I had a
really nice show originally where I was
like okay now we're gonna like take the
seat thing we're gonna do a some then
we're gonna have selfie compiled and
then it's gonna be on the web that
didn't work the reasons we talked about
before but so I wanted to stick with the
schedule I didn't want to let go this is
I guess the problem with publishing the
schedule ahead of time so for those of
you who are new to the stream I did and
that's the curriculum there but I did
actually show this is this is kind of
the schedule so I was yeah let's not
watch the stream on here while we're
watching the stream so this was this and
when I wrote this like you know which
was back you know five weeks ago I
thought Oh that'll be a piece of cake
using these tools no what did I know so
next week we're gonna have a little more
fun with with controllers with TV stuff
this is really gonna be around like
putting yourself in the headspace of
doing user interfaces with TV controls
so it controls on the TV or minimal it's
kind of like a keyboard I've got the
remote controls kind of like a keyboard
with only like four buttons you know you
know there's all a bunch other buttons
and some remotes but you know like it
you you want to be thinking like
left-right up-down and what you can do
with that now we're gonna take a little
break and have some fun with this we'll
probably more like today's session but
this is gonna be about building a
Nintendo car and I've been discussing
that very briefly with 88 bit music I
don't know if any of you have checked
him out he plays fantastic video game
music performances he's actually going
on later today
for a Friday evening session and hey you
know you can donate some money and he'll
do requests or if it's a slow time or if
you're new you actually always do
requests for new people he won't do it
for me I keep selling him um new but
he's like no and he's got a whole bunch
of like fun little NES game ideas and so
I said hey maybe we'll just do that on
the show you know I mean originally I
was gonna do something different but
yeah that might be what we do and anyway
so you can check this out later this was
like I said quite a bit lower level if
you want to talk more about it then you
know come to the discord and you know we
can chat more there I don't want to go I
think this is already like really dense
so but you know I had to clear you know
the backlog so that we can move forward
and this yes the Stupak your question
this ended up being like way more work
than I thought it was gonna be involved
for a show yes yeah I've actually been
all of the shows from the Friday
sessions I've been putting these on
YouTube so they're actually up here in
my past broadcasts yeah so I'll I'll get
this up here as well so I think it'll be
more interesting when we get to the
future FPGA stuff and we're gonna be
coming back to this but hopefully the
tools open-source by that point and
that's when we get down here so we'll
see we'll see what you all vote on for
viewers for viewers Choice there okay
well thank you everybody for dropping by
and watching thank you for the follows
and the subs it's all been fantastic and
don't be too hard on pianos he's a good
guy and yeah I will see you actually
Monday will be so what I'm doing just
format wise I'm moving the all lessons
are happening on Mondays that's for the
like you know the teaching segments
which is you know the learn segments the
Friday we'll stick to the I'll try to
stick to the hour format for Fridays
kind of dense show but hopefully not
this dense ever again I might revise the
the show
episodes after this because this was too
much material for one show but I do
appreciate you all hanging out and
supporting so I will see you later I
think
hoo-wee