E3 - Compression, your own streaming service

In this session, we look at the Art of Throwing Out Bits aka compression, starting from delta frames, run-length-encoding (RLE), clamping, and colorspace transforms (RGB888 to RGB777/RGB565), before discussing more advanced topics like resampling/scaling and basic color theory using our custom video/audio JSON codec.

This was originally livecoded on Twitch.

Transcript:

all right what is up everybody how are
you doing nightshade dude hello hello
other people watching this is episode 3
of CSG flix we're building a video
streaming service from scratch and for
those of you who missed out and believe
me you missed out last Wednesday session
was awesome the we we basically ended
with a version of the this video
encoding that we built from scratch
which I had video and audio that was
encoded in JSON so it's very accessible
JSON it's something that's incredibly
easy to parse and the kind of thing that
you can probably live code but we're
gonna get to some binary later I'm gonna
I'm gonna mention a little bit about
that as we go on but one of the
constraints I guess that we have with
how I'm building this is I want to make
sure that despite the fact that we're
doing things in JSON it's easy to
inspect it is a format that we're gonna
be able to repack in binary so I'm
sticking to bite sizes and some other
stuff that we'll get to a little bit
later but for now it's JSON I don't
think there's ever been a video encoding
done in JSON so you watched it here
first on twitch great well what do we
have in store for today we have so the
video and audio working we're playing it
in the browser as where we sort of left
off last time and now we are going to
try to compress it so this I title this
the art of throwing away bits so this is
kind of a fun exercise because we want
to try to throw away as many bits as we
can while still retaining a really
high-quality picture and high quality
audio now as you saw in the earlier
stream we already started by reducing
bits in some rather drastic ways just to
get it to be more manageable under this
as we were dealing with but this today
we're gonna talk about some some
compressing techniques and they're gonna
broadly fall into a few categories that
we're gonna cover as we start
whiteboarding so I'm not gonna go too
deep into any one of those like I've
said before any one of the topics that
we're covering in these eight topics to
build a video streaming service from
scratch is something you could study you
can really go down the rabbit hole there
you can do a lot of CS work you could do
ph d--'s you could work in the corporate
world for a while on any one of these
topics so we're we're just trying to
build enough to get a full end-to-end
service but possibly in the future if
there's interest we might start going
down some of those rabbit holes but for
today we're going to try to keep our
compression nice and simple so let's
let's see if that's doable so without
further ado let's let's do that
[Music]
all right I've got my glove on so we are
in episode 3 yeah all right this is and
we have a title here from my pain thank
you
this is CSG flix and I even left it in
the same color as the end of last stream
and hello Stupak how are you doing today
so today we're talking about compression
I'm gonna go very upper case on this
because it's very big topic but it needs
to be a very small topic hahaha okay
okey GaN how are you doing and welcome
to the stream so what are we gonna talk
well compression really falls into two
major categories and that's there's a
lot of techniques that you can use the
very large field but it really broadly
breaks down into techniques that are
lossless and techniques that are lossy
so what's the difference between these
two this reproduces exactly whatever you
compressed so yes we're not doing ml
today this do puck and sure thing
nightshade dude so this basically says
like if I take a file and I compact the
thing and I make a small file I need to
be able to expand it back and get
exactly the same original file so if
this if this said one thing it had
better say exactly the same thing after
its you know in its compressed form so
this this is one set of techniques that
we're going to use now this fortunately
is very well studied so I'm not going to
go too deep into this there's a lot of
lot of information available on it but
it does apply to what we're going to do
today so to get a little bit of a taste
of this we're going to use a relatively
simple version of this which is based on
something called RLE so we're going to
be doing some our early today because
it's I think it's something that we
through relatively quickly and I'll
point you at some of the other
techniques oh yeah and by the way kind
of the Stu pox comments you know
lossless compression when you get
exactly the same thing out that's
similar to something like zip okay so
that would be like zipper gzip or B zip
- or LH arc if you're back in the day or
whatever so there's there's a whole lot
of these but the important thing is that
I end up with exactly the same thing
that I started with so I had even after
compression I need to get back bit exact
version and whatever it is up here and
had better be exactly the same down here
lossy on the other hand is about
throwing away so lossless where word
just make small but exact so we're able
to reproduce from our original all the
way back to our original even if we
throw it into a small intermediate
format lossy on the other hand is about
not exact so this is kind of an
interesting field which is sort of more
of what can we throw away when we
actually go and compress this thing so
I'll pick a different Brown but so we
take our file and we make it really
small and when we expand it again we get
something similar to the original so
it's not exactly the same as the
original but it's something it should be
something fairly close to it and yes -
Stu pox comments this is something
that's more like a JPEG so JPEG would
fit into this category you know we're
gonna use some LSB tricks that I'm going
to go over later
and we're gonna cover some stuff in this
field we're gonna start with again
something relatively simple so we're
going to take the
it's and we're just gonna throw them
away where this is this is all about
having a trash can and yeah that's
that's that's our trash so we want to
take some of these bits throw them away
and we're never gonna get them again
actually they're gone so but we want to
try to get something close to the
original that is as small as we can kind
of get and hey what's up prime a genuine
mp4 dopey okay well yeah this this would
include things like mp3s and mp4s for
this is just straight on the audio side
and this is actually a packing format so
the the equivalent audio would be like
AAC that goes in here a l1 is something
that comes a bit later out of a
different generation but yes the he is
correct this is something that fits into
kind of a glossy gown okay so here we
have to be really careful because we are
losing quality so this is really all
about what quality can we lose that
still maintains the same visual image so
if we if we see our dude and he's
chilling out on the scene then we want
to say like okay maybe we don't need as
much detail over here because it's
mostly black as we do around the
character maybe the face like we want a
lot more detail here where people are
gonna notice it versus like you know
here we're like if this black is like
one shade of black and this black is
like just a slightly different shade of
black like no one's really gonna care
too much because it's we're not we're
not worried about making an exact
reproduction if we were then we would be
using lossless techniques and for images
that would include things like Targa
files and again these this other stuff
which is actually they're all using
versions of Lempel Ziv and Huffman
encoding so we're not gonna go this far
down the rabbit hole so that's let's not
go there and look at that seven people
were there yep
no no no BOTS I think actually might
show in 10 but I've come not to believe
whatever
viewcount hmm okay so what we're gonna
do today is we're we're gonna try to do
some version of a lossless and some
version of it loss leap lossy codecs so
what I want to play with let's let's go
to I don't know what are we feeling a
little more pink so what we're going to
start with is we well I mean again these
fields are really deep and I want to try
to keep the implementation as simple as
possible so that we can you know
reasonably get it done in a reasonable
amount of time so why don't we start off
with a few a few easy things to kind of
warm up so the overall plan that I'm
gonna follow and I'm not sure a plan
should be pink that might be a little
too much is we're gonna first do a lossy
technique so this is lossy and that's
gonna be scaling I like to joke about
this one a lot because you know this is
said this is exactly what you see when
you're watching a movie and they say
enhance enhance enhance and the trick
here is we basically take a large
picture we make it a small picture and
then we blow it back up to being a big
picture so this is like this is actually
a little bit more complicated than it
looks but fortunately since we're
working in the browser it's it's we get
it pretty much for free so I think it's
a pretty good place to start and the way
we're gonna do that is basically we're
just gonna play with the width and the
height so we're gonna say hey browser do
some magic for us and take this image
and resample it down to this size and
then later on we're gonna so this is
this would be in the encode phase and
later on when we're playing it back
let's you know resample it again and
expand it now that's gonna cause things
like you know if I had some really nice
detail here that's gonna turn into like
one block here but when I expand it
again I'm gonna get kind of a
area here so I'm not gonna have the same
visual fidelity that I had in my
original image but I'm gonna get roughly
the same idea and I and I can get it
relatively easy so we're gonna start
there that's actually gonna save us
quite a bit of space one of the metrics
we're gonna watch is sort of like if we
if we had things perfect how could how
much smaller can we make the encoded
file and still keep good quality so our
game is like small file but really good
quality so that's that's what we want to
do if we're if we're doing a good job
all right
so great we're gonna do some lossy scale
stuff we already touched on in the last
episode I'm just gonna quickly go over
it but we basically talked about deltas
and that was that's this is really a
lossless technique and in this we were
taking kind of a we were looking at the
scene and we were saying okay from one
frame to the next
most of the scene doesn't change like
saying in our example we had a guy who
was waving so this this frame is like
his hands just a little bit higher than
it was in this frame but like the vast
majority of this scene is exactly the
same as it was in that scene so we can
actually just take this part and encode
it and that's our Delta and basically
just reapply that and we can end up with
exactly the same image that would have
been there if we hadn't used Delta so
this actually is really really nice then
we're gonna switch gears well we're
gonna do one more and probably our last
lossy technique because sorry lossless
technique which is we're going to go
with our le and I don't know how
familiar all of you are with this this
is run length encoding and this is kind
of like the easiest compression I could
think of so if you think of this like if
we have a series of numbers like we have
one one one you know five five seven two
three then what we do in order to run
length and code as we look at
how long each run of the same thing is
so in this case I have five runs right
here's run one here's run - here's run
three four five and in this case I just
need to say if I want to compress this I
can say well alright take all these ones
count how many there are there's one two
three all right so three ones okay and
then here we have two fives and here we
have one seven and here we have one two
and then one three okay this is the run
length and coded version of this now in
this particular case it's not much
smaller than this was and and we could
be kind of like evil about how we choose
this original but I can tell you that in
a lot of color images especially in the
kind of stuff we're compressing today
these runs are usually a lot of zeroes
it's usually like zero five hundred
times and then some more zeroes and then
maybe we'll get like a 1 over here so
this compresses really nicely because
it's gonna say like okay this is just
500 zeroes and then you know 1 1 and 1 5
so this these two don't save us much
space but this actually saves us a ton
of space and based on images this sort
of thing happens a lot so that's um and
it's I think it's a pretty good taste of
like you know lossless technique that we
can get without going too far into the
implementation never came across it my
studies at work Wow yeah it's it's a
very old technique so it's not like but
you know it's funny how surprisingly
good some of this stuff is all right so
when we're done with all that we're then
gonna do like two little really simple
things which in our case there as it
back to our pink so we're still in
episode 3 and I have a let's go here so
then we're gonna move over to
clamping and clamping is all about like
saying well some colors don't really
matter that much so hey what's up Akshay
how are you doing going well just
getting ready to getting all this stuff
working yeah and and by the way okay
Oh Keegan yes you are correct our Lee
gets better if you pre-process with
Delta so you know there's different
stuff that's actually kind of getting
into what we're going to talk about a
little bit later but actuaries don't
usually use a code yes fair enough
Stupak so clamping is like we're using 8
bits you know per color type so we're
using RGB and this is red green blue and
we're using 8 bits for each of these so
these are 8 bits wide 8 we refer to this
whole format as RGB 8 8 8 which is also
you know more widely known as 24-bit
color right 8 times 3 because we have an
actuary here and Stupak can tell us so
that's great
so with clamping what we're gonna do is
we're gonna kind of like reduce the
range a little bit because we're gonna
say that like colors that are pretty
close to black so black would be zero
zero zero okay or in HTML notation be
zero zero zero zero we can do it is hex
but you know basically the same thing
doesn't really matter they're they're
just the number zero so zero what's the
difference between zero zero zero and
like 1 1 1 and like 17 17 17 not very
much in fact you'd need to have a really
sharp display to be able to see that
difference unless I put them like right
next to each other so what we can do
with clamping is we could basically just
say like if it's less than 20 then just
make it zero so this is really nice
because as row Keegan mentioned
for this stacks really nicely with
run-length encoding so if we have things
which are close to black that aren't
really gonna add much to our end image
then we can actually just turn them into
straight black and then say all right
well get longer runs okay great
then we get that far and clamping turns
out to be really easy by the way and
then just sort it to this example that I
brought up over here we're gonna try
outgoing we're gonna drop a bit we're
gonna try to drop the least important
bit and this is going to create a new
color space I'm going to call RGB 777
and in that space we're gonna take our
RG and B which was a 8 and we're gonna
make just one bit smaller but smaller
and the idea here is that if I chop off
the last bit then I'll end up with 777
for my size but this bit should be like
the difference between two colors that
are really close to each other so the
idea being that we shouldn't really see
much of a difference if we go 777 versus
if we go 8-8 yeah now this is nice
because now instead of using 24 bits per
pixel and by the way we have to multiply
that by the width by the height so
that's a lot of data just to begin with
you know I mean we could just send
everything in black and white and do one
bit encode and then we would save
ourselves a ton of work here but so
actually there you go that's that's the
ultimate streaming service hack it's
just black and white so what we could do
here is we could basically say like all
right if we do 777 we'll end up with
just 21 bits instead of 24 so we save
ourselves 3 bits per pixel and well
that's not really that big a deal
because if I do this in binary this is
going to end up padding out and some
other stuff but there is a more advanced
version of this which is used quite a
bit in in gamer E that was a that was a
crappy arrow so I'm gonna back that err
I'm not happy there so we're gonna yeah
we're gonna get more serious without a
rose and there's a version of this which
is referred to as RGB five six five and
this is kind of nice because it it uses
six bits for green and it uses five for
blue and red and that's that's really
kind of based on some bad artwork are I
and that's because we're more sensitive
to green than we are to red and blue so
if we if we throw if we're gonna keep
some bits we want to keep them on green
and less so on red and blue why do we
choose these numbers five six five well
add them up you get 16 bits and 16 bits
fits into two bytes nicely so we're not
gonna actually cover this today I'm just
gonna like show you this because it's a
lot easier to implement but the the same
technique that we're gonna use applies
here if we just chop off more bits you
know less here and more here and more
here then we'll end up with RGB five six
five okay so that is basically what
we're gonna do we are going to change
colors and we are going to implement all
that stuff we are going to do scale this
is and we're gonna spell - we're gonna
do scale down and up we're going to do
Delta we're just gonna look at it and
we're gonna put those letters in the
right order and we're going to do a RLE
which this is probably going to be most
of the meat of what we're going to talk
about we're gonna clamp which is really
nice and quick but it causes RLE to get
a lot better and we're going to do RGB
seven seven seven which is really more
just kind of a tease for what we could
do with five six five if we wanted to
spend a lot more implementation time
okay
so that's the plan it has a line under
it which means
it's serious okay
let's yeah let's let's let's get the
coating enough talk
chat let me know if there's any
questions about any of that stuff that
kind of came up so we're here we're in
the compression segment and we're trying
to go for max quality with the least
amount of bits so I'm not gonna recycle
this terrible joke over here so all
right this is exactly where I left it
off last time we have our episode 1 or
episode 2 now you can catch those on
Twitch I'm gonna move them over to
youtube later but we're gonna start in
exactly the same place but I want to try
to keep the code like as it was at the
end of the episode for when I check it
in later okay so let's just copy this
over to e3 let's go into our episode 3
and let's we don't need the drawing part
but let's let's go into screen and this
is sort of the code as we left I have
changed nothing if I've changed anything
call me on it because there should be
nothing different from the last time
great so question coming up here is is
compression algo dynamic based on users
TV like if users have crappy LCD versus
o LED in practice the answer is it
really depends on the streaming service
Stupak so a good streaming service the
answer is definitely they are different
but it is a little bit difficult I
shouldn't I shouldn't really just
categorize that as as good and bad it
actually breaks more along live versus
not live like if you if you're doing an
on live streaming service you have a lot
more time to do compressions so you can
be really careful about tuning your
recipe for like different classes of
devices versus if you're doing live
something more like twitch you really
can't do that you know you you the one
TV will look better than the other
mostly because of like single resampling
which is really an advanced topic that I
don't want to get into but the the short
version is like if we built a really
complete in our series we're only
building for one two
which is this browser here we're
building for chromium and that's it but
if we were going to do a really solid
job with this a professional product we
would then go and start dealing with all
of the different devices that are out
there and and not one encode per device
but we would take them in sort of
classes like okay this one has sub-pixel
rendering or this one has you know a
wider color range you know it has HDR or
this one has you know anyway different
resolutions so well with that that's
more of an advanced topic we'll get to
that later after the show if people are
interested in it
okay so what we did here which somewhat
unbelievably worked in a short amount of
time it doesn't surprise me at all that
it works it surprises me that it works
from live coding on Street in a short
amount of time because I've been coding
for too long but just to show you kind
of the state of affairs of where we were
this was our video player which is
currently not showing anything so we're
in a pretty clear green field here
except I guess I should call it a black
field and we were throwing a little
thing it's looking for a fav icon we're
not worried about that okay so this this
was based on having a encode of one
image and so if I wanted and we're doing
one video that was 30 seconds long so if
I want image 123 this is what it looks
like as we left it at the end of the
last session now this is in our JSON
format every one of these corresponds to
red green and blue red green and blue
red green and blue and you'll notice
these really long runs of the same
number which is why I wasn't kidding
we're gonna get a lot of stuff out of
RLE now you'll also notice that all of
this stuff is pretty much black actually
everything that's on this screen so far
is probably beyond our easy visual
perception so all of this once we add
clamping all of this is going to just
turn black and it's gonna run length
encoding to one night
and that's we're gonna stack these
compression techniques as we go okay so
where were we with the deltas that was
so the routine that I just showed you
that was this all we did was we opened
up the image after we you know unrolled
our video into a series of images and we
flattened it so that you know the RGB
was just here otherwise they would be in
sub arrays this would be a sub array
this would be a sub array and we then
just dump it out so this is the simplest
image format I could come up with it has
no header information which is a little
bit problematic because we don't know
what size this is we don't know how many
of these pixels correspond to one line
just looking at this I mean we're gonna
have to try to guess so a normal image
format would have this but it would also
have a little bit of header information
to tell you oh yeah this thing is 240
pixels high and it's 426 pixels across
we're just going to do that by
convention at the moment we're not we're
not too worried about like
commercializing this image format so
we'll just do it by convention over
specification all right so that's what
this did this was a slightly more
advanced version of it which basically
just took a starting image in this case
it always took the image 1 but we would
really want this to be whatever the
image was before this which we pass in
an ID and I'll just call it because it's
simply explaining it and it only is
going to calculate what's different from
image 1 so this was our Delta encode and
it's showing me all of the pixels that
changed since the first frame but I
don't really want all of the pixels that
changed since the first frame that made
a more dramatic example when we looked
at it but really I just want all of the
pixels that change since the image
before it so if I do that you're gonna
see most of these numbers are going to
disappear because each frame in a
sequence changes very very little in
fact there was I just saw a few yeah
there are a few pixels that changed
that's so you know when you're doing 30
frames a second like very little of the
information is changing from one to the
next file a patent supposed to come
after you
nice to pocket okay
so if they're gonna try to patent json
encoded images i I'm worried for
Humanity anyway that aside so great this
works this is very simple all I did was
I just because Python actually it's
simple it's it's harder than other
languages the reason it's so simple is
because I open the thing I open the
image I I take image 1 here I turn it
into an array I open image 2 which is
whatever was specified up here you know
and now I just change it so it's the one
before it too and I turn that into an
array I'm just gonna change this back we
debunked that in the last session I
turned that into an array and then what
I did was I XOR the two so I just want
the bits that need to move to go from
one to the other and that's nice because
XOR you can just keep XOR in its item
potent basically so if I keep XOR I'll
end up with the same thing if I take
something that I XOR and then I XOR it
again with the same thing I end up with
the original thing so this is lossless
like I said I can I can reconstruct
exactly the original image from it - the
the fact that JSON could play with the
formatting a little bit and you know a
space could be added here and it would
still technically be valid JSON but all
right for the pedantic people I'm on to
you ok so that's kind of what we did at
the end of that session so if you want
to see those implemented I highly
suggest checking out episode 2 but as
you can already see this is part of why
we're gonna we're so interested in run
length encoding you see all these
beautiful zeroes thousands of zeroes
that's gonna turn into just two numbers
how many zeroes and the number 0 if we
run length encoding so we're gonna get
quite a lot out of that okay
well that brings us to our next
topic which is run length encoding so
this stuff was you know nice and easy um
there's one little NIT I should probably
mention which is that right now this is
just an image format so this JSON that's
coming down it's just an image so to be
a video I have to just send a series of
images right now I'm sort of requesting
him one at a time I thought that was
simpler to see but if we wanted to
actually make this a full video format
we would have to add a little bit of
extra stuff to say like this is a video
these are the images and this is either
a delta frame or it's a regular image
frame so that our program in JavaScript
would know what to do with it do I need
to apply it to the previous image or do
I need to just reconstruct it from
scratch so there's a little bit of
metadata I'm sort of skipping over but
you know we might get to that if we
clean it up in the end okay so another
quick question so will this only send an
update when the image changes like if
JSON was all zeros would you need to
send that update or not if it was that
that would be another metadata item
basically we would need to add a 0 frame
to say there's a frame here but there's
nothing in it and then that would tell
the client like ok we need to do that
and by the way we're live encoding this
right now so like we're encoding it as
the clients playing it which is actually
more of a live thing in our service in
CSG flex we're gonna pre encode so we
already know what the video is gonna be
so we can actually encode it ahead of
time and we could just put the encoded
file on the server so we don't need to
do any of this logic life I'm just
showing it to your life just for
simplicity we can we can encode ahead of
time ok so how do we run length and code
well it turns out we need to build a
routine to do so we need to have a run
length and code and code routine and
what that's gonna do is it's gonna take
an array and
that array I'm just gonna make this
valid Python so that I don't kill my
server which is still running it's gonna
take this array and it's basically just
gonna run through and it's gonna count
how many zeros are there on - or well
it's gonna it's gonna say alright this
is it's gonna take the first element
which in this case is zero and then it's
gonna say how many other elements are
there that are exactly the same okay so
it needs to basically kind of scan ahead
and say you know where is that now
because I chose Python I get to use
Python so we're gonna do this the
incredibly simple way which is we're
going to we're going to use built in
stuff that's in Python so just to show
you this is straight Python I'm not
using anything in the virtual
environment so let's say that we had an
example image which had like seventeen
seventeen seventeen logical common
seventeen seventeen 1700s zero so the
first one is slightly black the first
pixel slightly black the second pixel is
totally black the third pixel is totally
black the fourth pixel has maybe some
ones all right so this is gonna be just
a little piece of what our image
encoding looks like so if we run length
and code this thing we want to end up
with three 17s right so that's you go
three 17s then you do six zeros and then
you do three ones so that would be the
run length and coded version of this and
by the way that run length encoding is
not really a standard so I could put the
number of repetitions and this is the
symbol I can put the symbol first and
the number of repetitions just adopt one
convention done for your product don't
don't don't flip between the two I think
I think the more classic version is you
put the number of repeats and then you
put the symbol so I'm going to use that
for today okay so if we build this
routine correctly this is our output we
go from this thing as our source to this
thing as a
output and then we've got it that's our
run-length encoding we're gonna need to
run length decode it in JavaScript so
we're not going to worry about that in
Python okay well turns out Python has
iteration tools which help with a lot of
things and they're built right into the
base Python library and yes I am
pitching you straight JavaScript devs to
also take have a little taste of Python
but let me show you why sometimes it's
cool
so from Eider tools let's import route
by and group by is intended to work
exactly the way you would think with a
database except we're gonna miss use a
little bit we're gonna say I want to do
a group by and we can by the way we can
look at what group by does group by
takes an iterable in our case it's gonna
be this array and a key that's up to you
we're not we're not gonna worry about
that and what it does is it will return
something which has keys and groups and
keys in this case are what I'm referring
to as symbols and groups is how many are
in that which is basically free run
length encoding if we're careful so why
don't we group by a and it's gonna pop
out an iteration object which I can just
cast and I'll end up with this is what
it looks like well I have 17 of this
iteration object then I have this
iteration object 0 so that this is sorry
it's not 17 that this is simple 17 and
it's got this many and then it's got
symbols 0 and it's got this many and
it's got symbol one and it's got this
many so the only real issue is I just
got to get rid of these group group or
objects I just have to turn them into
like how many are there and if I look at
one of them let's let's let's just break
this apart let's say for entry actually
I can say like you know name you know
the name of the symbol and the
the gloop thing this this grouper object
in group by of a prince named which we
already know is like 1701 and also take
this group or object which is another
iteration thing to say
just turn it into a list list of Wow
look at all that work it does for us we
get you we have 17 there's three 17s we
have zero there's six zeroes we have one
there's three ones which is if we look
up exactly what we want so there it is
that that is like run-length encoding in
like three statements guy how did you
know the only the only thing that we
have to kind of watch out for is we just
have to say this this is actually an
array of the number of entries so I
don't I don't want an array of it I just
want how many are in that array like
just just tell me like there's three 17s
there's six zeroes there's three ones
and then now I've got kind of the right
thing and if I reverse that order like I
just said like so that we put the repeat
and then we put the what the what the
thing is we get exactly what we
specified before which is we want three
seventeen six zeroes three ones done
great see you all later stream over so
all we have to do is just take this
thing and wrap it up into our endpoint
okay so let's do that let's I just
copied exactly what was in there except
it's not called a here it's called array
I know I'm not naming things very well
and hey what's up cowboy
how you doing so easy with Python
honestly though how long would it take
to make this in C++
Stupak if we get through this series
I'll coat it for you and C++ one find
out I mean it's not run length encoding
is not terribly hard but it is something
that like without this operator III
implemented it the other night and it
was like seventeen lines that was still
in Python in C++ with a little bit more
memory it's probably a little bit longer
than that but I mean it's not like
thousands of lines I mean it's like max
it's probably like 30 lines so but
anyway yeah that's let's do it so we're
gonna do the RLE of the array so at the
moment let's just to stay simple let's
start with the full image here and so
this is a full original image so we want
to see like three 17s and then three
eights and then three nines and then
well some sixes and a whole bunch of
nines so let's let's see if we can get
that out there okay so how do we do that
well this is the routine as it is so
this just gives us the regular array so
I can just pass this to our alien code
and we should be alright so let's our
alien code I a this needs to be turned
into this is a numpy list so we just
need to turn it into a regular Python
list and then let's just wipe out the
result and this is no longer valid
because now we're not returning that
thing we're just going to return the
result itself okay
so we do that and GroupWise not defined
because I forgot to import it so I need
to import from iter tools import by and
that and we get no that's nice there's
programming except because I printed it
out didn't I I don't really want to
print the thing out we need to have a
result here so let's yeah let's let's
get it let's get a result so I could
actually take all of this stuff and
a pendant you know that would be like
you know the really non pythonic way to
do this but if I just make this an array
and I say result dot pen this this stuff
here which would be the I want the
number which is the length of the list
of the group and the name which is
whatever symbol it happened to be and
then I return this result well we do
that and a pen takes one but I'm
determined to yeah because this is this
is actually a sub array here so I'm just
gonna append a little sub array there we
go alright so it's 3 17 s 3 eights 3 9 3
6 is 612 9s so there we go that's our
run length encoding now the only thing I
don't like about this is we're including
all of these as sub arrays so let's
let's not do that you know let's be good
let's be good to board and not pack our
JSON with unnecessary stuff so the way
we could do that is basically just split
this into two statements is the simplest
way to do it I can think of the better
way to do this is to use a 4 in is to
use a comprehension but I don't want to
be too pythonic on this stream so let's
let's do it kind of something that's
friendly to JavaScript Devon okay so
yeah hey what's up korean-american
barbecue yeah streaming service from
scratch mr. demon wolf so this is our
image format that's now run length
encoding done 609 s 1266 0 that's that's
it if I want to be a little more
pythonic about this I could you know now
clean it up later if we care ok so now
our next question is how do we know if
it's right right so before now I wasn't
actually showing this thing displayed so
let's let's get that image displayed
over here so let's let's split the
screen I'm gonna put the JavaScript back
on for everybody the JavaScript devs be
happy so here's the JavaScript we wrote
in the last stream just straight boring
JavaScript and we wrote this function
show image rate and all it really did we
drew on an off screen
don't worry about the off screen images
basically we're gonna draw on this black
thing this black canvas and we expanded
the format that we got because
unfortunately JavaScript uses
red-green-blue alpha and our alpha value
we just make it the maximum value 255
and it'll end up being a basically we
want the pixel to show up we're not
doing any fading effects we've we've pre
computed the values we want so we don't
need to actually send alpha or I could
just send 255 for every single pixel and
so this is basically the part that
displays it so this should be the only
thing that I really need to change and I
just have to basically account for runs
of numbers in that part so Spartan
greater no no more Joey we're in the
browser we got to be JavaScript I mean I
I actually did him scripting a little
bit on the first dream and I got a
little feedback like don't don't do C
please some guy it's so I'm gonna do
some Java scripts the best thing ever
made but it's the thing that we're gonna
use because that's what's generally
accepted in the browser yes yeah yeah no
trolling mr. demon well yeah why don't
you see how many lines you could do run
length encoding in JavaScript we'll talk
I mean I really like having this like
group by you know this does almost all
the work and I can actually turn this
into a comprehension do it in one line
so it's really a one-liner okay getting
a little far afield so basically I'm
just going to snag this thing but well
you know what I don't I don't want to
change it let's not change it in the
display so for right now let's just get
our image as it was so this is our
normal image
function so here I'm calling this is not
the normal image function so let's just
put this back to normal for a moment so
let's let's get rid of the run length
and coded version we run length encoding
but we're throwing it out so here's our
normal image and by the way a good tip
start from a working version before you
get crazy okay and I'm just going to
uncomment this image display that we did
in the last session so here this is for
showing Delta images we're not going to
worry about Delta at the moment so we've
downloaded image a like some particular
image we can make this one two three so
that we works on like we were looking at
and let's just display that so show
image array of this thing it's just we
we extended it so that it had to be a
context as well so this is you have to
specify which context to draw it on so
that's basically I want to draw it on
the top one which is called CTX so let's
show image array on CTX of this image
that got returned from the server all
right so we do that and we get our image
back that's what it should look like
okay so why do you draw off-screen I
covered that in the last session Korean
American barbecue but basically the idea
is that originally we started by drawing
on the screen and as we're changing
frames in our video sometimes the video
plays the part that we're drawing on
that's called racing the beam so where
we're drawing and it's displaying and
then that ends up with visual tearing so
we end up like you know kind of screwing
our video up a little bit so the better
thing to do is you draw off-screen get
it alright and then pull the whole thing
over real fast you know in in underlying
platforms you would actually just switch
you would switch buffers and say like ok
use video memory draw from here video
memory draw from here so that would be
double buffering
you could triple buffer depends on my
video memory yeah you catch up on the
other one if you're if you're curious
about that
what's your views on Microsoft I'm not
gonna answer that Unicorn glorious TV
that's Microsoft is Microsoft that's
I'll be tautological about it all right
so we're gonna ship we're showing our
image that's all cool so now we want to
display the run-length and coded
versions of that so let's build a quick
little function which is going to
basically this is displaying this format
so here we're actually we have it all
unrolled so if there's three 17s 3/8
we're actually displaying all of them
but we really want to switch over to the
run length encoding version that we just
created and which looks more like this
so there's three seventeen is there's
3/8 this is quite a bit smaller if we
look at it and by the way just to just
to give you a sense of how much smaller
let me I'll just hit this so you can you
can see we're gonna take this thing and
we're gonna send the result to word
count which is going to show us that
this is 133 K this is the curl output
but it matches with what we see there's
136 thousand bytes in the output that's
the run length encoding version if I do
the version that we had before we run
length encoding Wow look at that
we went from 930 1000 bytes to 136
thousand bytes just by doing like two
lines of Python so this is despite the
fact that the the technique is very
simple in the compression world it's
actually amazingly effective so we went
from a 909 K image size to a hundred and
33 K image size now I know it's JSON so
I'm not too worried about how many how
much K it is yet but that is something
that we could get to a little bit later
o rate private Jenai didn't notice you
raided thank you very much for the raid
I appreciate that I thought you were
here I didn't man I got
get a notification ah twitch noob and
Fred thank you for the follow I don't
even know if prime engines still here
I'll have to tell him later no platform
I used to host my own stream uses index
m3u8 this kind of yeah that's uh that's
an MPEG format so we're the the UA refer
what we'll get to that a little bit
later creates a buffer on the client
downloads and then downloads the yeah
that's exactly that's that's getting
into something a little bit different
but related in that case we're drawing
on the screen we're racing how fast the
screen is drawing but we could also
download in the background like the next
few frames and have them ready to go and
then we could actually build better
timing and better synchronization so
that's but the two concepts are
definitely related yeah he raided
because there were somehow seven people
plots viewing his offline channel wow he
is really popular but i Stupak I will
definitely thank him a little bit later
or if one of you is in his discord
please message him that he's wonderful
and thank you
so great let's yeah let's let's keep
going so we're gonna build our run
length decoder which is going to be Java
so we're coding in Python and we want to
decode it in in JavaScript we know that
we're saving ourselves like nine times
the file size I mean we're gone from
like 130 909 cade 133 okay that's it's
almost 10 times savings just from that
one change so yeah I know big streamers
even when it's offline they have like
five to ten thousand people just look
yeah I mean I'm guessing people just
leave their browsers open or maybe they
go oh wow he's he's broadcasting again
prime again I didn't know you were
streaming wait a minute who's this guy
so I yeah I don't know when any of you
figure out twitch let me know cuz I'd
love to know okay and because we're
doing video encoding I thought I'd be
coffee cup appropriate today
just saying okay
so let's get back to the fun so here
we're gonna pass an array in and our
goal is going to be to decode this
format so when we see three 17s turn it
into 3:17 so in this case we have sort
of like something that looks like an
array of three seventeen three eights
and on and we want to turn that into an
array that looks like seventeen
seventeen seventeen because there's
three of them eight eight eight and so
on so this is our goal is to turn this
into that I could make that more
visually appealing with an arrow so this
is the goal of our RLE to code okay like
me I have a BA excuse me you're very
intelligent bot mr. demon wolf slug plex
media server yeah you know a lot of a
lot of Xbox people went over to Plex so
yep same cup of coffee everybody yeah
yeah you're definitely a troll ba well
troll me if I get here let's let's see
how this this goes so what I'm gonna do
is I'm gonna build a result and we need
to basically create this array out of
this array here so basically let's just
run through it so let's I'm gonna use a
regular for loop iterator for this so I
is less than the array length and
normally we would just do like I gets I
plus one except today we're gonna
consume these in chunks of two because
we want to take the number of
repetitions and whatever the symbol is
at the same time so we're gonna jump up
by units of two we're gonna just assume
that we're being passed a valid array I
mean it you would actually have to check
like
is the thing even you know whatever but
doesn't matter we're gonna we're gonna
assume we're passing ourselves good data
real streaming service would need to
okay so what we're gonna do is we're
basically gonna say if we if we pull a
number down so our first basically
number of repetitions number of rent
that's not a good variable name let's
just say like repetition count naming is
the hard part of any program so let's
say that the repetition count is and
here I'm actually passing along an array
so I'm gonna say it's array sub it's the
first entry we get the repetition and
then we're gonna get the symbol which is
going to be array sub I plus 1 and so
we're gonna read in like 3 and then I'm
gonna call that repetition count we're
gonna read in the 17 we're gonna call
that the symbol and we're just gonna
basically stick that many into result
I'm gonna do is very naively which is
we're gonna build another counter called
J we're gonna say J is less than
repetition count and increment J then
we're gonna say results dot push you
know just just add to the end of the
array I know this is horrible on the
memory allocator right you know you
would pre allocate this with an array
buffer if you were doing the right thing
or you would actually just build the
image if you wanted to really perform it
so we're just going to push the symbol
so this little inner loop is going to
run as many times as there's repetitions
and each time it's just going to push
the symbol onto the end of the array and
we're just going to return that
so count the number of prior counts
yeah result that yeah this the Stupak
stop showing off the Python I know this
would be a really simple in Python we're
doing JavaScript now we're gonna just
unroll this thing into just the classic
food we could get fancy and use like
array you know I don't wanna get fancy
fancies later okay so theoretically if
we just take I 1 and now we call our LED
code on it I could actually just pass
that right here I could say our LED code
I Wow we're gonna get some crazy stuff
now we're getting into some fun video
things okay so this yeah that's the good
stuff okay so I think yeah I I'm
pretending that the original image file
is run length encoding even though it's
not so let's let's fix that hey look at
that
I really liked that was good a very
artistic I really you know what we
should really Paul it moments like that
like who thinks it's gonna work I'll
I'll get better with the polling but
yeah okay so well let's get rid of that
craziness that we just showed up it is
kind of pretty maybe I'll make that my
icon alright so we run length encoding
this thing done okay well let's go look
at our plan I mean that's that's in a
nutshell that already cut down our size
like factor of 9 that was a that's a
pretty big deal I mean for compression
now of course this is what we would call
very compressible
anyway it's I'm happy with it's just a
couple lines of code all right so where
were we we just did we looked at Delta
very briefly the last session scary I'm
gonna play with it a moment so we just
did our run length and code and let's do
a little clamping and clamping is fun
because most of these values that we're
looking at there and I'm just gonna go
back to the non run length encoding
version of this for a second most of
them in this case these are all pretty
much black which if I look at the actual
image I mean it's actually it should be
black these are all related to
compression artifacts from mpeg-4 so
they're not actually real and in our
case I'm gonna say that anything less
than I don't know 20 is just gonna be
black let's just clamp it to black
because all of these things I don't know
how it's showing up on your screen but I
mean they're pretty much showing up as
black on my screen anyway this is we're
looking at the first row that's that's
black this rows black that I mean we
don't get to any like business until
around here but again different types of
shots are going to encode differently
but for now we're staying with this
simple example I have the rights to my
own video so I can I can actually use
this on the street all right well let's
um just before we return this list right
now we're sort of we're run length
encoding the original list but before we
do that let's clamp it and I'm going to
use a little bit of Python beautiful
magic to do this because I can clamp in
one line in Python which is I'm gonna
say I'm gonna use numpy for this in the
image 1 array at every single location
in that image at every offset where the
value is less than 20 I could do less
than or equal to 20
then just set the value to zero so if I
look at the non run length and coded
version of this which is what's
currently displaying this should set all
of this stuff to zero okay one line of
Python exceed and now these values these
kind of stand out this is probably where
I'm drawing the CSG flix so this stuff
mostly 0 so what do you think this is
going to do to our run length and code
so right now our file is pretty much the
same size I mean these zeros are
slightly shorter than 255 of their
strings but what do you think this is
gonna do when we run length encoding
like I said this stuff stacks well it
turns out we're already run length
encoding it all we have to do is just
return the run length encoding version
Wow eight thousand three hundred and
three zeros followed by 125 followed by
two zeros followed by one so that and
you can tell by the scroll bar that
there's a lot less data but let's let's
be quantitative about it let's let's
let's go and look 59 K and even curl
switched over to showing bytes at this
point because now we've gone from 909 K
to 133 K with the run length encoding
but once we added clamping we're down to
59 K per image so that just got pretty
good so like I said before these
compression techniques they stack a lot
of them some of them don't stack like
you can't run length encoding on length
encoding it probably isn't going to work
but you can clamp and then run length
encoding that in the other way if I run
length encoding clamped that would be
crazy hey what's up Marco we're doing a
little luck we're doing some compression
today and I just a little python magic
so if I go and I reload this page
I can't almost exactly the same image
but now I'm literally from 900 K down to
59 it's amazing okay so clamping turns
out to be really good
like I said it would be nice and easy so
I thought that would be sort of fun okay
well can we do a little bit better I
mentioned one other little trick that we
could use which is extension to others
if we go back to our whiteboard so this
we we just implemented clamping and that
was sort of dirt simple because Python
but the idea was really like you know if
hidden my pen but you know the idea here
on clamping was if you know if it's if
like you know a particular pixel is less
than you know 20 I think I did less than
or equal to then just set it just make
it zero so that was that was our basic
clamp and and it doesn't really matter
if you do this in Python or the
javascript or whatever it's still a very
quick to implement kind of thing and it
stacks really nicely with run length
encoding because once once you've
clamped most of the values that are
close to to black then you know you're
gonna get much longer runs of numbers
especially around zeros okay so let's
take a look at this one this one's a
little bit a little bit more fun so not
that although I really like how easy the
other ones were and how well they worked
so in the case of RGB 777 we have and
I'm gonna be really nice and colorful
for this so we have our red bits we have
our green bits and we have our blue bits
okay and in this case we had 8 bits for
each so this is 8 bits wide from here to
here okay now if we're going to throw
some of these bits out which bits are we
gonna throw out
the it's sort of like if we were looking
at a number like if we looked at like
two hundred and fifty six thousand seven
hundred and twenty four yeah there were
two hundred fifty six thousand seven
hundred and twenty four votes on a
particular issue you know nobody really
needs to know this in fact they probably
don't need to know this and they're
probably okay if you just sort of round
that so if you just said there were two
hundred and fifty thousand votes
approximately like that is really kind
of good enough for that type of
measurement so if if we we can apply the
same sort of thing over here that the
these numbers that are sort of on the
left they're the most important and to
borrow computer science terminology we
call that the most significant and
that's kind of important because they're
the ones that affect the overall value
of the number the most as opposed to
these numbers over here which are also
called the least significant and by the
way bad that I used voting there because
your vote matters but let's let's change
this a number of polls I've run in my
stream okay well these numbers they
don't matter as much and when we're
talking with each other we we sort of
automatically compressed that anyway so
we refer to these as the least
significant or least important in
terminology sort of more English terms
or in more computer science terms more
elite least significant now in this case
this these are all digits right so this
is the most significant digit and this
is the least significant digit but we're
not using digits here these are not zero
to ten these are well zero nine
these aren't zero to nine these are
either zero or one and we refer to a
binary digit
fuckin spell we refer to a binary digit
as a bit in case you were ever wondering
what that came from so we have again
laid out exactly the same we have a most
significant bit and we have a least
significant bit and just corresponds
directly to this most significant digit
least significant digit except we're
using binary for it so in our example
that's over here so basically we want to
drop this stuff we want it we want to
take these and put them in the great bit
bucket in the sky so sorry bits you have
to go we're gonna get rid of you and the
way we're gonna do that is we're gonna
take all of these values and we're gonna
shift them over one take all of these
values shift them over all these values
shift them over and then what we'll do
is that'll be our compressed step and
then we're going to decode it with our
decompressed step by shifting them back
so here we were shifting stuff over to
the right and here we're gonna be
shifting stuff back to left that
corresponds to the binary operators
shift left shift right and shift left so
great nice and quick
it's circled bit confirmed yeah I can't
tell you which I they chose there by the
way okay so this is this should be nice
and simple as well to implement
hopefully so in our case what we really
want to do is coffee here okay so in our
case we have this array kind of in the
wrong format like right now we just have
a flat array which is what we want in
order to be able to send but we've lost
all of our information by the time we've
run length encoding and
clamped it so I'm gonna this is our
clamp step here and this is our our le
step here so I'm gonna just comment both
of those out so we're not going to
return that back we're gonna go back to
our normal like we haven't messed with
it yet
image dozen JSON okay well what we want
to do is for each of these values we
want to shift it over so the easy way to
do that is just shift it over the let's
let's let's do that I haven't tried that
might actually work so in this case I
want to take the I 1a and I need to I'm
gonna turn this into just a normal well
it'll turn itself into a normal Python
array so if I say for every value in I
won a then I'll give you a little bit
pythonic about this then take X and
shift it over one bit okay so if I take
that and I make that I'm gonna turn that
back into a numpy array because it's
gonna turn it into a normal Python array
at that point then I should be able to
just reassign this and the rest of my
code should work fine
there you go
so those 17s if you shift them over one
bit they become eight yeah that makes
sense because 8 is equal to yeah that's
that's a 1 into 7 bit and 1 in the 1 bit
in a 0 bit so that yeah I think I think
that's good word yeah so I'm doing a
little extra just to turn it back into a
numpy array and the reason I'm sorry
numpy array and the reason I want to do
that is because I wanted this operator
to still work and I want this operator
to still work so if I don't change the
fact that it's a numpy array then I
don't have to reek
anything but there you go now you'll
notice our run got even longer because
things that were kind of closed now
actually ended up translating themselves
over we're also working with we have
less range because before we had a two
you know we had an 8-bit range is 0 to
255 but now we're dealing with a 7-bit
range so we only have from 0 to 127 so
the numbers are all going to be smaller
by nature but it's just slightly less
bit so that's our RGB 777 and just to
make sure that that worked all we need
to do in JavaScript is we need to shift
it back so where's the best place to do
that here I'm just doing an RL e to code
you can't recover the clamp because the
clamping was lossy but the RL e was lost
less so in this case this is another
lossy transform why don't we work on the
are led coded version of this array so
let's just our lead to code I won and
let's we'll create a new function here
which is going to be called RGB 777 to
actually you know what I know the
simplest place I could do this is
actually I could just change it in the
show image
so when show image comes up it actually
looks at every single pixel already so
all I have to do is just take this thing
and shift it back over one
and I get exactly the same thing now
this is one bit less of fidelity on
every single color value so now I'm
using 21 bit color instead of 24 bit
color so I'm saving myself 3 bits per
well yeah I'm saving myself 3 bits per
pixel Wow man just I mean of course I
knew it was going to work but if I if I
by the way did not shift it back over
you'll notice it looks a little bit more
faded out and that's because I've now
pushed it
I pushed numbers that were from 0 to 255
I've compressed them to 0 to 127 when I
expand them again I'm never gonna get
that original bit back I don't know if
it was a 0 or a 1 but I'm gonna get it
to be in the right size range so if I
display it now it all just shows up as
sort of faded out but if I expand it
back then yeah there you go and what did
we do to the amount of bytes that we
passed on the wire what we did we
probably didn't change it very much
actually we did Wow
yeah because the numbers are all smaller
in JSON but so now we're down to 46 K
per image frame and that's nice but
what's really like I told you before
that was using we went from RGB 88 to
RGB 7 7 7 7 bits per for each component
but if we go even further to RGB 5 6 5
then when I turn this into a binary
format this is gonna fit into 2 bytes
which is really nice because that's 5
plus 6 11 Plus 5 is you know 16 so
that'll I'll take my 24 bits and I'll
turn it into 16 bits I'm saving myself a
lot of data being transferred so this
565 like I said is it's better but it's
not something that word we're gonna do
today okay
well there's really only one thing but
left that we I
said we would touch on and we didn't
which was scaling and in the case of
scaling and I don't want to push that
too far but I don't I don't want to
actually implement the scaling which is
actually a case of resampling which gets
into frequency transforms instead I'm
gonna do the really simple version of it
which is I'm going to let the browser do
the work for me so I'm gonna keep my
off-screen canvas kind of the same so
it's still gonna keep the same size
which is our off-screen canvas is
defined as 426 by 240 so that was
defined over here and when I draw it on
to the real canvas we could just size
that element up and I haven't tried this
so let's let's see how that works good
XML comment this out and let's yeah
let's figure this out so this would be
240 so that would actually become 480
and 426 I don't want to do any math 852
ok so if I do that okay I'm still
drawing into the bits that are on this
size so that actually won't do the job
for me because I'm actually bleeding the
thing directly but what I could do is at
this stage I could actually resize it I
could say like okay just just blow this
up for me and we'll just let the Dom do
the work so in this case I you know if I
really want to cheat you know so that me
do it today we could actually just say
like let's let's take the canvas element
which in this case yes yeah I built this
to use it for different yeah what's not
you know it's just it's gonna be crazy
I'll leave it up to you Chelsea if you
want to see it or not
we'll come back to that if you do want
to see it but basically the idea is that
we're just gonna draw this thing and
then we're gonna
like okay right now we're sort of bit
copying the whole thing over this is
doing a put image data so I could just
resize it right there if I wanted to and
I could just say you know use this
contacts we could get the underlying I
think I left that in global scope yeah
this is the off context and we could
take the we would need to get the canvas
back and with the canvas we would be
able to transform the width and the
height back so so I could take this it's
corresponding canvas and its width that
I could turn it back into 852 and this
thing and I could turn it back into 480
and I think that's the cheapest way
we're gonna get scaling to happen oh
yeah all right whatever
we'd have to I'd actually have to like
go and do this the right way
we we don't need to actually implement
the scale itself I can tell you that
what we what we could do is if we took
that image data there's actually a dot
there's a there's a routine in the
canvas for the scale which is basically
you just have to specify the hole size
then if it notices that it's bigger
it'll actually do the scale for you in
Hardware usually but so no no it's let's
let's not get into it's not really fun
oh did I forget to put Python in there
Stupak I yes that's what I get all right
well that's pretty neat this - this
little part here this we're not that
worried about this yeah that works
I mean I'm gonna I'm gonna toss that 480
out because you know we we're done with
that for you basically the the moral the
story is that at the other side I could
just
so I just have to be careful that I'm
not directly drawing on the canvas which
you know in my case of my image I think
I am so we can probably get fancy with
like oh yeah see I'm gonna end up going
down this rabbit hole if I'm not careful
which if they look at the MD are you
sure you have a routine on there that'll
kind of do our okay yeah you do have a
heightened it with and that's because oh
so this actually would be changing it
I'll do bug it later that's probably not
interesting or you could just transfer
it to an image bitmap and then use that
when you draw it on that's fine we could
do all sorts of sins in the browser if
we want to okay so that's let's see
scaling Delta's RLE clamp RGB 7 7 7
that's kind of it that's that already
compressed this down to a pretty good
size so I guess the question really from
there is if you wanted a more advanced
compression for example like the ones
that you referred to before in that mr.
demon wolf mentioned or some of the
others that came up in the channel it's
not it's actually using these kinds of
tricks so the first is you wouldn't use
run length encoding run length encoding
you know like this it worked really well
and I think it's relatively simple to
implement so I wanted to you know not
get too bogged down and building a
compression thing but we could actually
use something that uses the deflate
algorithm and deflate is this is based
on Huffman mumbles if it's actually
covered in RFC 1951 remembered correctly
yep deflate compressed data so you
actually can this is now off patent and
this is similar technique except it's
also going to add a Huffman code which
is a different technique but
it's going to end up building a codebook
and looking for longer runs so in our
case we were just basically saying how
often does this symbol happen like and
that's it and as soon as the run gets
broken up that's it that's as far as it
can compress here we're gonna kind of
look a little bit deeper we're gonna say
okay is there is how often does this
pattern repeat and reference it like so
build like a codebook like okay here's a
pattern alright here's the same pattern
a little bit later on wow that's just
pattern one so just put pattern one in
and then later on you could say okay
that's actually just pattern one again
so use that or that's another pattern
that happens a lot that's pattern too so
put pattern two in yours that's like I'm
slightly more advanced version of kind
of like this run length encoding and
that works better and that's basically
the deflate algorithm is what zip and
gzip they all use this B zip to uses a
slightly different version of it's it's
kind of like they're using the 77 stand
or the 78 standard but you can you can
read about that stuff later one of the
nice things about Z lib is it's already
implemented in JavaScript so if we
wanted to pull in a Z Lib this is a
pretty fast z Lib that's in pure
JavaScript if we don't want to count on
the browser at all we could actually
just pull this module in and add it to
our program and then we could actually
just Z lab is already built into Python
so I could just import I can import the
compression stuff and we could get even
better than what we're doing now but I'm
already kind of happy with 46 K and
Delta's are gonna be really small using
these techniques that we just saw so I
think that's a that's probably a pretty
good place to stop so I do want to say
that the best thing we can use better
lossless technique sort of like What's
in Z Lib we can play with bits we can
obviously stop packing it in JSON we can
actually create binary that'll give us a
huge savings right there we can base our
deltas
right now our deltas are based on the
frame before but if we allow that
to go further we could just find any
frame in a reasonable distance that's
similar to the frame that's there and
then just encode that so if we mark in
the start of that frame like okay
don't use the frame before use the frame
3 frames ago the difference is from that
then we could optimize if we if we do
more in encoding we could optimize like
the minimum Delta that we need to pass
and that's basically how MP you know
that's that's an MP for it or not sure
if we allow that to go that's that's
switching to that's called a P frame if
I go the other dough either way forward
or backward because who cares about time
traveling no problem then that's called
a B frame you know and basically I can
say yeah just build the Delta off those
other things now we can also use
frequency so what we can do is we can
take we've been working in the the time
domain we can transform this to the
frequency domain and we could take
blocks and say well the average color in
this block is this and so just think of
that that's another way of sort of doing
a lossy transform I could also you know
I could choose my regions carefully like
I use a larger region based you know a
variable size region I could probably
get some savings there wavelet encoding
uses that kind of trick or i can also
say that most video images they also
they don't just Delta like Delta was
literally a pixel by pixel comparison
but I could say that like for example in
a fast moving shot where the camera is
turning like if we're doing like a car
chase scene or something like that then
most of this image is the same as this
image all I had to do was just kind of
like move everything over a little bit
and then add the part that's missing so
that's that's a motion technique so we
could do motion techniques if we want to
get more advanced and that's basically
modern compression I mean it's it's a
very difficult field but the basics
really aren't you know like it makes
sense it came out of stuff that makes it
okay how come encodings a little
complicated if you haven't seen it
before but so I'd like to thank
everybody who dropped by today for
building a streaming service from
scratch now if we if we what I'm gonna
do off stream probably for the next time
is I'm gonna actually just pack these
things up into a single file like right
now I have a single image but if I just
put these into if I take this whole
array here and I put it if I nest it
inside a bigger array then I could say
like okay this is an image frame this is
a delta frame this is an audio frame you
know I could just mark that if I if I
just pack this JSON a little bit more so
I'll do that it's it's boring to watch
so I'll do that off camera before our
next stream but it's not gonna do
anything substantial there I'll show you
that at the start probably of the next
stream and yeah come back next time
we're gonna be doing UI I mean now that
we know how to build a video format in
JSON we got to pick out what we're gonna
watch and so I'm gonna take a few of
these clips and we're gonna we're gonna
build some UI with the kind of
constraints that you have to deal with
on a real TV we're gonna pretend our web
browser is a TV and we're gonna use a TV
remote control device of some sort I'm
not gonna mention what device we're
gonna use now we're not actually gonna
use the power glove will probably use
just the keys to simulate using a
infrared remote control but yeah that's
that's kind of what's coming at that
point we're yeah so UI is gonna be next
and then we're gonna get to the real fun
and the meat of how to build this over a
real internet connection now we've been
worried a little bit especially with
compression about how many bytes we're
sending let me tell you that the less
bytes you send the more Internet
connections and connectivity
circumstances it's going to work over so
it's always better to make sure you've
got good compression but once you know
you've got pretty good compression then
the real internet poses it's it's a it's
a playground like none other and like I
said this is
actually probably the largest area that
tends to make or break streaming
circuits and then we'll worry about this
other stuff as we get there so I'm gonna
stick to twice a week just for Matt why
so we're gonna keep going with
Wednesdays and Fridays until we get
through this stuff might need a little
break to prep this one but we'll see as
we get close to that but anyway
definitely want to thank all of you for
come