CS50 Video Player
    • 🧁
      • 🥧
      • 🍊
      • 🍿
    • 0:00:00Digital Audio
    • 0:05:03Graphics
    • 0:07:02RGB
    • 0:12:46Bitmap Format
    • 0:17:28Image Compression
    • 0:23:21Image File Formats
    • 0:27:01"Enhance"
    • 0:30:19Video Compression
    • 0:34:17Video File Formats
    • 0:36:433D Video
    • 0:00:00[MUSIC PLAYING]
    • 0:00:09SPEAKER 1: Multimedia, odds are you see it every day, you hear it every day,
    • 0:00:13but what is it?
    • 0:00:14Well, let's start with audio, what you hear coming out of a computer.
    • 0:00:17Turns out computers are really good at recording and playing back audio,
    • 0:00:20and they're really good at generating audio as well.
    • 0:00:23And they can do so using any number of file formats
    • 0:00:26where a file format is just a way of storing zeros and ones on disk in a way
    • 0:00:30that certain software knows how to interpret it.
    • 0:00:33So let's start with a particularly common file format
    • 0:00:36for musical instruments known as MIDI.
    • 0:00:38It turns out that using the MIDI format, M-I-D-I can you store effectively
    • 0:00:43the musical notes that compose some song.
    • 0:00:46And you can do this for different instruments,
    • 0:00:48and you can then play these instruments together
    • 0:00:50by telling the computer to interpret those notes
    • 0:00:52and then render them based on particular choices of instruments.
    • 0:00:56For instance, this here is a program called GarageBand on Mac OS,
    • 0:00:59and I've preloaded a MIDI file that I've downloaded online
    • 0:01:03and I daresay you will soon recognized the tune.
    • 0:01:06Let me go ahead and hit play.
    • 0:01:07[MUSIC PLAYING]
    • 0:01:26All right, well, that doesn't sound as good
    • 0:01:28as you might remember it sounding in the movie, but why is that?
    • 0:01:31Well, that's because my computer was synthesizing that music based only
    • 0:01:34on those musical notes.
    • 0:01:36So that wasn't an actual recording of an orchestra performing
    • 0:01:38that song, but rather it was a computer synthesizing or generating
    • 0:01:42the music based on an interpretation of those notes.
    • 0:01:45So MIDI is especially common among musicians who
    • 0:01:47want to share music with each other.
    • 0:01:49It's especially common in the digital musical space
    • 0:01:51where you do want the computer to synthesize the music for you.
    • 0:01:55But, of course, we humans are generally in the habit
    • 0:01:57of listening to songs as we know and love them on the radio or from CDs
    • 0:02:01back in the day or streaming media services.
    • 0:02:03And those are songs that have actually been performed typically by humans
    • 0:02:07and recorded often in a concert or in a sound studio, so they sound really,
    • 0:02:11really good and really really pristine.
    • 0:02:13Well, you don't have to use MIDI for those kinds of experiences
    • 0:02:16rather you can use any number of other file formats.
    • 0:02:19For instance, one of the earliest formats for audio
    • 0:02:21and still one of the most common for uncompressed audio
    • 0:02:24is called the wave file format, which can store data in an uncompressed form
    • 0:02:29so that you have a really, really high quality
    • 0:02:31versions of some audio recording.
    • 0:02:33But also popular and perhaps more popular among
    • 0:02:35consumers is that known as MP3 or MPEG3, which is a file format for audio that
    • 0:02:40uses compression to significantly reduce generally by a factor of more than 10
    • 0:02:45just how many bits are necessary to store some song on your hard drive
    • 0:02:49or on your music device or on your phone or any other form of technology
    • 0:02:53where you might store music.
    • 0:02:54And it does so by really throwing away zeros and ones
    • 0:02:58that we humans can't necessarily hear.
    • 0:03:00Now, some people will disagree, and true audio files
    • 0:03:03might disagree and insist that, actually,
    • 0:03:05you can tell the difference among these file formats,
    • 0:03:07but that may very well be the case because there's a trade off here.
    • 0:03:10If you want to use fewer bits and really fewer megabytes
    • 0:03:14to store your audio files, you might indeed
    • 0:03:15have to sacrifice some of the quality.
    • 0:03:17But the upside is that you might be able to store on your phone or your iPod
    • 0:03:21or some other device 10 times as much music as a result of that compression.
    • 0:03:25So audio compression is generally what's known as lossy, L-O-S-S-Y,
    • 0:03:29whereby you're actually losing some of the quality or the fidelity
    • 0:03:33of the music, but the gain is that you're using far less space to store
    • 0:03:37that information.
    • 0:03:38A similar file format in spirit is ACC, which is commonly used for audio files
    • 0:03:42as well as inside video files for audio.
    • 0:03:45And that's something that you might see when you download files from--
    • 0:03:48via iTunes, for instance, or the like.
    • 0:03:50And then there are streaming services these days
    • 0:03:52like Google Play and the Amazon store and Apple Music and Spotify, Pandora,
    • 0:03:56and others that don't necessarily transfer files outright
    • 0:03:59to your computer, but stream the bits to you so that they're actually
    • 0:04:03being played in real time so long as your internet connection can keep up
    • 0:04:06with the required bandwidth.
    • 0:04:08So how do we think about the quality of these recordings,
    • 0:04:10whether we're using any number of these file formats?
    • 0:04:13Well, you can think of it in terms of at least two parameters.
    • 0:04:16One is sampling frequency, the number of times per second
    • 0:04:19that we actually take a digital snapshot, so to speak,
    • 0:04:22of what it is the human would otherwise be hearing in person so as to then
    • 0:04:26represented digitally using zeros and ones.
    • 0:04:28And the second parameter would be the bit depth,
    • 0:04:30just how many bits are you using for that snapshot in time,
    • 0:04:34some number of times per second, in order
    • 0:04:36to represent the pitch and the volume and what it is the human is seeing.
    • 0:04:39And if you multiply those two values together,
    • 0:04:41the bit depth and the sample rate, will you
    • 0:04:43get just how many total bits are necessary to store
    • 0:04:45for instance one second of music?
    • 0:04:48And these file formats vary and allow you to vary
    • 0:04:51exactly what these parameters are.
    • 0:04:53So by using fewer bits, you might be able to save space
    • 0:04:56but get a lower quality recording, or if you want a super high quality
    • 0:04:59recording, you might use a higher bit rate all together.
    • 0:05:03So now let's transition to graphics, what we see in the world of multimedia.
    • 0:05:08Turns out here too there's multiple file formats for representing graphics.
    • 0:05:12And what is a graphic?
    • 0:05:13Well, graphic really if you think about it is just
    • 0:05:16a whole bunch of dots otherwise known as pixels both horizontally
    • 0:05:20and vertically.
    • 0:05:21Indeed most images that you and I see on the web, on our phones,
    • 0:05:24on our computers are rectangular in nature, though,
    • 0:05:26you can make some of the images transparent,
    • 0:05:29so they might appear to be other shapes.
    • 0:05:31But at the end of the day, all file formats for images
    • 0:05:33are rectangular in nature, and you can think of them
    • 0:05:36as just a grid of pixels or dots.
    • 0:05:38Now in the simplest form, each of those dots
    • 0:05:41might just be represented by a single bit, a 1 or a 0.
    • 0:05:45So for instance, here if you look far enough back,
    • 0:05:48is what appears to be a very happy smiley face.
    • 0:05:51But it's pretty simply implemented.
    • 0:05:53If you think of, again, this rectangular region
    • 0:05:55as just having a whole bunch of dots or pixels,
    • 0:05:57I've pretty much colored in in black only those dots necessary
    • 0:06:01to convey the idea of a happy face and left
    • 0:06:03in white any of the dots that are otherwise part of our background.
    • 0:06:07And you might then consider the white pixels
    • 0:06:09to be represented with a one, and the black pixels
    • 0:06:12to be represented with a zero or vice versa.
    • 0:06:14It doesn't really matter, so long as we're consistent in our file format.
    • 0:06:17And so if you take a step back, you can, kind of, sort of,
    • 0:06:20but it's really hard to see the same image even among those zeros and ones,
    • 0:06:25but that might be the simplest mapPNG from binary to an image.
    • 0:06:29You simply have to decide that there's some number of bits
    • 0:06:32horizontally, some number of bits vertically.
    • 0:06:34And if it's a 1, it's a white pixel, and if it's a 0,
    • 0:06:37it's a black pixel or equivalently vice versa.
    • 0:06:40But, of course, we don't generally use black and white images alone,
    • 0:06:44on the internet, on our phones, on our computers.
    • 0:06:46Indeed, the world would be pretty boring if it only looked like that.
    • 0:06:49And that's, indeed, how it looked way back in the day
    • 0:06:51even before there was digital and before we had file formats like this
    • 0:06:54when you just had black and white TV.
    • 0:06:56But that would really be similar in spirit
    • 0:06:59to what we're looking at here with some gray scales as well.
    • 0:07:02But here let's focus on color and the introduction
    • 0:07:05of color in a digital context, RGB, red, green, blue.
    • 0:07:09If you've ever heard this acronym, and even if you haven't, this
    • 0:07:12represents the three colors that can be mixed together really
    • 0:07:16to give us any color that we want--
    • 0:07:18RGB meaning red, green, and blue.
    • 0:07:21So using three different values, how much red
    • 0:07:24do you want, how much green do you want, how much blue do you want,
    • 0:07:26you can tell a computer to colorize each of those dots in a certain way.
    • 0:07:31Now if you have none of these colors, you'll actually get a black dot.
    • 0:07:35And if you have all of these colors mixed together in equal form,
    • 0:07:38you'll get a white dot.
    • 0:07:39But it's in the grades in between that you get all sorts of disparate colors.
    • 0:07:44So let's consider this.
    • 0:07:46Here is three bytes before you, and each is a byte, because each of these
    • 0:07:51is 8 bits where, again, a bit is just a 0 or a 1.
    • 0:07:54So I have eight bits here, eight bits here, and eight bits here.
    • 0:07:58The first byte of bits, first eight bits, is, of course, all ones
    • 0:08:02apparently.
    • 0:08:03The second byte is all zeros, and the third byte is all zeros as well.
    • 0:08:08So if you view each of these bytes, 1, 2,
    • 0:08:103 as representing how much of a certain color red, green, blue, RGB,
    • 0:08:17this appears to be a lot of red, because all of these bits
    • 0:08:20are ones, no green and no blue.
    • 0:08:25So are RGB, red, green, blue, lots of red, no green, no blue.
    • 0:08:30And so indeed this is how a computer would typically
    • 0:08:34using eight bits per color or 24 bits in total, 8 plus 8 plus 8,
    • 0:08:39would represent the number we know as red.
    • 0:08:42So that is to say if you think of this whole screen
    • 0:08:44as just one dot-- it's not quite a square.
    • 0:08:46It's a rectangle in this case-- but if you
    • 0:08:48think of this whole screen as just one dot,
    • 0:08:50if a computer wanted to make this dot red,
    • 0:08:53it would store a pattern of 24 bits, the first eight of which are all ones,
    • 0:08:57the second eight of which are all zeros, and the third of which
    • 0:09:00are all zeros as well.
    • 0:09:01And it will interpret the first of those eight bits
    • 0:09:03as meaning give me a lot of red, give me no green, give me no blue,
    • 0:09:07and thus you get a whole screen full of red or a whole pixel full of red.
    • 0:09:12What if we change it up?
    • 0:09:13What if we have a zero byte, a byte with all ones, and then another zero byte.
    • 0:09:19Thereby, making the red zero, the green all ones, and the blue all zeros.
    • 0:09:24Well, indeed, we'll get a screen filled with all green using
    • 0:09:28that encoding of 24 bits.
    • 0:09:29And you might guess in the end here, if we have zeros and zeros and then ones,
    • 0:09:33RGB, this time we're going to get blue.
    • 0:09:36That's how a computer using 24 bits would
    • 0:09:39represent a dot that's entirely blue.
    • 0:09:42Meanwhile, if you wanted represent black,
    • 0:09:44you would use all zeros for each of the R, G and B values,
    • 0:09:47and if you wanted to represent white, you would use all ones for each
    • 0:09:50of the R,G, and B values.
    • 0:09:52And you can get any number of colors in between these extremes
    • 0:09:55in any number of variations of red, green, and blue
    • 0:09:58by just mixing those colors together in different quantities.
    • 0:10:01Now it turns out when we talk about graphical file formats,
    • 0:10:04we don't typically talk in terms of or think in terms of binary.
    • 0:10:08We rather use something called hexadecimal.
    • 0:10:10Whereas binary just has two digits, zero and one
    • 0:10:13and whereas recall decimal has 10 digits zero through nine,
    • 0:10:17hexadecimal is a little different.
    • 0:10:18It has 16 possible digits.
    • 0:10:21And it's a little weird, but it's at least pretty straightforward.
    • 0:10:23Those 16 digits are zero through nine, and then
    • 0:10:27A through F. In other words, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F.
    • 0:10:35And so, of course, zero is the smallest number we can represent,
    • 0:10:38and 15 is going to be the largest number we can represent,
    • 0:10:42which is to say that F represents a 15.
    • 0:10:44So, in fact, let's consider an example.
    • 0:10:46Here is a pattern of eight bits, all of which are one.
    • 0:10:49Let me go ahead and add a little bit of space to these eight bits
    • 0:10:52just to separate them into two groups of four,
    • 0:10:54because it turns out one of the nice features of hexadecimal
    • 0:10:57mathematically is that each hexadecimal digit zero through F
    • 0:11:02represents in total, four bits.
    • 0:11:04Which is to say that we can take a number in binary like this,
    • 0:11:08look at it as two halves, one half of a byte followed
    • 0:11:11by another half of a byte, and use one hexadecimal digit instead
    • 0:11:15of four binary digits to represent the first four bits.
    • 0:11:18And then one other hexadecimal digit to represent the other four bits.
    • 0:11:22So we can take something that takes eight symbols represent and widdle it
    • 0:11:26down to just two, which is pretty convenient.
    • 0:11:28And so, in fact, it turns out that in hexadecimal
    • 0:11:32if we had all zeros, in hexadecimal that would just be 0.
    • 0:11:35But if we have all ones, 1, 1, 1 and we convert that to hexadecimal,
    • 0:11:40that's going to be if this is the one place and the twos places
    • 0:11:43and the fours place and the eights place, that's
    • 0:11:45going to be the number 15, otherwise known
    • 0:11:47in hexadecimal as F. Which is to say if you have a byte of bits,
    • 0:11:538 bits, all of which are ones, you can think of that same byte
    • 0:11:57as being two hexadecimal digits FF, as opposed to thinking of it
    • 0:12:02as 1, 1,1,1, 1, 1, 1, 1, it's just FF.
    • 0:12:07So it's a more succinct way of representing
    • 0:12:09the exact same information.
    • 0:12:11And so accordingly, if you want to think about red a little more succinctly,
    • 0:12:16you don't have to think about it in terms of eight ones and eights zeros
    • 0:12:19and eight zeros, you can think of it in terms of FF, 0, 0, 0,
    • 0:12:230 just because it's more succinct-- similarly for green, 0,0 F, F, 0,
    • 0:12:280 and for blue 0,0, 0,0, F,F. It's just a more succinct way of explaining
    • 0:12:34oneself, and indeed a lot of graphical editing programs like Photoshop being
    • 0:12:38one of the most popular actually use this notation certainly instead
    • 0:12:41of binary and also instead often of decimal just by convention.
    • 0:12:47So now let's consider some specific file formats.
    • 0:12:49If you're a PC user, you might not have seen this in a while,
    • 0:12:52but odds are when you did it was for quite
    • 0:12:54a few years, this beautiful rolling hill with a beautiful cloudy sky behind it.
    • 0:13:00This was, of course, the wallpaper or the background image
    • 0:13:03that came by default with Windows XP on operating system from Microsoft
    • 0:13:07for PC computers.
    • 0:13:09So the very first time you turn on your computer and, perhaps, logged in,
    • 0:13:12you would see a screen like this and maybe some of your icons
    • 0:13:14and your recycle bin and the like.
    • 0:13:16Now as an aside and spoiler, this is what that same hill apparently
    • 0:13:20looks like today.
    • 0:13:21So it hasn't necessarily aged well, but for our purposes what's interesting
    • 0:13:25here is what this image was stored as.
    • 0:13:28It turns out that this image originally was a bitmap file, BMP, or bitmap,
    • 0:13:33B-I-T-M-A-P, to pronounce it out loud.
    • 0:13:37And that file format really is what that word implies.
    • 0:13:40It's a map of bits.
    • 0:13:42It's a grid of bits, which is perfectly consistent with our definition
    • 0:13:45earlier of a very simple smiley face using just zeros
    • 0:13:49and ones or black and white dots.
    • 0:13:51This case, clearly, has many more colors than that,
    • 0:13:55and indeed it's certainly the case in general.
    • 0:13:57The graphical file formats on computers support
    • 0:14:00dozens of colors, hundreds of colors, thousands,
    • 0:14:02maybe even millions of colors, certainly, more than just
    • 0:14:05black and white alone.
    • 0:14:07But there's a finite amount of information here.
    • 0:14:09And even though this looks like a beautifully crisp green grassy area
    • 0:14:14and a beautifully blue--
    • 0:14:15a beautifully blue sky with some very smooth clouds,
    • 0:14:19if we actually zoom in on those clouds, you'll
    • 0:14:22see that indeed an image is really just a grid of dots.
    • 0:14:26In fact, let me zoom in on those clouds, and I've not done any alterations.
    • 0:14:30I simply used a graphical program to take that same sky
    • 0:14:33and zoom in, zoom in, zoom in as much as I can.
    • 0:14:36And as soon as you zoom in enough, you see that that cloud that previously
    • 0:14:40looked especially smooth to the human eye, really isn't.
    • 0:14:44It's just that my human eyes can't really see dots, especially
    • 0:14:47clearly when they're really small, and there's a very high resolution so
    • 0:14:51to speak-- a lot of pixels horizontally and a lot of pixels
    • 0:14:53vertically in an image.
    • 0:14:55But if I do zoom in on that, I actually do see the pixilation so to speak,
    • 0:15:00whereby you actually see the dots.
    • 0:15:01And you can see that those clouds are really just
    • 0:15:03roughly represented as a green--
    • 0:15:06as a grid of dots or a map of pixels, a rectangular region of pixels.
    • 0:15:13So that's all very interesting now, because it
    • 0:15:16would seem that we don't have an endless ability
    • 0:15:19to zoom and zoom and zoom in and see more
    • 0:15:21and more detail unless that information's already there.
    • 0:15:25And so, much like with audio, when you have the choice over just how many bits
    • 0:15:29to use, so in the world of images do you have
    • 0:15:32discretion over how many bits to use.
    • 0:15:34How many bits do you use to represent each dots color.
    • 0:15:37And that might indeed be just 8 bits for red, 8 bits for green, 8 bits for blue,
    • 0:15:41AKA 24-bit color, but resolution also play-- comes into play.
    • 0:15:47If you have an image that's only 100 pixels, for instance, by 100 pixels,
    • 0:15:52horizontally by vertically, it might only be this big.
    • 0:15:55Now that might not big enough to fill-- be big enough
    • 0:15:57to fill your whole background wallpaper on your computer,
    • 0:16:00and so you might try to scale it up or zoom in on it.
    • 0:16:04But when you do that, you're taking only a limited amount of information,
    • 0:16:07100 pixels by 100 pixels, and you're essentially
    • 0:16:09just duplicating those pixels making them bigger and blotchier just
    • 0:16:12to fill your screen.
    • 0:16:14Better would be to not start with an image with so few pixels,
    • 0:16:17but rather get a much higher resolution image.
    • 0:16:19And indeed, this is what you get with newer and better camera phones
    • 0:16:23these days, newer and bigger, better digital cameras
    • 0:16:26is among other things do you get higher and higher resolution.
    • 0:16:29More and more dots, so that the dots ultimately that we humans see
    • 0:16:33are so small on our screens, it looks ever more smooth
    • 0:16:37than, say, an image like this.
    • 0:16:39So generally speaking, higher resolution gives us
    • 0:16:41higher fidelity and a cleaner image.
    • 0:16:44The other factors in cameras certainly play into that as well.
    • 0:16:47But there's something else I notice here.
    • 0:16:50It seems a little silly that I'm using the same number of bits
    • 0:16:53to represent the color of every one of the dots on the screen.
    • 0:16:57Because even though I do see a few different shades
    • 0:16:59of gray or white in there and light blue and dark blue,
    • 0:17:02I see a lot of identical blue throughout this image.
    • 0:17:05There's a lot of redundancy, and indeed if we rewind,
    • 0:17:08there's a whole lot of blue in this image itself.
    • 0:17:11There's a whole bunch of similar white it would
    • 0:17:13seem in the middles of the clouds.
    • 0:17:14There's a whole bunch of similar looking green.
    • 0:17:16And yet we are using, it would seem by default, 24 pixels--
    • 0:17:2124 bits for every pixel, which just seems wasteful
    • 0:17:25even if one pixel is identical to the one next to it.
    • 0:17:28So it turns out that graphical file formats can often be compressed,
    • 0:17:33and this can be done in different ways.
    • 0:17:35It can be done losslessly or lossely.
    • 0:17:38So earlier you'll recall that I proposed shrinking audio files by throwing away
    • 0:17:45information that maybe my human ears can't necessarily hear
    • 0:17:47or my non-audio file might not even notice are missing.
    • 0:17:50And that would be lossy compression, and then I'm
    • 0:17:52just throwing information away assuming that the user's not going to notice.
    • 0:17:56But that's not always necessary.
    • 0:17:58Sometimes you can do lossless compression,
    • 0:18:00whereby you can use fewer bits to store the same information.
    • 0:18:04You just have to store it more intelligently.
    • 0:18:06So consider this example here where you have an apple against a blue backdrop
    • 0:18:11and that, much like our blue sky, seems pretty consistent throughout.
    • 0:18:16And so it seems a little silly intuitively
    • 0:18:18to record an image like this on disk as follows.
    • 0:18:21If we think about me being a verbalisation of a file format,
    • 0:18:25make this pixel blue, make this pixel blue, make this pixel blue,
    • 0:18:29make this pixel blue, make this pixel blue, make this pixel blue.
    • 0:18:32Literally saying the same sentence, or more technically
    • 0:18:36using the same 24 bits for every pixel across that entire row
    • 0:18:40even though my sentence might not be changing.
    • 0:18:43And so instead what a clever file format might do is this.
    • 0:18:47This is not what the user sees, but this is
    • 0:18:50what the file format could store with respect to all of that redundant blue.
    • 0:18:54Just remember, for instance, the leftmost
    • 0:18:57pixels color as by saying this pixel is blue,
    • 0:19:01and then for the rest of the row or scanline as it's called in an image
    • 0:19:05just say, and so are the rest of the pixels in this row.
    • 0:19:08So I can say much more concisely, essentially repeat this color
    • 0:19:13throughout the entirety of the rest of the row,
    • 0:19:15thereby saving myself any number of sentences let alone
    • 0:19:18any number of 24 bits.
    • 0:19:20And I can do that the same here, make this pixel blue
    • 0:19:23and then repeat that image--
    • 0:19:25that color again and again and again.
    • 0:19:26Now it gets a little less efficient as soon as we hit like the stem
    • 0:19:29on the apple, because then that sentence has to change.
    • 0:19:32Then we have to say something like make this pixel brown, make this pixel blue,
    • 0:19:36and then repeat again.
    • 0:19:37So we have to, kind of, stop and start if there's some obstruction in the way.
    • 0:19:41And the same thing for the red apple itself.
    • 0:19:43But just look based on the white at how much information we're potentially
    • 0:19:47saving or how many bits we're potentially saving,
    • 0:19:50and yet we're saving those bits in a way that the original information is
    • 0:19:54recoverable.
    • 0:19:55Just because we don't store 24 bits representing
    • 0:19:58blue for every one of these dots on the screen, doesn't mean we
    • 0:20:02can't display blue there just by interpreting this file
    • 0:20:05format a little more cleverly.
    • 0:20:07And so this is indeed how a file format might actually losslessly compress
    • 0:20:12itself using fewer bits to store the same image, but in a way
    • 0:20:16where you can recover the original image itself.
    • 0:20:20Now let's take a look at another example this time of lossy compression.
    • 0:20:24Here is a beautiful sunflower taken somewhere here on campus
    • 0:20:28at Harvard University.
    • 0:20:29This is a high quality JPEG photograph where JPEG is a popular file
    • 0:20:34format for photographs especially.
    • 0:20:37And this image here was somewhat compressed, but not very compressed.
    • 0:20:42In fact, only if I put my face really awkwardly close to the screen
    • 0:20:45do I see that it's a little bit blotchy way up close.
    • 0:20:49But from just a foot or so or beyond, it looks perfectly pristine.
    • 0:20:53But not if we compress this image further.
    • 0:20:56Suppose that this image is just too big to fit on my Facebook profile page,
    • 0:21:01or it's just too big to email to a friend via my phone.
    • 0:21:06In other words, I need to use fewer bits or fewer megabytes
    • 0:21:10even if it's a really big file to store this same image
    • 0:21:13and convey the gist of the image to that friend.
    • 0:21:16Now I see a little bit of blue and I do see a bunch of yellow,
    • 0:21:19but it's not quite the same clean pattern
    • 0:21:21that we saw with the apple or even the blissful blue sky
    • 0:21:25above the green grassy hill.
    • 0:21:27And so if I were instead using a file format that can still
    • 0:21:31be compressed, but lossily where we're actually throwing information away,
    • 0:21:36this might be the before image.
    • 0:21:39And now wait for it.
    • 0:21:40This might be the after image.
    • 0:21:43So it's still clearly a sunflower, though it looks
    • 0:21:46a little more sickly at this point.
    • 0:21:48But it definitely looks blotchier.
    • 0:21:50In fact, from a foot or more away, I can actually
    • 0:21:52see that my sky has become very pixelated.
    • 0:21:55It almost looks like Super Mario Bros. back in the old Nintendo systems
    • 0:21:58where you could really see the big dots.
    • 0:22:00And the greenery here is just a grid of pixels too,
    • 0:22:03and even the flower has really just become
    • 0:22:05a collection of dots that I ever so clearly see on the screen.
    • 0:22:09And certainly this flower looks none so good anymore.
    • 0:22:12So let's rewind.
    • 0:22:13This was before, after, before, after.
    • 0:22:19And so this is what it means to lossily compress an image.
    • 0:22:24I cannot go from this pretty poor version back to the original,
    • 0:22:28if I have achieved this compression by just throwing away some of those bits.
    • 0:22:33So whereas before I was very cleverly just
    • 0:22:35remembering repetition in the image, in this case using this file format,
    • 0:22:39especially when you really turn the virtual knob
    • 0:22:42and say compress this as much as you can.
    • 0:22:44Essentially what my graphical software is going to do
    • 0:22:47is start to use approximations.
    • 0:22:48Well, does this leaf here really need to be 20 different shades of green?
    • 0:22:52How about just two?
    • 0:22:54And that's why I get this big green blotch here and this other green blotch
    • 0:22:57here.
    • 0:22:57Does this sky really needs to be 30 different shades of blue?
    • 0:23:01How about two shades of blue and two shades of gray?
    • 0:23:04And so that might be a way to use less information to still represent
    • 0:23:07the same sky.
    • 0:23:09I don't know in this file format just how clear the sky used to be,
    • 0:23:13because those dots have essentially been thrown away and aggregated in this way.
    • 0:23:17But it makes for a much smile--
    • 0:23:19much smaller file format.
    • 0:23:21And so what are the formats that are disposable?
    • 0:23:23Well, there's any number of options out there today,
    • 0:23:25but perhaps the most common are these.
    • 0:23:28There's the bitmap file format, which was commonly
    • 0:23:30used originally in Windows and other contexts, not super common these days.
    • 0:23:34Certainly, not on the web, but does indeed
    • 0:23:36lay out all of your pixels in a grid essentially on disk of zeros and ones.
    • 0:23:41Meanwhile, there's gif, which is commonly used for low quality images
    • 0:23:48in multiple senses of the word.
    • 0:23:49This is often used for icons on the screen or clip art that you might see,
    • 0:23:53and it's also increasingly used for internet memes or the kinds of images
    • 0:23:57that you might forward along to friends or see
    • 0:23:59popping up on your screen in large part, because gifs can be animated.
    • 0:24:03So they're, sort of, a very low end version of a video file
    • 0:24:07where really it's like an image with--
    • 0:24:10it's like a video file with just a few images inside
    • 0:24:14of it that often play on the repeat, so one after the other
    • 0:24:17creating the illusion of some form of animation.
    • 0:24:19But the resolution of gifs tends to be not very high, although they
    • 0:24:24can be losslessly compressed, as we saw with the apple before,
    • 0:24:27but they only support 8-bit color.
    • 0:24:30And 8 bits can mean-- implies that we can only
    • 0:24:34have a total of 256 colors in the image itself, which limits the range.
    • 0:24:38And so they tend not to look great, especially when
    • 0:24:40large for things like photographs of humans and in grassy knolls.
    • 0:24:45JPEG, meanwhile, is the file format we saw just a moment ago
    • 0:24:49of that beautiful sunflowers.
    • 0:24:50This actually supports 24-bit color, but is lossily compress,
    • 0:24:54so you might lose some information when shrinking those image files,
    • 0:24:57but it allows you so many more colors that you can see images typically
    • 0:25:01with much higher fidelity at much greater quality.
    • 0:25:04Meanwhile, there's PNGs as well.
    • 0:25:07PNGs are commonly used for high quality graphics
    • 0:25:10that you might want to print or resize, supporting 24-bit color as well,
    • 0:25:15and are generally used for images that you might indeed
    • 0:25:18want to use in multiple contexts.
    • 0:25:20Not neccess-- not so much photographs, but other artwork that's higher quality
    • 0:25:25than gifs.
    • 0:25:25And here's just a few examples.
    • 0:25:27This is, perhaps, the most ridiculous animated gif that I could find.
    • 0:25:32This here being a cat flying through the sky.
    • 0:25:34And this is an animated gif in the sense that it's really just one
    • 0:25:37image after another, after another, after another,
    • 0:25:39and they're repeating again and again and again and again.
    • 0:25:43So even though it looks like motion, really you're
    • 0:25:45just seeing a bunch of images each of which
    • 0:25:47has the cat in a slightly different position,
    • 0:25:49and it's rainbow and the stars in a slightly different position.
    • 0:25:51And if you loop these again and again, it looks like the cat's moving,
    • 0:25:54but really you're just seeing a whole bunch of images every split second.
    • 0:25:58Meanwhile, here is another JPEG in addition to the sunflower earlier.
    • 0:26:02This is a beautiful shot of the ceiling here in Sanders Theater at Harvard
    • 0:26:06University, and JPEG really lends itself to photography,
    • 0:26:09because you have not only a huge range of colors,
    • 0:26:11you also have the choice not really to compress the files very much.
    • 0:26:14The fact that my sunflower got so ugly on the screen
    • 0:26:17was because I deliberately said compress that sunflower as much as you can,
    • 0:26:21but that doesn't need to be the case.
    • 0:26:23If you can afford to spend the bytes on disk
    • 0:26:25or you can afford to post a really big image on the internet,
    • 0:26:28then you can certainly use minimal compression
    • 0:26:30and capture a really beautiful image.
    • 0:26:33As for a PNG, here might be a good opportunity for a PNG,
    • 0:26:36a really high resolution version of say Harvard's crest
    • 0:26:38that you might want to print small on some piece of paper or large
    • 0:26:41on a banner or the like.
    • 0:26:43And so this might lend itself especially to an application like that.
    • 0:26:48Of course, we don't have an infinite amount of information
    • 0:26:51at our disposal in graphics.
    • 0:26:53Rather we only have the pixels and the dots
    • 0:26:56and the colors that are there when that image was saved in some file format.
    • 0:27:01And so it's quite all too common to see in popular television and film,
    • 0:27:06sort of, abuses of what it means to be a multimedia format and a graphical
    • 0:27:11file format at that.
    • 0:27:11Such that there's entered the lexicon this notion of enhance
    • 0:27:15where enhance essentially means apparently in the media
    • 0:27:19make this image as clearly readable as possible
    • 0:27:22no matter what format it was saved in.
    • 0:27:24And we can see some examples of that with this popular TV show here.
    • 0:27:29SPEAKER 2: We know.
    • 0:27:32SPEAKER 3: That at 9:15 Ray Santoya was at the ATM.
    • 0:27:35SPEAKER 2: The question is what was he doing at 9:16?
    • 0:27:39SPEAKER 3: Shooting the 9 millimeter at something.
    • 0:27:42Maybe he saw the sniper.
    • 0:27:43SPEAKER 2: [INAUDIBLE]
    • 0:27:45SPEAKER 3: Right.
    • 0:27:46Go back one.
    • 0:27:47SPEAKER 2: What do you see?
    • 0:27:56SPEAKER 3: Bring his face up full screen.
    • 0:28:00SPEAKER 2: His glasses.
    • 0:28:01SPEAKER 3: There's a reflection.
    • 0:28:12SPEAKER 2: [INAUDIBLE] baseball team.
    • 0:28:14That's their logo.
    • 0:28:15SPEAKER 3: And he's talking to whoever's wearing a jacket.
    • 0:28:18SPEAKER 2: We may have a witness.
    • 0:28:19SPEAKER 3: To both shootings.
    • 0:28:21SPEAKER 1: All right, let's take a closer look
    • 0:28:23at exactly what we just saw.
    • 0:28:25So they're watching this video of some bad guy presumably,
    • 0:28:28and they're trying to identify the suspect.
    • 0:28:30So they're really just looking at what's called
    • 0:28:32a frame in a video, which for all intensive purposes
    • 0:28:34is just an image inside of a video.
    • 0:28:37Because what's a video?
    • 0:28:38Well, much like the animation we saw a moment ago,
    • 0:28:40a video really is just a set of images being
    • 0:28:43shown really fast to the human eye generally
    • 0:28:45at a rate of 24 frames or images per second
    • 0:28:48or as many as 30 frames or images per second,
    • 0:28:50thereby creating the illusion of motion or really motion pictures.
    • 0:28:54But really it's just a whole bunch of pictures
    • 0:28:56being shown to us super quickly.
    • 0:28:57So here's one such picture.
    • 0:28:59And here apparently is the key to solving this mystery.
    • 0:29:02Indeed, if we enhance that glint in this fellow's eye,
    • 0:29:07we apparently see exactly this.
    • 0:29:08And by the magical incantation of enhance do we apparently see this.
    • 0:29:13And this is where reality breaks down.
    • 0:29:16If this is the entirety of the information that
    • 0:29:19has been stored in some file format and indeed you
    • 0:29:21can see the pixels and the pixelation, the blotchiness
    • 0:29:24because only so many bits and only so much resolution
    • 0:29:28was used to store that image and we are looking
    • 0:29:30at a tiny, tiny, tiny fraction of it in the reflection
    • 0:29:34of that fellow sunglasses, this is all the information that we might have.
    • 0:29:38Now, you might stare at this all day long
    • 0:29:40and, kind of, sort of, think that you see
    • 0:29:41who it is that had perpetrated this crime,
    • 0:29:46but you're certainly not going to get from that anything
    • 0:29:49close to the resolution of this, unless the original video
    • 0:29:52and, therefore, the original frame or image was as high resolution
    • 0:29:56as this output suggests.
    • 0:29:58So the information, the bits, the pixels aren't just there.
    • 0:30:01And even cartoons of today like Futurama know this.
    • 0:30:06SPEAKER 4: Magnify that death spear.
    • 0:30:10Why is it still blurry?
    • 0:30:11SPEAKER 5: That's the resolution we have.
    • 0:30:13Making it bigger doesn't make it clearer.
    • 0:30:16SPEAKER 4: It does on CSI Miami.
    • 0:30:19SPEAKER 1: All right, and what better segue then
    • 0:30:21to video file formats themselves then these excerpts from some actual videos.
    • 0:30:26Indeed, you can think of a video file format as very reminiscent of something
    • 0:30:30from the real world.
    • 0:30:30In fact, as a kid if you either made or played with these little flip books,
    • 0:30:34you might have had the ability to actually see something animated really
    • 0:30:38by just flipping through some physical pieces of paper really quickly.
    • 0:30:41Well, that's all a video format is in the digital age.
    • 0:30:45It is simply a file format that contains essentially a whole bunch of images
    • 0:30:49inside of it, each of which is shown to you so fast that there appears to be
    • 0:30:53the illusion just like this of motion.
    • 0:30:55And you're seeing 24 images per second, 30 images per second,
    • 0:30:59and it's not necessarily that they're all PNGs or JPEGS or gifs
    • 0:31:03or actual images inside of it, there's actually
    • 0:31:05more complicated and sophisticated ways of storing the information so you're
    • 0:31:09not just storing each of the frames.
    • 0:31:11You can actually use algorithms and mathematics
    • 0:31:13to actually go from one frame to another.
    • 0:31:15And indeed, there are some very clever opportunities
    • 0:31:19when it comes to videos for compressing video formats themselves.
    • 0:31:23We can certainly leverage within frames or Intraframe so to speak,
    • 0:31:27the exact same techniques that we saw earlier with something
    • 0:31:30like a gif and an apple where we can actually leverage the fact that there's
    • 0:31:33redundancy in a given frame of a video, throw that information away, and just
    • 0:31:37remember whole sky is blue or the whole rest of some line or row in a file
    • 0:31:42is blue and, therefore, save on information and bits.
    • 0:31:45But with videos you have another opportunity, because you don't just
    • 0:31:48have an individual picture, you have a picture in every subsequent picture,
    • 0:31:52which might look very similar as well.
    • 0:31:54In fact, if I hold very still for multiple seconds,
    • 0:31:59odds are almost everything in this video is
    • 0:32:02staying the same except for my mouth, apparently
    • 0:32:04my pointer finger and my lips and eyes as I blink,
    • 0:32:10but everything else about me is pretty much the same.
    • 0:32:13So why would you in your file format store all of the various colors
    • 0:32:18that we see behind me and around me?
    • 0:32:20You don't need to do that.
    • 0:32:21You can also leverage something called interframe compression, whereby
    • 0:32:26in simplest form you can take a look at the current frame of a video
    • 0:32:29and look at the next frame and decide what has changed.
    • 0:32:32And maybe look another frame after that, see
    • 0:32:34what has changed, and another frame after that and see what has changed.
    • 0:32:37And essentially store not every image from the starting point
    • 0:32:42to the ending point, but really just the differences between those frames that
    • 0:32:47are adjacent.
    • 0:32:48So for instance, if we start off with this bee here on a trio of flowers
    • 0:32:52and he moves and he moves and he moves, we could--
    • 0:32:56if not compressing this video and these four frames that
    • 0:32:59compose the video-- we could just store each of those images essentially as is,
    • 0:33:03even though flowers are not moving, the leaves are not moving.
    • 0:33:06The only thing that's moving is the bee.
    • 0:33:08Or we can be more clever about this just as we were
    • 0:33:10with the blue sky behind the apple.
    • 0:33:13We can recognize that between picture one and picture four,
    • 0:33:16or the first four frames of this video, the only thing that's moving
    • 0:33:20is indeed that bee.
    • 0:33:22So maybe we should store just what we'll call keyframes or a snapshot
    • 0:33:26in time of what the video looks like.
    • 0:33:29And then on each subsequent frame, essentially,
    • 0:33:31just remember what information has changed, in this case,
    • 0:33:34the position of the bee and leave it to the computer playing the video
    • 0:33:38to infer or interpolate these inner frames based
    • 0:33:42on those so-called keyframes.
    • 0:33:44Use a bit of clever math, use some algorithms to actually figure out
    • 0:33:47that, oh, here's where the bee now is.
    • 0:33:49Let me redraw the exact same flower and the exact same leaves behind that bee.
    • 0:33:53But I now only have to store really as many bits
    • 0:33:56as it takes to remember where the bee now is there,
    • 0:33:58where the bee now is here, and then just for good measure
    • 0:34:01to keep everything synchronized maybe every few frames
    • 0:34:03we'll have another keyframe that, even though it's a little expensive,
    • 0:34:06stores the entirety of the frame.
    • 0:34:08Just in case something goes wrong, we can guarantee ourselves
    • 0:34:11that we can reconstruct what the video actually is even if there's
    • 0:34:14a little bit of a glitch otherwise.
    • 0:34:17So what are the file formats that we have at our disposal?
    • 0:34:21Well, in the video world the terminology gets a little more complicated
    • 0:34:24in that there are a number of different solutions
    • 0:34:27to the problem of storing video.
    • 0:34:28And indeed these are what the world might call containers.
    • 0:34:31And a container is just as the name implies,
    • 0:34:33it's a digital container inside of which you can put multiple types of data.
    • 0:34:39And the types of data you might put into a container
    • 0:34:41would be a video track, like the actual footage
    • 0:34:44that you see on the screen, an audio track.
    • 0:34:46Which is the actual audio that you hear, maybe a secondary audio track.
    • 0:34:49If a film has been dubbed from one language into another,
    • 0:34:53you might have multiple audio tracks in the same container.
    • 0:34:55And then the software on your computer or even on your TV for that matter
    • 0:34:58that's playing back this video can actually
    • 0:35:00choose between those multiple audio formats.
    • 0:35:04You might have closed captions or some other track inside the container.
    • 0:35:07So long story short, a container really is just that.
    • 0:35:10It's this bucket inside of which is the video and the audio,
    • 0:35:12but maybe multiple formats thereof so that you can play them back
    • 0:35:16based on your own preferences.
    • 0:35:18So AVI is a very popular format that's been commonly used in the Windows world
    • 0:35:21for years, as has been DIVX.
    • 0:35:23MP4 and Quicktime have been more common on the side of Macs,
    • 0:35:27although MP4 is now pretty much universal across all browsers
    • 0:35:30and operating systems and more.
    • 0:35:32Otrosca is more of an open source container that's
    • 0:35:35meant to be even more versatile than these others
    • 0:35:38on this screen capable of storing any number of file formats inside.
    • 0:35:42And as to those formats inside, they might indeed be video.
    • 0:35:46They might indeed be audio.
    • 0:35:47But within those worlds realize there are
    • 0:35:49different ways of storing and encoding information,
    • 0:35:53and those inner most rappers use what are called codecs
    • 0:35:57where a codek is just a way of encoding information in a video
    • 0:36:00or in an audio file format.
    • 0:36:02And there's any number of these options as well, but perhaps
    • 0:36:05some of the most common these days is something called H.264 for video, which
    • 0:36:09is a way of storing video on disk inside of a container, or MPEG-4 part 2,
    • 0:36:13a little bit more verbosely.
    • 0:36:14A popular alternative there too.
    • 0:36:16And then in the world of audio files, two terms we've seen before,
    • 0:36:19and this is where the world gets a little confusing,
    • 0:36:21sometimes the container formats are the same as the actual media formats.
    • 0:36:25And in this case, AAC and MP3 can be standalone files
    • 0:36:29that you download and listen to in iTunes or some other software,
    • 0:36:33or they can be tracks inside of a container that actually provide a video
    • 0:36:40with the audio that accompanies it.
    • 0:36:43But there aren't just these two dimensional file formats, if you will.
    • 0:36:47There are increasingly three dimensional or virtual formats
    • 0:36:51as well that allow you to capture the entirety of spaces like this.
    • 0:36:55In fact, this is a picture that is knowingly a little bit distorted,
    • 0:36:58because if you look up and around in reality at this space,
    • 0:37:02it doesn't look so wide and stretched out.
    • 0:37:03And the stage definitely isn't curved like this,
    • 0:37:06but essentially what you're looking at now
    • 0:37:08is a 360 degree photograph of this exact stage.
    • 0:37:12And that image, even though it's effectively
    • 0:37:14a sphere that captures the entirety of this space,
    • 0:37:17it's essentially like you've taken a sphere and cut it around the edges
    • 0:37:20and then flattened it out, much like flattening a globe of the earth
    • 0:37:24into a rectangular region, and what you get
    • 0:37:26is something that's a little distorted.
    • 0:37:28But if you kind of stare at this for just a moment
    • 0:37:31and you imagine that the wooden stage here is really
    • 0:37:34meant to be a straight line and all of these seats
    • 0:37:36are supposed to be put together side by side,
    • 0:37:39you can imagine re-forming a sphere out of this otherwise flat
    • 0:37:43two-dimension image and putting yourself inside of it
    • 0:37:45and being able to experience a space like this.
    • 0:37:48So increasingly some of these same file formats that we've discussed,
    • 0:37:51among them JPEG, for instance, for photographs,
    • 0:37:53do you have the ability to inject what's called metadata, some additional often
    • 0:37:57textual data that the human looking at an image doesn't see.
    • 0:38:01But programs like Photoshop and browsers and applications
    • 0:38:04can actually read and realize, oh, this image
    • 0:38:07has not only a grid of pixels, compressed
    • 0:38:10or otherwise, color or otherwise, that I can display to the user,
    • 0:38:13there's also some additional metadata that tells me
    • 0:38:16how to display this image in a way that's much more immersive,
    • 0:38:20so that the image effectively wraps around the user.
    • 0:38:23Now the user might look a little silly doing so,
    • 0:38:25but if he or she has a headset quite like this one here,
    • 0:38:29he or she can take a look at this image, pull it up
    • 0:38:32on the digital screen that's before him, and thanks
    • 0:38:34to two small lenses, left eye and right eye, start to look up and down
    • 0:38:40and left and right and all around him or her and actually see a space like this
    • 0:38:44and experience it in 360 degree virtual reality.
    • 0:38:49So this is just a taste then of the file formats that currently exist,
    • 0:38:52that are on the horizon today, and just who knows what more will exist.
    • 0:38:55But at the end of the day, it all boils down to bits, to zeros and ones,
    • 0:38:59how you arrange them on disk, and what features you provide to the users
    • 0:39:02with which to capture their imagination.
  • CS50.ai
Shortcuts
Before using a shortcut, click at least once on the video itself (to give it "focus") after closing this window.
Play/Pause spacebar or k
Rewind 10 seconds left arrow or j
Fast forward 10 seconds right arrow or l
Previous frame (while paused) ,
Next frame (while paused) .
Decrease playback rate <
Increase playback rate >
Toggle captions on/off c
Toggle mute m
Toggle full screen f or double-click video