The Latest on
HTML5 Media

Silvia Pfeiffer

Use your cursor keys to move forward and backward through the slides.

Best viewed in Firefox 4beta

1
Overview

http://caniuse.com/

  • Video
  • Behind the scenes
  • Audio
2

Video Element & Video Manipulation

3
Innocent Web browser <video> element
              <video controls poster="HelloWorld.png" width="500">
                  <source src="HelloWorld.mp4" type="video/mp4">
                  <source src="HelloWorld.webm" type="video/webm">
                  <source src="HelloWorld.ogv" type="video/ogg">
              </video>
            
4
Well, not just HTML - here's the CSS
      video {
        border-radius: 12px;
        border: 1px solid rgba(0, 0, 0, 0.2);
      }
            
5
More CSS3: transitions
      video {                         video, video:hover {
        height:    200px;               position: relative;
        padding:    10px;               border:   5px solid black;
        left:       50px;               transition-property: all;
        top:        50px;               transition-duration: 0.5s;
        background-color: white;        transition-timing-function: linear;
      }                               }
      video:hover {
         height:   250px;
         padding:   15px;
         left:      25px;
         top:       25px;
         background-color: black;
       }
            
6
More cool CSS3: transforms (rotate)
7
Media and the JavaScript API
8
Custom Player using JS and CSS
9
SVG and Video

Cool filters, but only work in Firefox with in-line SVG

10
Grabbing the video frame into a Canvas
<video controls height="270" width="480" >
  <source src="HelloWorld.mp4"  type="video/mp4">
  <source src="HelloWorld.webm" type="video/webm">
</video>
<canvas width="400" height="300" style="border: 1px solid black;">
</canvas>
<script>
  window.onload = function() {
    initCanvas();
  }
  var context;
  function initCanvas() {
    video = document.getElementsByTagName("video")[0];
    canvas = document.getElementsByTagName("canvas")[0];
    context = canvas.getContext("2d");
    video.addEventListener("timeupdate", paintFrame, false);
  }
  function paintFrame() {
    context.drawImage(video, 0, 0, 160, 80);
  }
</script>
11
Processing Frame Data - a "scratch" canvas
        context.drawImage(video, 0, 0, w, h);
        frame = context.getImageData(0, 0, w, h);
        for (var i = 0; i < frame.data.length; i += 4) {
          r += frame.data[i];
          g += frame.data[i + 1];
          b += frame.data[i + 2];
        }
        ambience.style.backgroundColor = 'rgb('+r+','+g+','+b+')';
            
12
Pixel Manipulation
13
Pixel Manipulation - the code
    output = context.createImageData(w, h);

    // Loop over each pixel of input file
    for (x = 0; x < w; x++) {
      for (y = 0; y < h; y++) {
        // index in output image
        i = x + w*y;
        for (c = 0; c < 4; c++) {
          output.data[4*i+c] = frame.data[4*i+c];
        }
        // make pixels transparent
        r = frame.data[i * 4 + 0];
        g = frame.data[i * 4 + 1];
        b = frame.data[i * 4 + 2];
        if (!(r > 200 && g > 200 && b > 200))
          output.data[4*i + 3] = 0;
      }
    }

    // Draw the ImageData object.
    context.putImageData(output, 0, 0);
            
14
<video> and Canvas - reflection
              rctxt.translate(0,160);
              rctxt.scale(1,-1);
              gradient = rctxt.createLinearGradient(0, 105, 0, 160);
              gradient.addColorStop(1, "rgba(255, 255, 255, 0.3)");
              gradient.addColorStop(0, "rgba(255, 255, 255, 1.0)");
              rctxt.fillStyle = gradient;
              rctxt.drawImage(video, 0, 0, 320, 160);
            
15
<video> and Web Workers - face detection
    function isSkin(rn, gn, bn, base) {
      if (rn > 0.35 && rn < 0.5 && gn > 0.2 && gn < 0.5 && bn > 0.2 && bn < 0.35 
                    && base > 250) {
        return true;
      } else {
        return false;
      }
    }
            
16

PAUSE

17
Behind the scenes
  • Log onto Web Server
  • Upload Video and Poster
  • Create Web page
  • Make sure Mime Type is correct
  • Fallback for old browsers
18
Simple <video> element fallback
              <video controls poster="HelloWorld.png" width="500">
                  <source src="HelloWorld.mp4" type="video/mp4">
                  <source src="HelloWorld.webm" type="video/webm">
                  <source src="HelloWorld.ogv" type="video/ogg">
              </video>
              Your browser does not support the <code>video</code> element.
              <a href="HelloWorld.webm">Download it</a> instead.
            
19
Flash fallback for the <video> element
              <video controls poster="HelloWorld.png" width="500">
                  <source src="HelloWorld.mp4" type="video/mp4">
                  <source src="HelloWorld.webm" type="video/webm">
                  <source src="HelloWorld.ogv" type="video/ogg">
              </video>
              <object type="application/x-shockwave-flash" data="player.swf" width="854" height="504">
                <param name="allowfullscreen" value="true">
                <param name="allowscriptaccess" value="always">
                <param name="flashvars" value="file=video.mp4">
                <!--[if IE]><param name="movie" value="player.swf"><![endif]-->
                <p>Your browser can’t play HTML5 video. <a href="HelloWorld.webm">Download
                it</a> instead.</p>
              </object>
            
20

Audio Element & Audio Manipulation

21
Innocent browser <audio> element
              <audio controls>
                  <source src="revolve.mp3" type="audio/mp4">
                  <source src="revolve.ogg" type="audio/ogg">
              </audio>
            
22
Mozilla Audio API show-off
Get Firefox 4 beta

http://videos.mozilla.org/serv/blizzard/audio-slideshow/
23

Media Accessibility

24
Key Focus Media Accessibility
  • Captions and Subtitles
  • Audio Descriptions
  •  
  • External Text Files
  • In-band content
25
Three key specifications
26
Captions, Subtitles, Text Descriptions
WEBVTT FILE

1
00:00:08,124 --> 00:00:10,742
Workstations and high end personal computers have been able to

2
00:00:10,742 --> 00:00:14,749
manipulate digital audio pretty easily for about fifteen years now.

3
00:00:14,749 --> 00:00:17,470
It's only been about five years that a decent workstation's been able

4
00:00:17,470 --> 00:00:21,643
to handle raw video without a lot of expensive special purpose hardware.
            
27
Example with Subtitles in JavaScript
28
Internationalized Subtitles
WEBVTT FILE

00:00:15.000 --> 00:00:17.950 A:end D:vertical
在左边我们可以看到...

00:00:18.160 --> 00:00:20.080 A:end D:vertical
在右边我们可以看到...

00:00:20.110 --> 00:00:21.960 A:end D:vertical S:70% L:40%
...捕蝇草械.
            
29
Karaoke-Style Paint-On Subtitles
WEBVTT FILE

1
00:00:10.000 --> 00:00:12.210
<00:00:10.035>Chocolate <00:00:11.000>Rain

2
00:00:12.210 --> 00:00:15.910
<00:00:13.250>Some <00:00:13.500>stay <00:00:13.750>dry <00:00:14.25>and 
<00:00:14.50>others <00:00:15.00>feel <00:00:15.25>the <00:00:15.50>pain

3
00:00:15.910 --> 00:00:15.920
<00:00:16.000>Chocolate <00:00:16.500>Rain
          
30
Markup

The <track> Element

<video src="media/sintel.ogv" poster="media/poster_sintel.png" controls width="50%">

  <track kind="subtitles" src="text/vid1-en.vtt" srclang="en" label="English" default>
  <track kind="subtitles" src="text/vid1-fr.vtt" srclang="fr" label="Français">
  <track kind="subtitles" src="text/vid1-de.vtt" srclang="de" label="Deutsch">

  <track kind="captions" src="text/vid1-cap.vtt" srclang="en" label="English">

  <track kind="descriptions" src="text/vid1-desc.vtt" srclang="en" label="English">

  <track kind="chapters" src="text/vid1-chap.vtt" srclang="en" label="English">

  <track kind="metadata" src="text/vid1-json.vtt" srclang="en" label="English">

</video>
            
31
Next Steps
  • WebVTT format improvements
  • JS implementations
  • Browser Vendors to support markup and format
32
More Details: Shameless Plug

Definitive Guide to HTML5 video book cover

For the details, see my book:

The Definitive Guide to HTML5 Video

33
Thank you
Questions?

              Silvia Pfeiffer
              blog.gingertech.net
              silviapfeiffer1@gmail.com
              @silviapfeiffer
            
Thanks go to the Blender organisation and to Creative Commons for their videos.
34