Client-side Video Search Inside

Published: 2016-09-23 05:00 -0400

Below the video use the input to search within the captions. This is done completely client-side. Read below for how it was done.

As part of thinking more about how to develop static websites without losing functionality, I wanted to be able to search inside a video.

To create the WebVTT captions file I used random words and picked 4 randomly to place as captions every 5 seconds throughout this 12+ minute video. I used an American English word list, randomly sorted it and took the top 100 words. Many of them ended with “’s” so I just removed all those for now. You can see the full word list, look at the WebVTT file, or just play the video to see the captions.

sort -R /usr/share/dict/american-english | head -n 100 > random-words.txt

Here’s the script I used to create the WebVTT file using our random words.

#!/usr/bin/env ruby
random_words_path = File.expand_path '../random-words.txt', __FILE__
webvtt_file_path = File.expand_path '../search-webvtt.vtt', __FILE__

def timestamp(total_seconds)
  seconds = total_seconds % 60
  minutes = (total_seconds / 60) % 60
  hours = total_seconds / (60 * 60)
  format("%02d:%02d:%02d.000", hours, minutes, seconds)
end

words = File.read(random_words_path).split
cue_start = 0
cue_end = 0
File.open(webvtt_file_path, 'w') do |fh|
  fh.puts "WEBVTT\n\nNOTE This file was automatically generated by http://ronallo.com\n\n"
  144.times do |i|
    cue_words = words.sample(4)
    cue_start = i * 5
    cue_end = cue_start + 5
    fh.puts "#{timestamp(cue_start)} --> #{timestamp(cue_end)}"
    fh.puts cue_words.join(' ')
    fh.puts
  end
end

The markup including the caption track looks like:

<video preload="auto" autoplay poster="https://siskel.lib.ncsu.edu/SCRC/AV9_FM_1-boiling-process/AV9_FM_1-boiling-process.png" controls>
  <source src="https://siskel.lib.ncsu.edu/SCRC/AV9_FM_1-boiling-process/AV9_FM_1-boiling-process.mp4" type="video/mp4">
  <source src="https://siskel.lib.ncsu.edu/SCRC/AV9_FM_1-boiling-process/AV9_FM_1-boiling-process.webm" type="video/webm">
  <track id="search-webvtt" kind="captions" label="captions" lang="en" src="/video/search-webvtt/search-webvtt.vtt" default>
</video>

<p><input type="text" id="search" placeholder="Search the captions..." width="100%" autocomplete='off'></p>
<div id="result-count"></div>
<div class="list-group searchresults"></div>
<script type="text/javascript" src="/javascripts/search-webvtt-726f2ffd.js"></script>

In the browser we can get the WebVTT cues and index each of the cues into lunr.js:

var index = null;
// store the cues with a key of start time and value the text
// this will be used later to retrieve the text as lunr.js does not
// keep it around.
var cue_docs = {};
var video_elem = document.getElementsByTagName('video')[0];
video_elem.addEventListener("loadedmetadata", function () {
  var track_elem = document.getElementById("search-webvtt");
  var cues = track_elem.track.cues;
  index = lunr(function () {
    this.field('text')
    this.ref('id')
  });

  for (var i = 0; i <= cues.length - 1; i++) {
    var cue = cues[i];
    cue_docs[cue.startTime] = cue.text;
    index.add({
      id: cue.startTime,
      text: cue.text
    });
  }
});

We can set things up that when a result is clicked on we’ll get the data-seconds attribute and make the video jump to that point in time:

$(document).on('click', '.result', function(){
  video_elem.currentTime = this.getAttribute('data-seconds');
});

We create a search box and display the results. Note that the searching itself just becomes one line:

$('input#search').on('keyup', function () {
  // Get query
  var query = $(this).val();
  // Search for it
  var result = index.search(query);
  var searchresults = $('.searchresults');
  var resultcount = $('#result-count');
  if (result.length === 0) {
    searchresults.hide();
  } else {
    resultcount.html('results: ' + result.length);
    searchresults.empty();
    // Makes more sense in this case to sort by time than relevance
    // The ref is the seconds
    var sorted_results = result.sort(function(a, b){
      if (a.ref < b.ref) {
        return -1;
      } else {
        return 1;
      }
    });

    // Display each of the results
    for (var item in sorted_results) {
      var start_seconds = sorted_results[item].ref;
      var text = cue_docs[start_seconds];
      var seconds_text = start_seconds.toString().split('.')[0];
      var searchitem = '<a class="list-group-item result" data-seconds="'+ start_seconds +'" href="#t='+ start_seconds + '">' + text + ' <span class="badge">' + seconds_text + 's</span></a>';
      searchresults.append(searchitem);
    }
    searchresults.show();
  }
});

And that’s all it takes to create search within for a video for your static website.

Video from Boiling Process with Sine Inputs–All Boiling Methods.