The Complete Video Solution for Web and Mobile Developers

This article was originally published on Cloudinary Blog. Thank you for supporting the partners who make SitePoint possible.

Videos in web sites and apps are starting to catch up with images in terms of popularity and they are a constantly growing part of the media strategy for most organizations. This means bigger challenges for developers who need to handle these videos in their web sites and mobile apps. Cloudinary’s here to solve developer needs around image and video management. In this article, we are going to introduce Cloudinary’s complete cloud-based video management solution for developers.

Here are just some highlights:

• Video upload API and UI widget – upload your videos directly to cloud storage
• Programmable and interactive interface for managing video assets
• Real-time video transcoding and manipulation via CDN delivery URLs
• Customizable video player with user engagement analytics (see a demo)
• Live video streaming directly from web and mobile apps (see a demo)
• AI based video tagging and transcription

Website Videos Are Becoming Mainstream

Videos are now responsible for about 25% of the average download bandwidth of web sites (SpeedCurve analysis). As you can see in the chart below, this reflects a huge growth of about 400% compared to just two years ago. 2017 is definitely the year of video and while <video> didn’t kill the <img> tag and (probably) never will, managing and delivering videos in modern web sites and mobile apps involves growing complexities for developers.

When Cloudinary’s service was publicly launched back in 2012, our first mission was to solve image management needs for web and app developers: from uploading images from any device and storing them in the cloud, manipulating the images on-the-fly to match any graphic design and screen resolution, to dynamically optimizing the images and delivering them via a fast CDN to worldwide users. Then, in May 2015, we expanded our solution and introduced Cloudinary’s cloud-based service for video upload, real-time manipulation and optimized viewing.

Our new offer provided the same cloud-based service API for both images and videos. While the image management space keeps evolving, since 2015, wehave also continued enhancing our video transcoding capabilities. Today, about 30% of Cloudinary’s 5000 paying customers already upload and manipulate tens of millions of videos every month, and this number is growing quite rapidly.

The challenges developers face with the videos in their web sites tend to be more complex than images. The video files can be huge, which means longer upload & download times and very CPU-intensive transcoding and manipulation. The set of potential devices, resolutions, video formats and video codecs is large and confusing. The desired optimal user experience requires modern video players with user engagement statistics and, in some cases, also monetization capabilities.

Today we are excited to introduce the next generation of our cloud-based video management service – even more advanced real-time video transcoding together with a modern video player, live video streaming, AI based video tagging and transcript, and more; All aimed at simplifying the video workflow for web and mobile developers while improving and enhancing the end-user experience.

A Complete Video Management Solution for Developers

Whether you are delivering top quality professional videos or user-generated clips, whether you have an eCommerce site, news channel, travel forum, or advertising agency, the back-end challenges involved in quickly uploading and then delivering optimized, high quality video to any device in any location is always there, as are the challenges of adjusting the output to match your design needs, and providing a great front-end user experience. And all these challenges mount if you want to broadcast that video live or integrate and share that content in social networks.

Cloudinary addresses all this and more by providing the following capabilities as part of a single, streamlined solution:

Upload, Storage and Administration

An end-to-end solution for dynamic videos in web sites and apps starts from the ability to upload directly from the browser or mobile apps. A single line of code allows users to upload any image or video file to the cloud without even going through your servers:

  Cloudinary::Uploader.upload(file, 
            resource_type: :video, public_id: "sea_turtle")

You can also use the upload widget, which provides a built-in user interface for your users to select and upload image and video files.

Uploaded videos are stored securely in the cloud. Once uploaded, you can manage your cloud-based database of media files using our administrative API or using Cloudinary’s Digital Asset Management user interface.

Real-Time Video Transcoding, Manipulation and Streaming

Video files might be uploaded in various different formats, codecs, resolutions and aspect ratios. These properties will most likely not match the design of your site and the various devices, browsers and resolutions your visitors use. Videos are delivered to web sites using HTTP/S URLs. Cloudinary supports format conversion, video codec optimization, and resizing and cropping of videos using regular CDN delivery URLs. The video transcoding and manipulation is performed according to the instructions in the URL, while the video processing is done in real-time, on-the-fly, in the cloud when the first user accesses the URL.

For example, below is a video as originally uploaded followed by a web friendly MP4 200×200 cropped version of the video. The transcoding and cropping is done on-the-fly by adding the w_200,h_200,c_fill,g_north dynamic manipulation instructions to the video delivery URL.

https://res.cloudinary.com/demo/video/upload/w_200,h_200,c_fill,g_north/sea_turtle.mp4

There are plenty of additional video manipulation building blocks that you can mix & match to create your desired composite videos. These include effects, filters, overlays of images, videos and text, and more. Below you can see a more advanced example that applies a color saturation decrease filter and adds an image (watermark), as well as video and text overlays at selected times within the video.

https://res.cloudinary.com/demo/video/upload/ar_21:9,c_fill,w_500,ac_none/e_saturation:-50/l_cloudinary_icon,g_north_east,e_brightness:200,o_40,x_5,y_5,w_120/l_text:Roboto_34px_bold:Cute.Animals,co_white,g_west,x_10,so_2/l_video:funny_dog,w_200,g_south_east,y_10,x_10,so_2/sea_turtle.mp4

Videos can be converted to different formats simply by modifying the file extension. For example, changing the extension to ‘.m3u8’ will automatically generate all the index files required for our built-in HLS and MPEG-DASH adaptive bitrate streaming. You can see more online video transcoding examples in the following demo: https://demo.cloudinary.com/video/

Customizable Video Player

The examples above demonstrated URL-based back-end techniques that you can use to generate and deliver videos. But Cloudinary wanted to take it further and provide developers with a complete, yet simple solution for addressing the front-end video playing experience as well.

A new Cloudinary Video Player is now publicly available. The player can be initiated with a single line of code that accepts a video ID and automatically builds video manipulation and delivery URLs. Web friendly video formats such as MP4 are used and HLS & MPEG-DASH adaptive bitrate streaming is automatically set-up.

The video player can be initiated either using HTML markup or programmatically using JavaScript:

var vplayer = cld.videoPlayer("demo-player", {
   publicId: 'rafting',
   loop: true,
   controls: true,
   autoplayMode: 'on-scroll',
   transformation: { width: 400, crop: 'limit ' },
   posterOptions: {publicId: 'mypic', transformation { effect: ['sepia']}},
   sourceTypes: ["hls", "mp4"],
 })

The player has two built-in look & feel themes that can be further customized. It supports recommended videos suggestions and automatic playlist creation for a given tag assigned to multiple videos. You can track user engagement by monitoring events that can be automatically sent directly to analysis systems such as Google Analytics. See the video player documentation for more details.

The video player is an open source project based on the popular VideoJS open source video player, with its large ecosystem of plugins and customizations.

Interactive examples of the video player can be found in our video player demo page.

Live Video Streaming with Real-Time Transcoding

The common flow of first uploading videos and when done, delivering them to users is gradually clearing the way to a more advanced flow of live video streaming. Cloudinary now offers (beta) support for live streaming of video content directly from web sites and applications.

Video transcoding and manipulation is done in real-time on the live stream exactly in the same way it is done on pre-uploaded videos – simultaneously generating multiple different versions out of the original video – different resolutions, cropping modes, encoding quality levels, watermarks, effects, personalized text overlays and more.

Live streaming is based on the WebRTC protocol, and you can instruct Cloudinary to automatically stream the videos directly to Facebook or YouTube using the RTMP protocol.

You can try out the full live streaming experience via the mobile web demo application

AI-Based Video Tagging and Transcription

If your web application has many videos or supports user-generated video content, smart video management automation would make your life easier and might improve user engagement.

Automatic Video Subtitle Creation

Auto-playing muted video became very popular in news sites and social networks such as Facebook. New versions of web browsers even limit the auto playing capabilities and prevent auto-playing with sound. To make verbal videos effective even when muted, subtitles are necessary. And if you want to support uploading videos to a social feed with muted auto-play, you probably need the videos to have the subtitles already embedded in them.

AI-based video transcript is now available as a fully integrated add-on powered by Google’s Cloud Speech API. By setting Cloudinary’s raw_convert upload API parameter to google_speech, the audio channel of the video is automatically processed and a transcript file is generated in your media library.

Cloudinary::Uploader.upload("lincoln.mp4", :resource_type => :video, :raw_convert  => "google_speech")

Generating a video with embedded subtitles based on automatic transcription is as simple as adding another parameter to your dynamic video delivery URL (in the example below, adding l_subtitles:lincoln.transcript). You can even enhance the subtitles with additional options such as your choice of font, font size, color, etc. The original video (which didn’t include any subtitles), now includes automatically generated captions based on Google’s speech to text AI engine.

https://res.cloudinary.com/demo/video/upload/l_subtitles:arial_20:lincoln.transcript,co_yellow/lincoln.mp4

Automatic Video Tagging

It’s a common practice to organize your media database or eCommerce product catalog by categorizing and tagging your images and videos to better match uploaded content to your users. Cloudinary now supports automatic AI-based tagging of uploaded videos.

The automatic tagging is available as a fully integrated add-on powered by Google’s Cloud Video Intelligence. By setting Cloudinary’s categorization upload API parameter to google_video_tagging, the video is automatically analyzed, and a list of detected tag categories is included in the response. If you also set an auto-tagging level, then any category that exceeds the requested confidence level automatically gets added to the resource’s tag list.

Cloudinary::Uploader.upload("turtle.mp4", :resource_type => :video, :categorization => "google_video_tagging", :auto_tagging => 0.7)

Below is a sample response from the automatic tagging. Categories with a confidence level above the given threshold are automatically assigned. Following that, you can see the full list of all detected categories and the time segment that each suggested tag applies to. You can list, browse, delete and search images and videos by the automatically assigned tags, either via API or our interactive UI.

"tags": [ "turtle", "animal" ...],
"data": [
  {"tag": "turtle", "start_time_offset"=>0.0, "end_time_offset"=>13.44, 
    "confidence": 0.93},
  {"tag": "animal", "start_time_offset"=>0.0, "end_time_offset"=>13.44, 
    "confidence": 0.93} ...  ]

Image and Video, not Image And Video

In the first couple of years after publicly launching Cloudinary in 2012, I wrote most of the technical blog posts that we published. But we’ve grown, and it’s been quite a while since I’ve written one. When I decided to write this one introducing the next generation of our video solution, I thought that it would be a quick task…

Well, I was quite wrong – it was not quick at all. As you can see, trying to cover the highlights of the existing components and the new features of our video solution resulted in this longer-than-expected blog post, yet I still feel that I skipped so many cool features and use cases.
As you probably already understand, we are proud of our enhanced Image and Video Management service and its new capabilities. We even made a slight yet significant update on our home page to clearly show where video fits within our complete solution:

We invite you to try out the video solution with its new capabilities and share your feedback (community page or forums). And for our part, we’ll keep working to further build and enhance it according to your feature requests and suggestions.

All video management features are available now in all our plans including the free plan. You can create your free account here.